Question 580
A medical-answer evaluation service shares more patient information with external reviewers than they need. A claims advisor logs approval status but discards specialists' correction reasons. A legal compliance assistant encounters disputed interpretations that could carry binding consequences. Which THREE oversight controls address these separate risks? Choose THREE.
- Require stronger approval paths for exceptional high-impact actions
- Give reviewers only necessary sensitive information
- Use risk-based sampling for low-impact decisions
- Require named authorization before moving money
- Escalate binding legal interpretations to qualified reviewers
- Capture reviewer overrides as structured improvement signals
Explanation
Correct Answers
B:Minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; review workflows should preserve decision quality while protecting privacy; its corresponding workstream should be validated by unauthorized field exposure and successful review accuracy after data minimization.
E:Define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; a consequence-aware test checks whether unsupported authoritative decisions are avoided; its corresponding workstream should be validated by correct escalation of ambiguous legal cases with reviewer attribution.
F:Record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; structured override trends reveal whether feedback improves the system over time; its corresponding workstream should be validated by recurrence rate of categorized, previously corrected error types.
Incorrect Answers
A:Apply risk-tiered approvals, including dual control or independent sign-off where policy demands, to exceptionally consequential mutations; a uniform low-friction approval does not reflect differing blast radius and operational risk; its natural acceptance measure is critical-action dual-approval compliance and zero bypasses during staged tests, which does not satisfy the separate workstreams being assessed.
C:Risk-tiered sampling can reduce review load for reversible low-impact outputs, but the three issues here concern unnecessary patient-data disclosure to reviewers, unrecorded expert corrections, and legally ambiguous cases requiring qualified escalation.
D:Separate recommendation from execution and require an authenticated human approval bound to the exact amount, payee, and transaction; general user authentication does not confer approval for a specific financial transfer; its natural acceptance measure is zero disbursements without an auditable transaction-specific approver, which does not satisfy the separate workstreams being assessed.
Question 581
A newly hired operations team is taking ownership of a live Claude workflow. Which observed drill outcome would demonstrate that its runbook is usable without help from the original architects?
- Successful completion of standard support drills by staff who did not build the system
- Consumer contract pass rate under timeouts, invalid arguments, and permission denials
- Ability to identify affected decisions after simulated policy and workload changes
- Reproducibility of a sampled historical answer using its full manifest
Explanation
Correct Answer
A:Independent operator readiness is stronger evidence than architect-led demonstrations; measure successful completion of standard support drills by staff who did not build the system at the release gate; provide system overview, access setup, troubleshooting flowcharts, monitoring thresholds, and named support responsibilities.
Incorrect Answers
B:Contract tests validate the documented behavior under non-happy paths; define typed inputs, outputs, error semantics, idempotency, required scopes, and audit obligations for tool boundaries; this tests consumer contract pass rate under timeouts, invalid arguments, and permission denials; the actual release condition is successful completion of standard support drills by staff who did not build the system.
C:Assumption-aware ADRs support controlled adaptation rather than accidental drift; record architectural context, alternatives, decision rationale, constraints, and explicit conditions for revisiting choices; this tests ability to identify affected decisions after simulated policy and workload changes; the actual release condition is successful completion of standard support drills by staff who did not build the system.
D:Versioned provenance enables defensible debugging and rollback; maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; this tests reproducibility of a sampled historical answer using its full manifest; the actual release condition is successful completion of standard support drills by staff who did not build the system.
Question 582
A multi-squad software organization connects Claude Code to project-scoped MCP resources. Which identity-based negative test would prove that developers cannot cross the boundaries of other teams' repositories?
- Subagent task completion with zero out-of-scope tool access
- Denial of prohibited commands after attempts to override local and project configuration
- Access denial for cross-project MCP resources under user identity tests
- Secret-scanning results and blocked exfiltration on seeded repository credentials
- Hook execution and blocked noncompliant operations under simulated bypass attempts
Explanation
Correct Answer
C:Scope checks confirm that connector convenience has not expanded data visibility; measure access denial for cross-project MCP resources under user identity tests at the release gate; choose appropriate configuration scope and per-user or service authorization for MCP servers, avoiding shared cross-project secrets.
Incorrect Answers
A:Role-isolation tests prove that decomposition also restricts authority; give each delegated role explicit objectives, context limits, and tool permissions with clear return artifacts; this tests subagent task completion with zero out-of-scope tool access; the actual release condition is access denial for cross-project MCP resources under user identity tests.
B:Policy precedence tests validate that enterprise controls remain effective; use organization-managed policy and supported permission configuration so enforced restrictions cannot be casually overridden by per-user convenience settings; this tests denial of prohibited commands after attempts to override local and project configuration; the actual release condition is access denial for cross-project MCP resources under user identity tests.
D:Source and runtime scans target the actual paths by which developer secrets can leak; use approved secret storage, scoped environment access, and redaction while avoiding credentials in repositories, prompts, or logs; this tests secret-scanning results and blocked exfiltration on seeded repository credentials; the actual release condition is access denial for cross-project MCP resources under user identity tests.
E:Gate testing verifies that a real program, not a conversational promise, enforces the rule; configure supported hooks for deterministic checks at appropriate lifecycle points, with clear failure behavior and maintenance ownership; this tests hook execution and blocked noncompliant operations under simulated bypass attempts; the actual release condition is access denial for cross-project MCP resources under user identity tests.
Question 583
A scientific report contains numerical conclusions that do not appear in its cited papers. A separate invoice parser occasionally emits invalid structured records when optional notes contain unexpected formatting. Which TWO diagnostic practices address the factual and schema failures? Choose TWO.
- Verify whether claims are supported by supplied sources
- Investigate retrieval recall before changing the language model
- Separate external tool faults from model decisions
- Revalidate few-shot examples after policy changes
- Trace malformed outputs to prompt or schema boundaries
Explanation
Correct Answers
A:Compare material claims against cited evidence and flag unsupported statements for abstention or review; claim-level verification targets hallucinations that ordinary coherence checks miss; its corresponding workstream should be validated by supported-claim rate on source-checked summaries with seeded unsupported numbers.
E:Inspect instruction clarity, output-schema validation, and edge-case examples before assuming a core model reasoning failure; edge-specific parser tests identify contract failures separately from content comprehension; its corresponding workstream should be validated by schema error rate across multiline, null, and unusual-field edge cases.
Incorrect Answers
B:Inspect search coverage, index freshness, filters, and chunk retrieval before attributing the regression to Claude's reasoning; model upgrades cannot recover information that never reaches the answer context; its natural acceptance measure is recall of newly published policy passages and removal of stale hits, which does not satisfy the separate workstreams being assessed.
C:Propagate typed tool errors, retries, and unknown states instead of converting transport failures into negative facts; prompt changes alone cannot restore a failing service or make an empty placeholder a valid business result; its natural acceptance measure is correct 'unknown' handling and no false denials during injected API failures, which does not satisfy the separate workstreams being assessed.
D:Audit examples and prompt-version precedence when behavior consistently imitates outdated demonstrations; index re-embedding cannot resolve a stale rule embedded directly in the prompt template; its natural acceptance measure is correct labels on scenarios affected by the policy revision, which does not satisfy the separate workstreams being assessed.
Question 584
An employee copilot reads emails and PDFs from external senders. Which approach best meets the stated requirement?
- Use claim verification to mitigate confident fabrication
- Validate tool-result provenance and semantics
- Constrain outbound channels to prevent data exfiltration
- Threat-model prompt injection across lower-trust inputs
- Counter automation bias with visible uncertainty and review
Explanation
Correct Answer
D:Model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; a realistic attack corpus exercises both model behavior and execution controls; evaluate the design with unauthorized tool-action success rate over injected emails and documents.
Incorrect Answers
A:Require support from retrieved source passages and provide an explicit unknown path for unsupported claims; fluency and citation formatting cannot establish factual support on their own; this alternative would measure proportion of numeric claims supported by independently inspected references; it does not establish unauthorized tool-action success rate over injected emails and documents for the described use case.
B:Preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; a successful HTTP status does not mean returned data has decision authority; this alternative would measure zero external-result fields promoted to internal approvals in test cases; it does not establish unauthorized tool-action success rate over injected emails and documents for the described use case.
C:Restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; this alternative would measure blocked unauthorized destinations and absence of confidential fields in egress logs; it does not establish unauthorized tool-action success rate over injected emails and documents for the described use case.
E:Design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; a nominal human checkpoint is weak when the interface encourages blind confirmation; this alternative would measure reviewer error interception on seeded incorrect recommendations; it does not establish unauthorized tool-action success rate over injected emails and documents for the described use case.
Question 585
Assistants give different instructions because each developer relies on private session notes and undocumented local preferences. Which corrective change best addresses the root cause?
- Version team-wide CLAUDE.md guidance with the repository
- Limit MCP connections by project and user identity
- Keep credentials out of prompts and versioned configuration
- Use specialized subagents with bounded tools and instructions
- Use deterministic hooks to enforce required developer gates
Explanation
Correct Answer
A:Put reviewed repository-specific guidance in a shared CLAUDE.md file with appropriate scope and code review; reproducible onboarding shows whether team guidance is discoverable and current; verify the repair using consistency of generated changes with repository build and test conventions across fresh sessions.
Incorrect Answers
B:A working connection does not prove the assistant is restricted to the user's intended project resources; choose appropriate configuration scope and per-user or service authorization for MCP servers, avoiding shared cross-project secrets; that control is assessed by access denial for cross-project MCP resources under user identity tests; the incident instead requires consistency of generated changes with repository build and test conventions across fresh sessions.
C:Masking output after committing a secret cannot undo credential exposure or grant least privilege; use approved secret storage, scoped environment access, and redaction while avoiding credentials in repositories, prompts, or logs; that control is assessed by secret-scanning results and blocked exfiltration on seeded repository credentials; the incident instead requires consistency of generated changes with repository build and test conventions across fresh sessions.
D:Delegation without role scoping merely multiplies the number of powerful agents; give each delegated role explicit objectives, context limits, and tool permissions with clear return artifacts; that control is assessed by subagent task completion with zero out-of-scope tool access; the incident instead requires consistency of generated changes with repository build and test conventions across fresh sessions.
E:A written instruction alone is best-effort and does not guarantee enforcement of a critical command gate; configure supported hooks for deterministic checks at appropriate lifecycle points, with clear failure behavior and maintenance ownership; that control is assessed by hook execution and blocked noncompliant operations under simulated bypass attempts; the incident instead requires consistency of generated changes with repository build and test conventions across fresh sessions.
Question 586
A quality team disputes generated inspection decisions and needs to reconstruct which source revision and tool responses were used. Which strategy should be prioritized for this requirement?
- Link each output to source and processing trace IDs
- Check model output against a business schema before side effects
- Normalize incoming documents into a validated input contract
- Capture reviewer corrections as labeled feedback
- Use a queue and asynchronous workers for bursty workloads
Explanation
Correct Answer
A:Correlation IDs, source versions, and stage logs establish end-to-end lineage across extraction, retrieval, inference, and review; end-to-end provenance lets reviewers investigate disputed outcomes without assuming the final text is self-evidencing; evaluate the design with fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition.
Incorrect Answers
B:Use structured output parsing, semantic validation, and a separate authorized command handler so generated text is never treated directly as a write instruction; output parsing constrains generated data but does not by itself manage retry idempotency or transport durability; this alternative would measure invalid write payloads rejected before execution, with zero partial supplier mutations; it does not establish fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition for the described use case.
C:An input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values; input normalization addresses upstream ambiguity, not the authorization needed before writing to a finance ledger; this alternative would measure percentage of ingested documents meeting schema checks before any Claude call; it does not establish fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition for the described use case.
D:Preserve authorized reviewer labels linked to case versions and turn them into regression examples with data-retention controls; a feedback dataset improves future evaluation; it does not guarantee that the current response is timely or properly permissioned; this alternative would measure regression pass rate on previously corrected clause-omission cases; it does not establish fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition for the described use case.
E:Decouple intake from processing through durable queues, bounded workers, and status callbacks rather than keeping user connections open; queueing absorbs bursts but does not improve a weak answer rubric or justify a higher-privilege tool set; this alternative would measure job completion rate and queue age during the peak intake window; it does not establish fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition for the described use case.
Question 587
An account assistant incorrectly rejects a set of valid customer requests during a regional outage. Which strategy should be prioritized for this requirement?
- Establish a documented escalation path for service incidents
- Define SLOs and error budgets for agreed user outcomes
- Clarify shared responsibility for the service boundary
- Set observability and reporting expectations with stakeholders
Explanation
Correct Answer
A:Define incident severity, on-call ownership, manual fallback authority, and customer communication triggers; incident response tests verify that operational ownership works under pressure; evaluate the design with time to appropriate escalation and fallback activation during drills.
Incorrect Answers
B:Establish measurable user-facing service objectives, reporting windows, and error-budget policies tied to accountable actions; generic uptime claims can conceal slow or unusable answer experiences; this alternative would measure sLO compliance and prescribed actions during a simulated error-budget breach; it does not establish time to appropriate escalation and fallback activation during drills for the described use case.
C:Define provider, platform, application, security, and business-owner responsibilities in an explicit operating model; a vendor SLA typically does not assign responsibility for every custom integration and decision rule; this alternative would measure ownership coverage for failures across model, application, data, and tool layers; it does not establish time to appropriate escalation and fallback activation during drills for the described use case.
D:Agree on outcome-aligned service reports, cadence, escalation thresholds, and audiences during solution definition; platform telemetry is necessary but not equivalent to business service reporting; this alternative would measure delivery of agreed business and risk indicators on schedule with decision-ready interpretation; it does not establish time to appropriate escalation and fallback activation during drills for the described use case.
Question 588
A document-intake service will pass extracted figures from images and spreadsheets to downstream Claude steps. Before activating the model calls, the team needs to prove its normalized intake contract catches malformed records. What is the best release measure?
- Measured regression pass rate on previously corrected clause-omission cases
- Invalid write payloads rejected before execution, with zero partial supplier mutations
- Percentage of ingested documents meeting schema checks before any Claude call
- Job completion rate and queue age during the peak intake window
Explanation
Correct Answer
C:A schema-conformance gate reveals extraction failures before they can contaminate reasoning; measure percentage of ingested documents meeting schema checks before any Claude call at the release gate; an input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values.
Incorrect Answers
A:Retesting concrete corrections detects whether a release reintroduces known business mistakes; preserve authorized reviewer labels linked to case versions and turn them into regression examples with data-retention controls; this tests regression pass rate on previously corrected clause-omission cases; the actual release condition is percentage of ingested documents meeting schema checks before any Claude call.
B:Pre-write enforcement proves that the model cannot bypass the target system's integrity contract; use structured output parsing, semantic validation, and a separate authorized command handler so generated text is never treated directly as a write instruction; this tests invalid write payloads rejected before execution, with zero partial supplier mutations; the actual release condition is percentage of ingested documents meeting schema checks before any Claude call.
D:Measuring backlog and completion under load tests the exact burst tolerance the design promises; decouple intake from processing through durable queues, bounded workers, and status callbacks rather than keeping user connections open; this tests job completion rate and queue age during the peak intake window; the actual release condition is percentage of ingested documents meeting schema checks before any Claude call.
Question 589
A prompt team repeatedly tunes against the same twenty examples and sees continuously rising scores. Which design decision directly addresses this constraint?
- Use deterministic code checks for objective output contracts
- Include hostile prompt-injection and no-answer probes
- Protect a separate holdout set for regression decisions
- Use expert rubrics for nuanced qualitative outputs
- Stratify a golden dataset by real workload segments
Explanation
Correct Answer
C:Maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; untouched data estimates whether improvements transfer beyond the examples used during tuning; evaluate the design with holdout quality stability across successive prompt versions.
Incorrect Answers
A:Apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; subjective model grading is unnecessary and less reliable for an objectively checkable parser contract; this alternative would measure zero schema violations accepted by the downstream consumer across generated cases; it does not establish holdout quality stability across successive prompt versions for the described use case.
B:Add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; testing normal prompts alone cannot estimate resilience to malicious or unsatisfiable requests; this alternative would measure attack success rate and safe handling of intentionally unanswerable prompts; it does not establish holdout quality stability across successive prompt versions for the described use case.
D:Use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; pure string comparison is unsuitable when many substantively correct phrasings exist; this alternative would measure expert-rated factual fidelity and calibrated grader agreement on sampled explanations; it does not establish holdout quality stability across successive prompt versions for the described use case.
E:Build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; a random sample alone can underrepresent important low-frequency and high-consequence events; this alternative would measure per-stratum pass rates with no uncovered critical scenario classes; it does not establish holdout quality stability across successive prompt versions for the described use case.
Question 590
A software squad wants AI-assisted API documentation after a major service refactor. Which option addresses the immediate design need?
- Review AI-generated diffs for logic and security regressions
- Break large refactors into independently reviewable changes
- Use test-first acceptance cases for generated changes
- Generate documentation from verified source and review it
- Run reproducible CI gates on assisted code
Explanation
Correct Answer
D:Ground documentation generation in current code, schemas, and tests, then validate examples and have owners review changes; executable examples reveal fabricated or obsolete instructions; evaluate the design with documented endpoint examples executing successfully against the reviewed build.
Incorrect Answers
A:Require focused code review of the actual diff, threat-relevant tests, and approval before merging sensitive changes; successful compilation does not establish preservation of security controls; this alternative would measure detection of seeded authorization regressions during peer review and CI; it does not establish documented endpoint examples executing successfully against the reviewed build for the described use case.
B:Use small scoped changes, compatibility checkpoints, and separate pull requests tied to specific tests; one massive patch can reduce initial coordination but increases regression and rollback risk; this alternative would measure review time and defect localization per incremental change set; it does not establish documented endpoint examples executing successfully against the reviewed build for the described use case.
C:Define executable tests and expected behavior before relying on AI-generated implementation, then review failures and iterate; merely reading generated code for plausibility does not replace tests of business invariants; this alternative would measure passing unit and property tests including specified rounding boundaries; it does not establish documented endpoint examples executing successfully against the reviewed build for the described use case.
E:Execute standardized linting, type checks, tests, and security scanning in versioned CI environments before merges; local success can be environment-dependent and should not be the only release assurance; this alternative would measure cI pass reproducibility on clean builds with locked dependencies; it does not establish documented endpoint examples executing successfully against the reviewed build for the described use case.