Question 190
An expense assistant receives PDFs, spreadsheets, and photographs with inconsistent field names, units, and missing metadata. What action best fits the business and technical boundary?
- Normalize incoming documents into a validated input contract
- Check model output against a business schema before side effects
- Use a queue and asynchronous workers for bursty workloads
- Persist workflow state and make retries idempotent
Explanation
Correct Answer
A:An input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values; a schema-conformance gate reveals extraction failures before they can contaminate reasoning; evaluate the design with percentage of ingested documents meeting schema checks before any Claude call.
Incorrect Answers
B:Use structured output parsing, semantic validation, and a separate authorized command handler so generated text is never treated directly as a write instruction; output parsing constrains generated data but does not by itself manage retry idempotency or transport durability; this alternative would measure invalid write payloads rejected before execution, with zero partial supplier mutations; it does not establish percentage of ingested documents meeting schema checks before any Claude call for the described use case.
C:Decouple intake from processing through durable queues, bounded workers, and status callbacks rather than keeping user connections open; queueing absorbs bursts but does not improve a weak answer rubric or justify a higher-privilege tool set; this alternative would measure job completion rate and queue age during the peak intake window; it does not establish percentage of ingested documents meeting schema checks before any Claude call for the described use case.
D:Persist state transitions and use idempotency keys at side-effect boundaries to avoid duplicate transactions across replays; durable-state control prevents duplicate effects but cannot decide whether a refund is substantively authorized; this alternative would measure repeated and interrupted test runs yielding exactly one committed credit per approved case; it does not establish percentage of ingested documents meeting schema checks before any Claude call for the described use case.
Question 191
A safety incident classifier confuses 'near miss' and 'actual injury' even after receiving correct definitions. Which intervention is most appropriate for this use case?
- Start with clear zero-shot instructions for unambiguous tasks
- Break multi-step reasoning into checkable intermediate outputs
- Use contrastive examples when two outputs look similar
- Add few-shot examples for difficult boundary cases
Explanation
Correct Answer
D:Provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases; held-out boundary performance shows whether examples teach the distinction rather than merely echoing training examples; evaluate the design with recall on previously confused boundary examples outside the prompt set.
Incorrect Answers
A:A concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; zero-shot prompting may fail where domain edge cases or specialized formatting rules are not expressible clearly in prose; this alternative would measure label accuracy and token use versus a heavier example-based prompt; it does not establish recall on previously confused boundary examples outside the prompt set for the described use case.
B:Require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; requesting longer explanations without validation can make unsupported conclusions appear more persuasive; this alternative would measure intermediate artifact validity and final score agreement with a policy-engine oracle; it does not establish recall on previously confused boundary examples outside the prompt set for the described use case.
C:Show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; adding examples without highlighting the differentiating condition can reinforce superficial formatting only; this alternative would measure correct type selection on unseen evidence-versus-action pairs; it does not establish recall on previously confused boundary examples outside the prompt set for the described use case.
Question 192
A coding copilot analyzes incident bundles that contain deliberately planted credential strings. Before allowing broader operational use, which measured result is the strongest proof of safe disclosure behavior?
- Measured valid schema rate and zero unauthorized enum values reaching the consumer
- Secret-exposure rate under seeded credential-bearing diagnostic inputs
- Rate of user messages that fail to override defined task boundaries in regression tests
- Prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use
- Out-of-scope refusal and handoff accuracy on a policy-boundary test set
Explanation
Correct Answer
B:Seeded canaries allow a measurable check of unwanted disclosure paths; measure secret-exposure rate under seeded credential-bearing diagnostic inputs at the release gate; redact sensitive fields upstream, prohibit secret reproduction in instructions, and screen outbound content before release.
Incorrect Answers
A:Parser-level rejection tests verify the actual interface boundary; request a documented structured format and validate schema, allowed values, and business semantics before downstream use; this tests valid schema rate and zero unauthorized enum values reaching the consumer; the actual release condition is secret-exposure rate under seeded credential-bearing diagnostic inputs.
C:Boundary-specific tests confirm that changing user content cannot rewrite the application contract; separate stable system-level policy from user content and tool results, retaining consistent privilege boundaries across requests; this tests rate of user messages that fail to override defined task boundaries in regression tests; the actual release condition is secret-exposure rate under seeded credential-bearing diagnostic inputs.
D:Adversarial retrieval tests exercise the trust boundary rather than only neutral documents; clearly separate lower-trust source material from application instructions and enforce permissions at the tool layer as well; this tests prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use; the actual release condition is secret-exposure rate under seeded credential-bearing diagnostic inputs.
E:Boundary tests demonstrate task scoping rather than only fluent responses on ordinary requests; define the assistant's permitted tasks, out-of-scope handling, and escalation behavior in stable application-level instructions, backed by independent access enforcement; this tests out-of-scope refusal and handoff accuracy on a policy-boundary test set; the actual release condition is secret-exposure rate under seeded credential-bearing diagnostic inputs.
Question 193
A policy portal publishes amended entitlements and revokes obsolete documents. Which acceptance measurement would show that the retrieval index reflects publication and withdrawal events promptly?
- Recall at k and evidence-coverage rate for known-answer queries
- Unauthorized chunk exposure rate under staff permission matrices, targeting zero
- Publication-to-search freshness lag and absence of withdrawn clauses in results
- Retrieval recall before and after a complete embedding-index migration
Explanation
Correct Answer
C:Update/delete tests prove the index tracks the authoritative corpus; measure publication-to-search freshness lag and absence of withdrawn clauses in results at the release gate; build a monitored ingestion path for adds, updates, deletes, and effective dates, with freshness checks and removal of superseded chunks.
Incorrect Answers
A:Separating retrieval metrics identifies which pipeline stage needs repair; measure retrieval recall, ranking relevance, and evidence coverage on labeled query-document pairs before optimizing generation; this tests recall at k and evidence-coverage rate for known-answer queries; the actual release condition is publication-to-search freshness lag and absence of withdrawn clauses in results.
B:The security test must inspect retrieved inputs, not just final visible answers; apply document permissions to retrieval candidates before material enters model context or logs; this tests unauthorized chunk exposure rate under staff permission matrices, targeting zero; the actual release condition is publication-to-search freshness lag and absence of withdrawn clauses in results.
D:Version-aligned search tests detect vector-compatibility regressions; record embedding model and preprocessing versions, re-embed consistently, and use controlled index migration; this tests retrieval recall before and after a complete embedding-index migration; the actual release condition is publication-to-search freshness lag and absence of withdrawn clauses in results.
Question 194
The full solution is built before testing the worst-quality scans, then fails on the key document class. Which action best removes the underlying failure mode?
- Record nonfunctional requirements as measurable thresholds
- Obtain sign-off on explicit acceptance criteria and exclusions
- Validate risky assumptions through narrow experiments
- Interview each role that owns a distinct outcome or risk
- Map the owner and authority of every required data source
- Capture external dependencies and acceptance risks explicitly
Explanation
Correct Answer
C:Identify high-risk assumptions and design small feasibility spikes with explicit success and stop criteria; evidence-based discovery reduces commitment to unverified technical assumptions; verify the repair using resolution rate of prioritized unknowns through measured prototypes.
Incorrect Answers
A:Functional task lists do not tell designers what constitutes acceptable production operation; specify capacity, tail latency, availability, privacy, and recovery objectives with measurement windows and ownership; that control is assessed by agreed p95, availability, and load thresholds exercised in an acceptance environment; the incident instead requires resolution rate of prioritized unknowns through measured prototypes.
B:Informal meeting consensus may not survive staff changes or new deployment expectations; document in-scope outcomes, exclusions, metrics, responsibilities, and go-live gates for accountable approval; that control is assessed by signed acceptance criteria mapped to demonstrable tests and named owners; the incident instead requires resolution rate of prioritized unknowns through measured prototypes.
D:A sponsor's high-level vision does not fully describe operational controls or edge cases; use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders; that control is assessed by coverage of approved requirements traced to affected stakeholder groups; the incident instead requires resolution rate of prioritized unknowns through measured prototypes.
E:A list of desired features cannot grant access or establish source truthfulness; create a source inventory with accountable owners, sensitivity, update rate, and approval dependencies before committing to the design; that control is assessed by percentage of required sources with approved owners, legal uses, and integration paths; the incident instead requires resolution rate of prioritized unknowns through measured prototypes.
F:An AI model benchmark cannot predict every upstream system's production behavior; maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; that control is assessed by verified readiness of high-impact external dependencies against test quotas and outage cases; the incident instead requires resolution rate of prioritized unknowns through measured prototypes.
Question 195
A research application makes expensive requests hours apart, with repeated background text on only some calls. Which realistic cost test would reveal whether maintaining prompt-cache breakpoints provides a net benefit?
- Net cost per accepted request across realistic interarrival times
- Prefix reuse ratio when request metadata changes across calls
- Cache-read input tokens and end-to-end latency over repeated requests
- Task success with reduced irrelevant instruction tokens across mixed workflows
Explanation
Correct Answer
A:Reuse-adjusted economics tests whether the caching decision pays off under actual traffic; measure net cost per accepted request across realistic interarrival times at the release gate; assess eligible caching windows, traffic frequency, and write/read economics against real reuse intervals before selecting a cache policy.
Incorrect Answers
B:Controlled request pairs reveal avoidable cache invalidation; move changing metadata after the stable prefix so cache keys remain reusable across otherwise similar requests; this tests prefix reuse ratio when request metadata changes across calls; the actual release condition is net cost per accepted request across realistic interarrival times.
C:Cache-hit telemetry verifies the optimization rather than relying on estimated token savings; put reusable instructions and reference material in a consistent prefix and use supported prompt-cache controls while placing variable requests afterward; this tests cache-read input tokens and end-to-end latency over repeated requests; the actual release condition is net cost per accepted request across realistic interarrival times.
D:On-demand loading must retain quality while avoiding unnecessary context overhead; keep Skills discoverable and load task-relevant instructions or resources on demand, maintaining scope and permission constraints; this tests task success with reduced irrelevant instruction tokens across mixed workflows; the actual release condition is net cost per accepted request across realistic interarrival times.
Question 196
Concurrent workers modify one shared summary and intermittently overwrite each other's findings. Which action best removes the underlying failure mode?
- Parallelize only truly independent branches
- Define input and output contracts for each subtask
- Partition research by source and reconcile conflicts explicitly
- Checkpoint long-running work for replay and recovery
Explanation
Correct Answer
A:Isolate independent tasks, retain separate results, then join only when required dependencies have completed; deterministic merge checks measure integrity rather than only wall-clock speed; verify the repair using correct merged clause counts across repeated parallel execution races.
Incorrect Answers
B:Contracts define allowed shapes and semantics but cannot ensure a source is truthful; typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions; that control is assessed by contract test pass rate for independently upgraded stages and invalid payload rejection; the incident instead requires correct merged clause counts across repeated parallel execution races.
C:Simple partitioning can preserve conflicting claims unless the merge step includes a reconciliation rule; analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; that control is assessed by conflict detection recall and citation coverage across independently processed source sets; the incident instead requires correct merged clause counts across repeated parallel execution races.
D:Checkpointing reduces replay cost but cannot correct a defective classification rubric; durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects; that control is assessed by successful recovery after injected worker failures without duplicated writes or stages; the incident instead requires correct merged clause counts across repeated parallel execution races.
Question 197
A multilingual public-benefits advisor is successful overall but produces more incorrect guidance for one language population. Separately, a scholarship eligibility tool offers no meaningful way to dispute a recommendation based on inaccurate records. Which TWO fairness and accountability measures should be adopted? Choose TWO.
- Design an override path proportionate to consequences
- Publish scope, limitations, and residual risks to stakeholders
- Test accessibility of generated explanations and interfaces
- Provide a route to challenge significant recommendations
- Measure errors across relevant user subgroups
Explanation
Correct Answers
D:Provide a contest and correction process with accountable human ownership for significant impacts; remedy effectiveness measures whether affected people can obtain meaningful reconsideration; its corresponding workstream should be validated by time to resolve substantiated challenges and proportion of cases with documented review.
E:Evaluate suitable subgroup error rates and access outcomes, with privacy-aware sampling and appropriate fairness review; group-specific evaluation reveals concentrated harm masked by pooled scores; its corresponding workstream should be validated by quality and escalation parity across predeclared language segments.
Incorrect Answers
A:Preserve an authorized override and recovery route scaled to decision consequences, with feedback recorded for evaluation; human review after an irreversible action may be too late to prevent harm; its natural acceptance measure is successful intervention before seeded high-impact incorrect actions are committed, which does not satisfy the separate workstreams being assessed.
B:Maintain clear documentation of intended use, unsupported scenarios, measured performance, and escalation requirements; a generic disclaimer cannot compensate for hidden high-risk use or unavailable oversight; its natural acceptance measure is accuracy of user understanding of supported versus unsupported tasks, which does not satisfy the separate workstreams being assessed.
C:Include accessibility requirements, assistive-technology testing, and inclusive content rubrics in evaluation and design; factual accuracy alone does not make the interface usable to everyone; its natural acceptance measure is task completion by users of tested assistive technologies on representative flows, which does not satisfy the separate workstreams being assessed.
Question 198
Latency and output quality worsen as unrelated pages consume most input tokens. Which corrective change best addresses the root cause?
- Schedule non-urgent work through bounded batch processing
- Reduce context using retrieval and relevance filtering
- Tune cost using the rate of acceptable outcomes
- Route straightforward requests to a validated economical model
Explanation
Correct Answer
B:Retrieve relevant excerpts and retain enough surrounding evidence for interpretation rather than indiscriminately filling the context window; evidence efficiency tests whether trimming preserves the information that matters; verify the repair using supported-answer rate per input token on labeled audit questions.
Incorrect Answers
A:Batch pipelines are inappropriate where a person needs a subsecond answer for an active decision; use suitable batch orchestration and queue control to trade immediate response time for predictable throughput and expense; that control is assessed by cost and successful throughput per nightly processing window; the incident instead requires supported-answer rate per input token on labeled audit questions.
C:Minimizing token counts without measuring quality can make a workflow more expensive in practice; optimize total cost per successful output with floor constraints on completeness, safety, and latency; that control is assessed by all-in cost per accepted summary at a fixed mandatory-section completion rate; the incident instead requires supported-answer rate per input token on labeled audit questions.
D:Using the cheapest model universally could impair complex or safety-critical cases; evaluate a smaller model on straightforward request strata and escalate only where measurable capability is needed; that control is assessed by accepted resolution rate and cost per case across routed difficulty levels; the incident instead requires supported-answer rate per input token on labeled audit questions.
Question 199
A platform team plans a model revision that advertises better general benchmarks. Before migrating existing typed consumers, what result would prove the release preserves the application's required behavior?
- Macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic
- No contract regressions and acceptable quality/latency deltas on the proposed release
- Per-segment acceptance rate across language, length, and noisy-input strata
- Correct resolution of cross-exhibit conflicts at the required latency and review cost
Explanation
Correct Answer
B:Version-aware tests make improvement claims meaningful for the deployed application; measure no contract regressions and acceptable quality/latency deltas on the proposed release at the release gate; treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability.
Incorrect Answers
A:Joint quality and price measurement prevents choosing a cheap model that fails costly minority categories; compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; this tests macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic; the actual release condition is no contract regressions and acceptable quality/latency deltas on the proposed release.
C:Stratified scores expose failures that an overall aggregate conceals; representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; this tests per-segment acceptance rate across language, length, and noisy-input strata; the actual release condition is no contract regressions and acceptable quality/latency deltas on the proposed release.
D:A targeted complex-case benchmark tests the capability that routine classification metrics miss; escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; this tests correct resolution of cross-exhibit conflicts at the required latency and review cost; the actual release condition is no contract regressions and acceptable quality/latency deltas on the proposed release.
Question 200
A research summarizer may produce numerical findings even when the cited source lacks those numbers. What should the solution architect recommend?
- Validate tool-result provenance and semantics
- Counter automation bias with visible uncertainty and review
- Use claim verification to mitigate confident fabrication
- Threat-model prompt injection across lower-trust inputs
Explanation
Correct Answer
C:Require support from retrieved source passages and provide an explicit unknown path for unsupported claims; grounding tests detect confident fabrication at the statement level; evaluate the design with proportion of numeric claims supported by independently inspected references.
Incorrect Answers
A:Preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; a successful HTTP status does not mean returned data has decision authority; this alternative would measure zero external-result fields promoted to internal approvals in test cases; it does not establish proportion of numeric claims supported by independently inspected references for the described use case.
B:Design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; a nominal human checkpoint is weak when the interface encourages blind confirmation; this alternative would measure reviewer error interception on seeded incorrect recommendations; it does not establish proportion of numeric claims supported by independently inspected references for the described use case.
D:Model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; stronger wording in the system prompt is not a complete defense against an authorized dangerous tool; this alternative would measure unauthorized tool-action success rate over injected emails and documents; it does not establish proportion of numeric claims supported by independently inspected references for the described use case.