Question 1
An archival-document AI project is at the discovery gate, with uncertain scan quality and expensive manual fallback. The steering committee wants evidence that the most consequential unknowns were tested before full funding. Which result should it request?
- Resolution rate of prioritized unknowns through measured prototypes
- Coverage of approved requirements traced to affected stakeholder groups
- Signed acceptance criteria mapped to demonstrable tests and named owners
- Percentage of required sources with approved owners, legal uses, and integration paths
- Verified readiness of high-impact external dependencies against test quotas and outage cases
Explanation
Correct Answer
A:Evidence-based discovery reduces commitment to unverified technical assumptions; measure resolution rate of prioritized unknowns through measured prototypes at the release gate; identify high-risk assumptions and design small feasibility spikes with explicit success and stop criteria.
Incorrect Answers
B:Traceable requirements reveal whether important operational perspectives were omitted; use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders; this tests coverage of approved requirements traced to affected stakeholder groups; the actual release condition is resolution rate of prioritized unknowns through measured prototypes.
C:An explicit agreement limits scope drift and creates a common release standard; document in-scope outcomes, exclusions, metrics, responsibilities, and go-live gates for accountable approval; this tests signed acceptance criteria mapped to demonstrable tests and named owners; the actual release condition is resolution rate of prioritized unknowns through measured prototypes.
D:Data ownership completeness measures feasibility before engineering spends effort; create a source inventory with accountable owners, sensitivity, update rate, and approval dependencies before committing to the design; this tests percentage of required sources with approved owners, legal uses, and integration paths; the actual release condition is resolution rate of prioritized unknowns through measured prototypes.
E:Dependency validation closes gaps between prototype success and operational feasibility; maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; this tests verified readiness of high-impact external dependencies against test quotas and outage cases; the actual release condition is resolution rate of prioritized unknowns through measured prototypes.
Question 2
One team interprets a timeout as a retryable failure while another assumes the write never happened. Which action best removes the underlying failure mode?
- Maintain an end-to-end data-flow and trust-boundary diagram
- Publish an incident runbook with containment and rollback steps
- Version prompts, model routes, tools, and evaluation artifacts together
- Specify tool schemas, errors, and authorization contracts
Explanation
Correct Answer
D:Define typed inputs, outputs, error semantics, idempotency, required scopes, and audit obligations for tool boundaries; contract tests validate the documented behavior under non-happy paths; verify the repair using consumer contract pass rate under timeouts, invalid arguments, and permission denials.
Incorrect Answers
A:A feature list does not reveal crossing of sensitive trust boundaries; a current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; that control is assessed by completeness of sensitive-data flow paths identified during security review; the incident instead requires consumer contract pass rate under timeouts, invalid arguments, and permission denials.
B:A high-level diagram alone is not an executable incident response procedure; maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; that control is assessed by successful containment time in a simulated stale-index incident; the incident instead requires consumer contract pass rate under timeouts, invalid arguments, and permission denials.
C:Recording only the model name omits other behavior-changing dependencies; maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; that control is assessed by reproducibility of a sampled historical answer using its full manifest; the incident instead requires consumer contract pass rate under timeouts, invalid arguments, and permission denials.
Question 3
A contract review needs synthesis across conflicting clauses, exceptions, and multiple document exhibits. What is the most targeted architecture choice?
- Benchmark a smaller model for routine classification
- Select a model path compatible with required modalities
- Evaluate model choice against the real input distribution
- Use stronger reasoning capacity for genuinely complex decisions
Explanation
Correct Answer
D:Escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; a targeted complex-case benchmark tests the capability that routine classification metrics miss; evaluate the design with correct resolution of cross-exhibit conflicts at the required latency and review cost.
Incorrect Answers
A:Compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; a lightweight route should not be assumed suitable for complex multi-step decisions without a representative evaluation; this alternative would measure macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic; it does not establish correct resolution of cross-exhibit conflicts at the required latency and review cost for the described use case.
B:Confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; model selection based solely on text benchmark accuracy does not prove image interpretation reliability; this alternative would measure correct equipment identification and recommendation accuracy on labeled photographs; it does not establish correct resolution of cross-exhibit conflicts at the required latency and review cost for the described use case.
C:Representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; vendor benchmark rankings do not establish suitability for a domain-specific and skewed workload; this alternative would measure per-segment acceptance rate across language, length, and noisy-input strata; it does not establish correct resolution of cross-exhibit conflicts at the required latency and review cost for the described use case.
Question 4
A procurement assessment service now records extracted provisions, applicable exceptions, and the final policy score as separate artifacts. Which evidence would validate the complete reasoning pipeline rather than a convincing narrative alone?
- Explanations generated only for records that passed the eligibility stage
- Recall on previously confused boundary examples outside the prompt set
- Intermediate artifact validity and final score agreement with a policy-engine oracle
- Correct separation of document claims and application instructions in adversarial input tests
Explanation
Correct Answer
C:Checkable milestones provide evidence of reasoning quality beyond rhetorical fluency; measure intermediate artifact validity and final score agreement with a policy-engine oracle at the release gate; require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof.
Incorrect Answers
A:A prerequisite gate catches unvalidated reasoning before downstream text is produced; use a staged prompt chain where a verified intermediate classification is required before the next generation step; this tests explanations generated only for records that passed the eligibility stage; the actual release condition is intermediate artifact validity and final score agreement with a policy-engine oracle.
B:Held-out boundary performance shows whether examples teach the distinction rather than merely echoing training examples; provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases; this tests recall on previously confused boundary examples outside the prompt set; the actual release condition is intermediate artifact validity and final score agreement with a policy-engine oracle.
D:The acceptance set targets role confusion caused by mixed prompt sections; use clearly named sections or XML-style tags so Claude can distinguish task policy, input, and supporting documents; this tests correct separation of document claims and application instructions in adversarial input tests; the actual release condition is intermediate artifact validity and final score agreement with a policy-engine oracle.
Question 5
An archive team classifies millions of historic contract fragments overnight, with results required only the next morning. Which choice most directly satisfies the documented goal?
- Use a bounded tool-using agent for exploratory investigation
- Ground dynamic policy answers with retrieval
- Use a deterministic workflow for known ordered stages
- Use offline batch processing when immediate interactivity is unnecessary
Explanation
Correct Answer
D:Batch-oriented orchestration handles throughput, retries, and cost planning while preserving consistent model instructions for a noninteractive workload; throughput and unit economics measure the benefit that a user-facing latency metric would miss; evaluate the design with completed valid classifications per batch window and cost per accepted record.
Incorrect Answers
A:Bounded agentic planning chooses follow-up tools from observed evidence while enforcing time, budget, and tool permissions; agent autonomy adds cost and uncertainty to tasks whose steps are already predictable and linear; this alternative would measure resolution of unseen diagnostic branches within a specified tool-call and time budget; it does not establish completed valid classifications per batch window and cost per accepted record for the described use case.
B:Retrieval from a versioned, access-filtered source grounds answers in current documents and can expose the authoritative citation; retrieval does not create new authoritative policy; it depends on freshness, correct permissions, and source quality; this alternative would measure correct citations to currently effective policy versions across update tests; it does not establish completed valid classifications per batch window and cost per accepted record for the described use case.
C:Explicit state transitions preserve required order, testing points, and deterministic branch conditions; Claude can supply judgment inside selected stages; a rigid sequence becomes cumbersome when the next action is genuinely unknown until exploratory results arrive; this alternative would measure rate of recommendations backed by all required completed stage checks; it does not establish completed valid classifications per batch window and cost per accepted record for the described use case.
Question 6
A drafting assistant receives negative reviewer feedback but prompt and model changes occur several times per week. Which strategy should be prioritized for this requirement?
- Track safety policy events across product updates
- Associate human corrections with exact release versions
- Alert on unexpected token and cost anomalies
- Instrument end-to-end execution spans for causal debugging
Explanation
Correct Answer
B:Log consented feedback alongside prompt revision, model route, source version, and case context for reproducible analysis; version-linked labels enable targeted, testable corrections; evaluate the design with reproduction of sampled reviewer defects against their original configuration.
Incorrect Answers
A:Monitor both harmful-output violations and false-positive blocks by version and user segment; a falling count of harmful completions can coexist with an unusable overrestrictive assistant; this alternative would measure safety precision/recall and legitimate-task completion before and after the change; it does not establish reproduction of sampled reviewer defects against their original configuration for the described use case.
C:Monitor input/output tokens, cache reads, per-stage costs, and cost-per-task anomalies with accountable ownership; request-volume alerts alone miss a sudden rise in tokens per individual workflow; this alternative would measure detection lag for injected per-request token inflation; it does not establish reproduction of sampled reviewer defects against their original configuration for the described use case.
D:Correlation and span IDs across components allow latency and failure attribution along the actual path; model usage totals do not reveal which integration failed within a particular request; this alternative would measure percent of failures with a complete trace to the responsible stage; it does not establish reproduction of sampled reviewer defects against their original configuration for the described use case.
Question 7
A policy assistant consumes search snippets that may contain adversarial instructions disguised as system messages. Which choice most directly satisfies the documented goal?
- Use a machine-readable output contract and validator
- Delimit retrieved content as untrusted evidence
- Define an explicit abstention path for insufficient evidence
- Write explicit roles, scope, and prohibited operations
Explanation
Correct Answer
B:Clearly separate lower-trust source material from application instructions and enforce permissions at the tool layer as well; adversarial retrieval tests exercise the trust boundary rather than only neutral documents; evaluate the design with prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use.
Incorrect Answers
A:Request a documented structured format and validate schema, allowed values, and business semantics before downstream use; human-readable examples alone do not guarantee that all runtime outputs satisfy machine contracts; this alternative would measure valid schema rate and zero unauthorized enum values reaching the consumer; it does not establish prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use for the described use case.
C:Tell the assistant when to identify missing evidence, ask for clarification, or escalate, and reinforce this with evidence validation; a fallback message cannot repair a poor retrieval index; it only prevents unsupported certainty; this alternative would measure unsupported-answer rate and appropriate abstention precision on no-answer cases; it does not establish prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use for the described use case.
D:Define the assistant's permitted tasks, out-of-scope handling, and escalation behavior in stable application-level instructions, backed by independent access enforcement; role instructions improve behavior but cannot grant or restrict database permissions on their own; this alternative would measure out-of-scope refusal and handoff accuracy on a policy-boundary test set; it does not establish prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use for the described use case.
Question 8
A classifier upgrade changes field semantics and breaks the downstream reviewer without a validation failure. How should the design be corrected?
- Define input and output contracts for each subtask
- Partition research by source and reconcile conflicts explicitly
- Separate planning proposals from privileged execution
- Checkpoint long-running work for replay and recovery
- Divide work along independently verifiable artifacts
Explanation
Correct Answer
A:Typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions; versioned interfaces expose breaking changes before a multi-team deployment; verify the repair using contract test pass rate for independently upgraded stages and invalid payload rejection.
Incorrect Answers
B:Simple partitioning can preserve conflicting claims unless the merge step includes a reconciliation rule; analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; that control is assessed by conflict detection recall and citation coverage across independently processed source sets; the incident instead requires contract test pass rate for independently upgraded stages and invalid payload rejection.
C:Planning/execution separation does not fix whether the recommended plan uses current policy data; use a planning artifact that is reviewed and handed to a distinct authorized executor; never treat proposed tool calls as permission to mutate state; that control is assessed by unauthorized mutations during plan-only tests, with a target of zero; the incident instead requires contract test pass rate for independently upgraded stages and invalid payload rejection.
D:Checkpointing reduces replay cost but cannot correct a defective classification rubric; durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects; that control is assessed by successful recovery after injected worker failures without duplicated writes or stages; the incident instead requires contract test pass rate for independently upgraded stages and invalid payload rejection.
E:Artifact decomposition controls work structure but does not set the organization-wide release SLA; separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries; that control is assessed by stage-specific defect localization and repair without redoing previously validated stages; the incident instead requires contract test pass rate for independently upgraded stages and invalid payload rejection.
Question 9
A travel-booking assistant loses customers during occasional very slow sessions, even though its mean response time looks acceptable. Separately, an invoice-summary product shows low inference charges but requires substantial analyst rework before a report can be accepted. Which TWO measurement practices address the customer experience commitment and true unit economics, respectively? Choose TWO.
- Use a control group to establish incremental business impact
- Measure productivity after correction and escalation
- Translate response-time expectations into a percentile SLO
- Track cost per accepted business outcome
- Choose model and architecture tiers by case impact
Explanation
Correct Answers
C:Define a p95 or p99 target against real end-to-end requests, including retrieval and tools, rather than reporting an average model time; tail measurement detects the bad experiences that averages can obscure; its corresponding workstream should be validated by end-to-end p95 latency and error rate against the agreed service objective.
D:Combine inference, retrieval, tools, review, and rework costs into cost per accepted deliverable; unit cost tied to accepted results distinguishes economical quality from cheap failed attempts; its corresponding workstream should be validated by total operational cost divided by approved documents meeting the quality rubric.
Incorrect Answers
A:Use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; before-after counts without a credible comparison group may misattribute unrelated changes to the model; its natural acceptance measure is incremental resolution lift compared with a comparable non-assisted cohort, which does not satisfy the separate workstreams being assessed.
B:Measure net time saved after review, appeals, and escalations, segmented by complexity, before claiming productivity; counting drafts as completed outcomes overlooks downstream human effort and defect consequences; its natural acceptance measure is end-to-end completed cases per paid work-hour with rework and appeal volumes, which does not satisfy the separate workstreams being assessed.
E:Route by validated task complexity and consequence, using stronger reasoning or human review where evidence justifies the incremental spend; a uniformly premium model can raise cost without proportionate value on simple, low-risk tasks; its natural acceptance measure is quality and total cost segmented by request risk tier under a fixed accuracy floor, which does not satisfy the separate workstreams being assessed.
Question 10
A public-funding proposal writer needs an independent check that every material assertion is supported before publication. In a different workflow, an online returns bot routes simple questions through several expensive specialists without improving outcomes. Which TWO orchestration changes address these distinct problems? Choose TWO.
- Run independent specialists in parallel with a merge contract
- Delegate only specialized subtasks that add measurable value
- Separate independent critique from generation
- Version the handoff artifacts between agents
- Use a coordinator to route distinct specialist tasks
Explanation
Correct Answers
B:Introduce a specialist only when its independent expertise, tool scope, or parallelism beats a simpler call on representative evaluations; measuring the contribution of each delegate reveals needless orchestration layers; its corresponding workstream should be validated by marginal quality improvement per added specialist against its incremental cost and latency.
C:A distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; targeted seeded errors test whether critique catches defects rather than merely rewriting prose; its corresponding workstream should be validated by verified claim support rate after independent critique on seeded unsupported drafts.
Incorrect Answers
A:Parallel fan-out followed by deterministic aggregation can shorten wall-clock time if each branch has its own contract and errors are handled; parallel fan-out is unsafe when tasks mutate the same state or depend on each other's evolving outputs; its natural acceptance measure is p95 end-to-end latency and deterministic merge completeness under one-branch failures, which does not satisfy the separate workstreams being assessed.
D:Immutable, versioned handoff artifacts and typed schemas prevent one agent from consuming ambiguous or mismatched intermediate state; artifact versioning governs exchange correctness, not whether the underlying market facts were researched accurately; its natural acceptance measure is zero stale-artifact merges across concurrent revision and retry scenarios, which does not satisfy the separate workstreams being assessed.
E:A coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; specialist routing adds handoffs and can be wasteful for one small, homogeneous transformation; its natural acceptance measure is completed reports with correctly scoped evidence and no cross-specialty data leakage, which does not satisfy the separate workstreams being assessed.