Question 570
A research workflow permits multiple specialist workers to revise shared findings while some workers retry after failures. Which acceptance outcome demonstrates that the final report cannot incorporate obsolete revisions?
- Verified claim support rate after independent critique on seeded unsupported drafts
- Zero stale-artifact merges across concurrent revision and retry scenarios
- p95 end-to-end latency and deterministic merge completeness under one-branch failures
- Completed reports with correctly scoped evidence and no cross-specialty data leakage
Explanation
Correct Answer
B:Version-aware handoffs can be tested deterministically under racing writes; measure zero stale-artifact merges across concurrent revision and retry scenarios at the release gate; immutable, versioned handoff artifacts and typed schemas prevent one agent from consuming ambiguous or mismatched intermediate state.
Incorrect Answers
A:Targeted seeded errors test whether critique catches defects rather than merely rewriting prose; a distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; this tests verified claim support rate after independent critique on seeded unsupported drafts; the actual release condition is zero stale-artifact merges across concurrent revision and retry scenarios.
C:This validates concurrency benefits without silently dropping a missing assessment; parallel fan-out followed by deterministic aggregation can shorten wall-clock time if each branch has its own contract and errors are handled; this tests p95 end-to-end latency and deterministic merge completeness under one-branch failures; the actual release condition is zero stale-artifact merges across concurrent revision and retry scenarios.
D:Routing correctness must include scope enforcement, not just readable final reports; a coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; this tests completed reports with correctly scoped evidence and no cross-specialty data leakage; the actual release condition is zero stale-artifact merges across concurrent revision and retry scenarios.
Question 571
A legal search returns a useful exception paragraph whose meaning depends on definitions earlier in its parent section. Which intervention is most appropriate for this use case?
- Combine lexical and semantic candidates when queries mix both
- Adopt lexical search for exact identifiers and literals
- Rerank candidate passages for question relevance
- Retrieve parent context after finding a specific chunk
Explanation
Correct Answer
D:Use child-chunk retrieval for recall and fetch validated parent context or adjacent governing sections for interpretation; context reconstruction tests whether narrow hits are interpreted in their governing scope; evaluate the design with conditional-rule accuracy with and without parent-document expansion.
Incorrect Answers
A:Hybrid retrieval joins exact-term and semantic evidence, then applies metadata filtering and ranking; either modality alone leaves a known class of mixed-format queries uncovered; this alternative would measure joint exact-identifier accuracy and semantic-symptom recall on mixed queries; it does not establish conditional-rule accuracy with and without parent-document expansion for the described use case.
B:Keyword or lexical matching is effective for codes, exact phrases, and terms whose characters carry the meaning; semantic similarity alone can rank conceptually related but operationally different identifiers; this alternative would measure exact-ID hit rate at the top retrieval ranks; it does not establish conditional-rule accuracy with and without parent-document expansion for the described use case.
C:Apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; a bigger first-stage candidate set without selection can dilute context and increase cost; this alternative would measure relevant-procedure rank and end-to-end answer support rate after reranking; it does not establish conditional-rule accuracy with and without parent-document expansion for the described use case.
Question 572
A support assistant spawns four subagents for every short refund question and suffers extra latency with no quality lift. Which intervention is most appropriate for this use case?
- Use a coordinator to route distinct specialist tasks
- Separate independent critique from generation
- Version the handoff artifacts between agents
- Delegate only specialized subtasks that add measurable value
Explanation
Correct Answer
D:Introduce a specialist only when its independent expertise, tool scope, or parallelism beats a simpler call on representative evaluations; measuring the contribution of each delegate reveals needless orchestration layers; evaluate the design with marginal quality improvement per added specialist against its incremental cost and latency.
Incorrect Answers
A:A coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; specialist routing adds handoffs and can be wasteful for one small, homogeneous transformation; this alternative would measure completed reports with correctly scoped evidence and no cross-specialty data leakage; it does not establish marginal quality improvement per added specialist against its incremental cost and latency for the described use case.
B:A distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; a critic agent does not repair missing source documents or guarantee that all facts are verifiable; this alternative would measure verified claim support rate after independent critique on seeded unsupported drafts; it does not establish marginal quality improvement per added specialist against its incremental cost and latency for the described use case.
C:Immutable, versioned handoff artifacts and typed schemas prevent one agent from consuming ambiguous or mismatched intermediate state; artifact versioning governs exchange correctness, not whether the underlying market facts were researched accurately; this alternative would measure zero stale-artifact merges across concurrent revision and retry scenarios; it does not establish marginal quality improvement per added specialist against its incremental cost and latency for the described use case.
Question 573
An email-triage application has labeled categories, short inputs, and a strict high-volume cost ceiling. What is the most defensible next step?
- Use stronger reasoning capacity for genuinely complex decisions
- Use confidence-aware routing with a bounded fallback
- Evaluate model choice against the real input distribution
- Gate model-version updates on regression tests
- Benchmark a smaller model for routine classification
Explanation
Correct Answer
E:Compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; joint quality and price measurement prevents choosing a cheap model that fails costly minority categories; evaluate the design with macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic.
Incorrect Answers
A:Escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; high reasoning capacity is wasteful if the task only extracts a known field or matches a stable label; this alternative would measure correct resolution of cross-exhibit conflicts at the required latency and review cost; it does not establish macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic for the described use case.
B:Combine an evaluated uncertainty trigger with escalation to a stronger model or reviewer, using explicit limits and post-route quality measurement; blindly using one model tier on all cases neither controls the hardest error modes nor optimizes cost; this alternative would measure high-risk escalation recall and total cost under the required error ceiling; it does not establish macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic for the described use case.
C:Representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; vendor benchmark rankings do not establish suitability for a domain-specific and skewed workload; this alternative would measure per-segment acceptance rate across language, length, and noisy-input strata; it does not establish macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic for the described use case.
D:Treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; newer model branding alone cannot ensure identical tool-use or output behavior in an existing workflow; this alternative would measure no contract regressions and acceptable quality/latency deltas on the proposed release; it does not establish macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic for the described use case.
Question 574
A health information assistant is evaluated on clinically equivalent paraphrases and misleading but polished explanations. Which assessment would detect factual nuance without overrewarding exact wording?
- Measured holdout quality stability across successive prompt versions
- Inter-annotator agreement and unresolved-label rate after adjudication
- Per-stratum pass rates with no uncovered critical scenario classes
- Expert-rated factual fidelity and calibrated grader agreement on sampled explanations
Explanation
Correct Answer
D:A semantic rubric tests meaning while allowing legitimate variation in language; measure expert-rated factual fidelity and calibrated grader agreement on sampled explanations at the release gate; use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness.
Incorrect Answers
A:Untouched data estimates whether improvements transfer beyond the examples used during tuning; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; this tests holdout quality stability across successive prompt versions; the actual release condition is expert-rated factual fidelity and calibrated grader agreement on sampled explanations.
B:Label reliability is necessary before model score differences can be interpreted; use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; this tests inter-annotator agreement and unresolved-label rate after adjudication; the actual release condition is expert-rated factual fidelity and calibrated grader agreement on sampled explanations.
C:Stratified coverage prevents abundant easy cases from hiding unsupported workflows; build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; this tests per-stratum pass rates with no uncovered critical scenario classes; the actual release condition is expert-rated factual fidelity and calibrated grader agreement on sampled explanations.
Question 575
A product-planning team runs researcher, analyst, and writer agents that share mutable intermediate documents. What should the solution architect recommend?
- Delegate only specialized subtasks that add measurable value
- Adopt a coordinator to route distinct specialist tasks
- Version the handoff artifacts between agents
- Separate independent critique from generation
Explanation
Correct Answer
C:Immutable, versioned handoff artifacts and typed schemas prevent one agent from consuming ambiguous or mismatched intermediate state; version-aware handoffs can be tested deterministically under racing writes; evaluate the design with zero stale-artifact merges across concurrent revision and retry scenarios.
Incorrect Answers
A:Introduce a specialist only when its independent expertise, tool scope, or parallelism beats a simpler call on representative evaluations; more agents by themselves do not provide additional knowledge, authority, or better outcomes; this alternative would measure marginal quality improvement per added specialist against its incremental cost and latency; it does not establish zero stale-artifact merges across concurrent revision and retry scenarios for the described use case.
B:A coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; specialist routing adds handoffs and can be wasteful for one small, homogeneous transformation; this alternative would measure completed reports with correctly scoped evidence and no cross-specialty data leakage; it does not establish zero stale-artifact merges across concurrent revision and retry scenarios for the described use case.
D:A distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; a critic agent does not repair missing source documents or guarantee that all facts are verifiable; this alternative would measure verified claim support rate after independent critique on seeded unsupported drafts; it does not establish zero stale-artifact merges across concurrent revision and retry scenarios for the described use case.
Question 576
The assistant asks for approval again or acts without a record after the conversational buffer is truncated. How should the design be corrected?
- Persist operational decisions outside transient chat history
- Attach provenance metadata to selected context
- Compact extended sessions without discarding commitments
- Allocate input and output token budgets per workflow step
- Remove obsolete tool output while preserving current evidence
Explanation
Correct Answer
A:Store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger; durable-state testing proves behavior survives context eviction; verify the repair using correct recovery of approvals and pending actions after forced session reset.
Incorrect Answers
B:Provenance metadata does not guarantee the upstream search selected the right document in the first place; supply source identity, version, effective date, and retrieval relevance alongside each chunk so the model can attribute evidence correctly; that control is assessed by citation correctness by effective-date version in temporal regression questions; the incident instead requires correct recovery of approvals and pending actions after forced session reset.
C:Blind summarization can lose critical exceptions unless durable commitments are explicitly represented; store a structured, reviewable summary of decisions, open issues, and provenance; inject it at checkpoints rather than replaying every chat turn; that control is assessed by retention of signed-off requirements after long simulated conversations; the incident instead requires correct recovery of approvals and pending actions after forced session reset.
D:Merely expanding every context window can increase cost while leaving output truncation unaddressed; reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; that control is assessed by complete mandatory report sections under the worst supported input size; the incident instead requires correct recovery of approvals and pending actions after forced session reset.
E:Pruning without version markers can erase the evidence needed to explain the incident's evolution; prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; that control is assessed by current-state citation accuracy across log-update sequences; the incident instead requires correct recovery of approvals and pending actions after forced session reset.
Question 577
A knowledge assistant retrieves many related articles but first-pass results often place the right paragraph just outside the displayed context. What is the most defensible next step?
- Move slow operations into asynchronous jobs with progress states
- Rerank a short candidate set for ambiguous retrieval queries
- Parallelize independent read-only tool calls
- Cache safe, version-keyed retrieval responses
Explanation
Correct Answer
B:Retrieve a bounded candidate set and rerank against the actual question to concentrate useful evidence without feeding all candidates to the model; a quality-latency frontier shows whether an extra ranking pass is worth its overhead; evaluate the design with grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths.
Incorrect Answers
A:Acknowledge an asynchronous job, persist its state, and let the client poll or receive authorized completion notifications; a longer synchronous timeout would still tie up request resources and expose users to unreliable waits; this alternative would measure job completion reliability and duplicate-submission rate during slow-system simulations; it does not establish grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths for the described use case.
C:Execute independent lookups concurrently with bounded timeouts and aggregate the results after all necessary dependencies complete; parallel execution offers no benefit when one call requires another's output or modifies shared state; this alternative would measure end-to-end p95 improvement with all required sources still present; it does not establish grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths for the described use case.
D:Use a cache keyed by query, permissions where relevant, and content version, with invalidation tied to source updates; an unversioned cache can improve speed while silently sacrificing freshness and access correctness; this alternative would measure repeated-query latency with zero stale or cross-scope cache responses after updates; it does not establish grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths for the described use case.
Question 578
A multi-turn service keeps shared instructions stable while choosing different connector subsets as needed. Which workload result would show that dynamic tool selection still preserves economical prefix caching?
- Measured schema-token consumption and invalid-argument rate after staged loading
- Irrelevant tool exposure per intent with no loss of required task completion
- Cache-read tokens and total context cost while changing selected tool subsets
- Correct endpoint contract use with lower documentation-token overhead
- Intent-classification accuracy before any tool disclosure or call
Explanation
Correct Answer
C:The combined test reveals interaction between discovery strategy and prompt-prefix economics; measure cache-read tokens and total context cost while changing selected tool subsets at the release gate; isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary.
Incorrect Answers
A:The test balances context savings against input-contract reliability; keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; this tests schema-token consumption and invalid-argument rate after staged loading; the actual release condition is cache-read tokens and total context cost while changing selected tool subsets.
B:Exposure and completion metrics together reveal whether dynamic loading is too aggressive; select a bounded task-specific capability group after intent routing with conservative fallback for ambiguous requests; this tests irrelevant tool exposure per intent with no loss of required task completion; the actual release condition is cache-read tokens and total context cost while changing selected tool subsets.
D:Targeted documentation fetches should improve focus without losing necessary details; discover resource summaries first and fetch the specific authoritative documentation section at the point of need; this tests correct endpoint contract use with lower documentation-token overhead; the actual release condition is cache-read tokens and total context cost while changing selected tool subsets.
E:Phase separation verifies that the assistant understands the task before acting; separate goal interpretation and policy decisions from a downstream staged capability-discovery step; this tests intent-classification accuracy before any tool disclosure or call; the actual release condition is cache-read tokens and total context cost while changing selected tool subsets.
Question 579
A warranty-processing pipeline has introduced enforced checkpoints before the recommendation stage. Reviewers inject records with missing purchase dates and mismatched serial numbers. Which result verifies the sequence is respected?
- Rate of recommendations backed by all required completed stage checks
- Resolution of unseen diagnostic branches within a specified tool-call and time budget
- Completed valid classifications per batch window and cost per accepted record
- Disputed-case resolution with every refund still passing the identity and approval gates
Explanation
Correct Answer
A:Stage-completion evidence verifies that mandatory controls were not left to agent discretion; measure rate of recommendations backed by all required completed stage checks at the release gate; explicit state transitions preserve required order, testing points, and deterministic branch conditions; Claude can supply judgment inside selected stages.
Incorrect Answers
B:The agent must demonstrate flexible discovery without escaping its operational guardrails; bounded agentic planning chooses follow-up tools from observed evidence while enforcing time, budget, and tool permissions; this tests resolution of unseen diagnostic branches within a specified tool-call and time budget; the actual release condition is rate of recommendations backed by all required completed stage checks.
C:Throughput and unit economics measure the benefit that a user-facing latency metric would miss; batch-oriented orchestration handles throughput, retries, and cost planning while preserving consistent model instructions for a noninteractive workload; this tests completed valid classifications per batch window and cost per accepted record; the actual release condition is rate of recommendations backed by all required completed stage checks.
D:This tests flexible investigation and deterministic control at the same time; place bounded exploratory reasoning inside one stage of a deterministic state machine; enforce approval and write permissions outside the agent; this tests disputed-case resolution with every refund still passing the identity and approval gates; the actual release condition is rate of recommendations backed by all required completed stage checks.
Question 580
A medical-answer evaluation service shares more patient information with external reviewers than they need. A claims advisor logs approval status but discards specialists' correction reasons. A legal compliance assistant encounters disputed interpretations that could carry binding consequences. Which THREE oversight controls address these separate risks? Choose THREE.
- Require stronger approval paths for exceptional high-impact actions
- Give reviewers only necessary sensitive information
- Use risk-based sampling for low-impact decisions
- Require named authorization before moving money
- Escalate binding legal interpretations to qualified reviewers
- Capture reviewer overrides as structured improvement signals
Explanation
Correct Answers
B:Minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; review workflows should preserve decision quality while protecting privacy; its corresponding workstream should be validated by unauthorized field exposure and successful review accuracy after data minimization.
E:Define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; a consequence-aware test checks whether unsupported authoritative decisions are avoided; its corresponding workstream should be validated by correct escalation of ambiguous legal cases with reviewer attribution.
F:Record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; structured override trends reveal whether feedback improves the system over time; its corresponding workstream should be validated by recurrence rate of categorized, previously corrected error types.
Incorrect Answers
A:Apply risk-tiered approvals, including dual control or independent sign-off where policy demands, to exceptionally consequential mutations; a uniform low-friction approval does not reflect differing blast radius and operational risk; its natural acceptance measure is critical-action dual-approval compliance and zero bypasses during staged tests, which does not satisfy the separate workstreams being assessed.
C:Risk-tiered sampling can reduce review load for reversible low-impact outputs, but the three issues here concern unnecessary patient-data disclosure to reviewers, unrecorded expert corrections, and legally ambiguous cases requiring qualified escalation.
D:Separate recommendation from execution and require an authenticated human approval bound to the exact amount, payee, and transaction; general user authentication does not confer approval for a specific financial transfer; its natural acceptance measure is zero disbursements without an auditable transaction-specific approver, which does not satisfy the separate workstreams being assessed.