Question 550
A case-management office wants to know whether AI assistance reduces its actual workload after corrections and appeals, not just the time to first draft. What result should be reported?
- End-to-end completed cases per paid work-hour with rework and appeal volumes
- Total operational cost divided by approved documents meeting the quality rubric
- Incremental resolution lift compared with a comparable non-assisted cohort
- End-to-end p95 latency and error rate against the agreed service objective
Explanation
Correct Answer
A:Net throughput reflects business output rather than the volume of AI text produced; measure end-to-end completed cases per paid work-hour with rework and appeal volumes at the release gate; measure net time saved after review, appeals, and escalations, segmented by complexity, before claiming productivity.
Incorrect Answers
B:Unit cost tied to accepted results distinguishes economical quality from cheap failed attempts; combine inference, retrieval, tools, review, and rework costs into cost per accepted deliverable; this tests total operational cost divided by approved documents meeting the quality rubric; the actual release condition is end-to-end completed cases per paid work-hour with rework and appeal volumes.
C:A defensible counterfactual is required to credit the architecture with the observed benefit; use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; this tests incremental resolution lift compared with a comparable non-assisted cohort; the actual release condition is end-to-end completed cases per paid work-hour with rework and appeal volumes.
D:Tail measurement detects the bad experiences that averages can obscure; define a p95 or p99 target against real end-to-end requests, including retrieval and tools, rather than reporting an average model time; this tests end-to-end p95 latency and error rate against the agreed service objective; the actual release condition is end-to-end completed cases per paid work-hour with rework and appeal volumes.
Question 551
Relevant documents become difficult to retrieve because stored vectors and new query vectors are not comparable. Which corrective change best addresses the root cause?
- Filter retrieval by access control before model exposure
- Evaluate retrieval quality independently of generation
- Preserve citation metadata through retrieval and synthesis
- Version embeddings and reindex when representation changes
Explanation
Correct Answer
D:Record embedding model and preprocessing versions, re-embed consistently, and use controlled index migration; version-aligned search tests detect vector-compatibility regressions; verify the repair using retrieval recall before and after a complete embedding-index migration.
Incorrect Answers
A:A model instruction to ignore confidential items is not a data-access boundary once the text is supplied; apply document permissions to retrieval candidates before material enters model context or logs; that control is assessed by unauthorized chunk exposure rate under staff permission matrices, targeting zero; the incident instead requires retrieval recall before and after a complete embedding-index migration.
B:A stronger model cannot reliably cite evidence that retrieval never supplies; measure retrieval recall, ranking relevance, and evidence coverage on labeled query-document pairs before optimizing generation; that control is assessed by recall at k and evidence-coverage rate for known-answer queries; the incident instead requires retrieval recall before and after a complete embedding-index migration.
C:A similarity score is not a source identifier and cannot support a traceable citation; store source IDs, locations, dates, and stable links with chunks and propagate them to answer citation validation; that control is assessed by fraction of material claims linked to resolvable source locations; the incident instead requires retrieval recall before and after a complete embedding-index migration.
Question 552
Text embeddings return narrative descriptions of invoices but cannot produce a complete auditable numerical aggregation. Which action best removes the underlying failure mode?
- Use lexical search for exact identifiers and literals
- Rerank candidate passages for question relevance
- Query structured data through authorized SQL or typed APIs
- Combine lexical and semantic candidates when queries mix both
- Use semantic vector search for paraphrased knowledge questions
Explanation
Correct Answer
C:Use a governed structured query interface with parameter checks and business semantics for exact aggregates; a structured oracle detects missing rows and arithmetic mistakes; verify the repair using exact aggregate agreement with a trusted query on authorized records.
Incorrect Answers
A:Semantic similarity alone can rank conceptually related but operationally different identifiers; keyword or lexical matching is effective for codes, exact phrases, and terms whose characters carry the meaning; that control is assessed by exact-ID hit rate at the top retrieval ranks; the incident instead requires exact aggregate agreement with a trusted query on authorized records.
B:A bigger first-stage candidate set without selection can dilute context and increase cost; apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; that control is assessed by relevant-procedure rank and end-to-end answer support rate after reranking; the incident instead requires exact aggregate agreement with a trusted query on authorized records.
D:Either modality alone leaves a known class of mixed-format queries uncovered; hybrid retrieval joins exact-term and semantic evidence, then applies metadata filtering and ranking; that control is assessed by joint exact-identifier accuracy and semantic-symptom recall on mixed queries; the incident instead requires exact aggregate agreement with a trusted query on authorized records.
E:Semantic search is not guaranteed to respect exact version strings, numerals, or access boundaries without additional controls; vector retrieval can match conceptual similarity between user wording and document text, followed by evidence verification; that control is assessed by recall on human-authored paraphrases absent from manual headings; the incident instead requires exact aggregate agreement with a trusted query on authorized records.
Question 553
A team wants Claude to classify short warranty claims into three well-defined categories with complete label descriptions. What action best fits the business and technical boundary?
- Break multi-step reasoning into checkable intermediate outputs
- Add few-shot examples for difficult boundary cases
- Chain prompts when one result is a prerequisite for another
- Start with clear zero-shot instructions for unambiguous tasks
- Use contrastive examples when two outputs look similar
- Separate instructions and source regions with clear tags
Explanation
Correct Answer
D:A concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; controlled comparisons show whether added prompt complexity yields a genuine benefit; evaluate the design with label accuracy and token use versus a heavier example-based prompt.
Incorrect Answers
A:Require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; requesting longer explanations without validation can make unsupported conclusions appear more persuasive; this alternative would measure intermediate artifact validity and final score agreement with a policy-engine oracle; it does not establish label accuracy and token use versus a heavier example-based prompt for the described use case.
B:Provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases; examples are not a substitute for a clean class definition or representative test labels; this alternative would measure recall on previously confused boundary examples outside the prompt set; it does not establish label accuracy and token use versus a heavier example-based prompt for the described use case.
C:Use a staged prompt chain where a verified intermediate classification is required before the next generation step; chaining every trivial substep increases latency without improving reliability for simple independent tasks; this alternative would measure explanations generated only for records that passed the eligibility stage; it does not establish label accuracy and token use versus a heavier example-based prompt for the described use case.
E:Show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; adding examples without highlighting the differentiating condition can reinforce superficial formatting only; this alternative would measure correct type selection on unseen evidence-versus-action pairs; it does not establish label accuracy and token use versus a heavier example-based prompt for the described use case.
F:Use clearly named sections or XML-style tags so Claude can distinguish task policy, input, and supporting documents; tagged presentation cannot replace authenticating a source or enforcing repository access rights; this alternative would measure correct separation of document claims and application instructions in adversarial input tests; it does not establish label accuracy and token use versus a heavier example-based prompt for the described use case.
Question 554
A company maintains hundreds of connectors, most irrelevant to any one employee request. Which choice most directly satisfies the documented goal?
- Preserve stable prefixes while dynamically discovering tools
- Retrieve detailed reference material only when requested
- Use searchable tool discovery for large catalogs
- Fetch detailed schemas only for shortlisted actions
Explanation
Correct Answer
C:Expose a lean search or discovery interface and load relevant detailed tools only after intent is established; discovery quality must preserve necessary capabilities while shrinking unused schema context; evaluate the design with tool-choice accuracy and context-token savings on varied employee requests.
Incorrect Answers
A:Isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary; progressive discovery can lose its token advantage if dynamic tool definitions repeatedly destroy cache reuse; this alternative would measure cache-read tokens and total context cost while changing selected tool subsets; it does not establish tool-choice accuracy and context-token savings on varied employee requests for the described use case.
B:Discover resource summaries first and fetch the specific authoritative documentation section at the point of need; reference discovery without source-version checks can still supply outdated interface information; this alternative would measure correct endpoint contract use with lower documentation-token overhead; it does not establish tool-choice accuracy and context-token savings on varied employee requests for the described use case.
D:Keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; schema-on-demand still needs strict parameter validation at execution time; this alternative would measure schema-token consumption and invalid-argument rate after staged loading; it does not establish tool-choice accuracy and context-token savings on varied employee requests for the described use case.
Question 555
A regional risk service runs three independent screening branches and merges their findings into one approval packet. Chaos tests intentionally delay or fail one branch. What measurement best captures safe orchestration performance?
- Marginal quality improvement per added specialist against its incremental cost and latency
- Verified claim support rate after independent critique on seeded unsupported drafts
- Completed reports with correctly scoped evidence and no cross-specialty data leakage
- p95 end-to-end latency and deterministic merge completeness under one-branch failures
Explanation
Correct Answer
D:This validates concurrency benefits without silently dropping a missing assessment; measure p95 end-to-end latency and deterministic merge completeness under one-branch failures at the release gate; parallel fan-out followed by deterministic aggregation can shorten wall-clock time if each branch has its own contract and errors are handled.
Incorrect Answers
A:Measuring the contribution of each delegate reveals needless orchestration layers; introduce a specialist only when its independent expertise, tool scope, or parallelism beats a simpler call on representative evaluations; this tests marginal quality improvement per added specialist against its incremental cost and latency; the actual release condition is p95 end-to-end latency and deterministic merge completeness under one-branch failures.
B:Targeted seeded errors test whether critique catches defects rather than merely rewriting prose; a distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; this tests verified claim support rate after independent critique on seeded unsupported drafts; the actual release condition is p95 end-to-end latency and deterministic merge completeness under one-branch failures.
C:Routing correctness must include scope enforcement, not just readable final reports; a coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; this tests completed reports with correctly scoped evidence and no cross-specialty data leakage; the actual release condition is p95 end-to-end latency and deterministic merge completeness under one-branch failures.
Question 556
A shared AI assistant performs database actions on behalf of multiple business units. Which option addresses the immediate design need?
- Issue short-lived scoped credentials for tools
- Record the user, tenant, and action in audit trails
- Use delegated user identity for per-user data access
- Verify tenant ownership of resource identifiers
- Require separate approval for high-impact mutations
Explanation
Correct Answer
B:Propagate authenticated actor, tenant, delegated scope, request ID, and action outcome into tamper-resistant audit records; end-to-end attribution makes unauthorized use investigable rather than anonymous; evaluate the design with audit reconstruction of actor, authority, target, and result for sampled mutations.
Incorrect Answers
A:Prefer tightly scoped, expiring credentials with secure rotation and audited issuance for the precise operation; encrypting a shared permanent key at rest does not mitigate broad privilege after the key is exposed; this alternative would measure credential-expiry and scope-denial tests after simulated token compromise; it does not establish audit reconstruction of actor, authority, target, and result for sampled mutations for the described use case.
C:Carry an appropriate user-bound authorization context to downstream services and enforce record-level access per request; application-level authentication alone does not prove permission to read every record returned by a shared account; this alternative would measure zero cross-user CRM record disclosures in permission-matrix tests; it does not establish audit reconstruction of actor, authority, target, and result for sampled mutations for the described use case.
D:Check the requested resource against the authenticated tenant and role at the data service rather than trusting user-provided identifiers; syntactically valid IDs and a logged-in session do not establish ownership of a resource; this alternative would measure denial of cross-tenant identifier enumeration attempts in direct tool tests; it does not establish audit reconstruction of actor, authority, target, and result for sampled mutations for the described use case.
E:Bind high-impact actions to a separately verified approval and narrowly scoped execution credential; a general sign-in session does not imply user consent to every destructive operation in that session; this alternative would measure zero critical network changes without a recorded action-specific approval; it does not establish audit reconstruction of actor, authority, target, and result for sampled mutations for the described use case.
Question 557
A financial FAQ assistant must avoid inventing terms when no authorized source covers a customer's product. Which option addresses the immediate design need?
- Delimit retrieved content as untrusted evidence
- Define an explicit abstention path for insufficient evidence
- Use a machine-readable output contract and validator
- Limit disclosure of secrets and confidential instructions
Explanation
Correct Answer
B:Tell the assistant when to identify missing evidence, ask for clarification, or escalate, and reinforce this with evidence validation; an abstention test verifies whether uncertainty is handled safely without refusing answerable questions; evaluate the design with unsupported-answer rate and appropriate abstention precision on no-answer cases.
Incorrect Answers
A:Clearly separate lower-trust source material from application instructions and enforce permissions at the tool layer as well; delimiters aid instruction hierarchy but do not sanitize malicious destinations or replace output security controls; this alternative would measure prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use; it does not establish unsupported-answer rate and appropriate abstention precision on no-answer cases for the described use case.
C:Request a documented structured format and validate schema, allowed values, and business semantics before downstream use; human-readable examples alone do not guarantee that all runtime outputs satisfy machine contracts; this alternative would measure valid schema rate and zero unauthorized enum values reaching the consumer; it does not establish unsupported-answer rate and appropriate abstention precision on no-answer cases for the described use case.
D:Redact sensitive fields upstream, prohibit secret reproduction in instructions, and screen outbound content before release; a prohibition in the prompt alone cannot make exposed credentials safe or replace upstream data minimization; this alternative would measure secret-exposure rate under seeded credential-bearing diagnostic inputs; it does not establish unsupported-answer rate and appropriate abstention precision on no-answer cases for the described use case.
Question 558
The full document archive overflows the context budget and buries the relevant paragraph. What design correction resolves the reported failure?
- Allocate input and output token budgets per workflow step
- Remove obsolete tool output while preserving current evidence
- Retrieve only relevant material instead of full archives
- Compact extended sessions without discarding commitments
Explanation
Correct Answer
C:Apply query-driven retrieval and evidence selection before prompting Claude, retaining citations and permission filters; relevance per token measures whether context compression preserves the needed evidence; verify the repair using answer support and retrieved-token volume on a sampled policy-question set.
Incorrect Answers
A:Merely expanding every context window can increase cost while leaving output truncation unaddressed; reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; that control is assessed by complete mandatory report sections under the worst supported input size; the incident instead requires answer support and retrieved-token volume on a sampled policy-question set.
B:Pruning without version markers can erase the evidence needed to explain the incident's evolution; prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; that control is assessed by current-state citation accuracy across log-update sequences; the incident instead requires answer support and retrieved-token volume on a sampled policy-question set.
D:Blind summarization can lose critical exceptions unless durable commitments are explicitly represented; store a structured, reviewable summary of decisions, open issues, and provenance; inject it at checkpoints rather than replaying every chat turn; that control is assessed by retention of signed-off requirements after long simulated conversations; the incident instead requires answer support and retrieved-token volume on a sampled policy-question set.
Question 559
A Claude assistant integrates HR records, vector retrieval, external model calls, and ticket updates. Which strategy should be prioritized for this requirement?
- Publish an incident runbook with containment and rollback steps
- Document decisions with assumptions and reversal triggers
- Maintain an end-to-end data-flow and trust-boundary diagram
- Prepare operator onboarding and support handoff materials
Explanation
Correct Answer
C:A current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; architecture diagrams are useful when they expose where protection and authority change; evaluate the design with completeness of sensitive-data flow paths identified during security review.
Incorrect Answers
A:Maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; a high-level diagram alone is not an executable incident response procedure; this alternative would measure successful containment time in a simulated stale-index incident; it does not establish completeness of sensitive-data flow paths identified during security review for the described use case.
B:Record architectural context, alternatives, decision rationale, constraints, and explicit conditions for revisiting choices; configuration exports show what exists but rarely why it was chosen; this alternative would measure ability to identify affected decisions after simulated policy and workload changes; it does not establish completeness of sensitive-data flow paths identified during security review for the described use case.
D:Provide system overview, access setup, troubleshooting flowcharts, monitoring thresholds, and named support responsibilities; source code alone rarely communicates the operational decisions and permission steps operators need; this alternative would measure successful completion of standard support drills by staff who did not build the system; it does not establish completeness of sensitive-data flow paths identified during security review for the described use case.
Question 560
A finance assistant reads sensitive invoices before using a notification service. During a simulated hostile browsing session, which result would prove that private fields cannot be sent to unapproved destinations?
- Measured unauthorized tool-action success rate over injected emails and documents
- Reviewer error interception on seeded incorrect recommendations
- Blocked unauthorized destinations and absence of confidential fields in egress logs
- Maximum resource usage and orderly termination on deliberately unsolvable tasks
Explanation
Correct Answer
C:Network- and tool-level evidence catches leakage not visible in user-facing text; measure blocked unauthorized destinations and absence of confidential fields in egress logs at the release gate; restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails.
Incorrect Answers
A:A realistic attack corpus exercises both model behavior and execution controls; model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; this tests unauthorized tool-action success rate over injected emails and documents; the actual release condition is blocked unauthorized destinations and absence of confidential fields in egress logs.
B:The review test measures meaningful human oversight, not click-through compliance; design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; this tests reviewer error interception on seeded incorrect recommendations; the actual release condition is blocked unauthorized destinations and absence of confidential fields in egress logs.
D:Budget stress tests show whether runaway loops fail safely; set per-task step, token, cost, and elapsed-time budgets with a clear abort and escalation state; this tests maximum resource usage and orderly termination on deliberately unsolvable tasks; the actual release condition is blocked unauthorized destinations and absence of confidential fields in egress logs.