Question 390
Human sign-off remains present on paper, but audits show reviewers never inspect contradictory source details. How should the design be corrected?
- Validate tool-result provenance and semantics
- Counter automation bias with visible uncertainty and review
- Constrain outbound channels to prevent data exfiltration
- Threat-model prompt injection across lower-trust inputs
Explanation
Correct Answer
B:Design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; the review test measures meaningful human oversight, not click-through compliance; verify the repair using reviewer error interception on seeded incorrect recommendations.
Incorrect Answers
A:A successful HTTP status does not mean returned data has decision authority; preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; that control is assessed by zero external-result fields promoted to internal approvals in test cases; the incident instead requires reviewer error interception on seeded incorrect recommendations.
C:Redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; that control is assessed by blocked unauthorized destinations and absence of confidential fields in egress logs; the incident instead requires reviewer error interception on seeded incorrect recommendations.
D:Stronger wording in the system prompt is not a complete defense against an authorized dangerous tool; model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; that control is assessed by unauthorized tool-action success rate over injected emails and documents; the incident instead requires reviewer error interception on seeded incorrect recommendations.
Question 391
An operations connector has changed to time-bounded, narrowly scoped credentials. The security drill exposes a test token and attempts unauthorized access after expiry. Which observation demonstrates reduced credential impact?
- Zero cross-user CRM record disclosures in permission-matrix tests
- Denial of cross-tenant identifier enumeration attempts in direct tool tests
- Credential-expiry and scope-denial tests after simulated token compromise
- Zero critical network changes without a recorded action-specific approval
Explanation
Correct Answer
C:Short validity and minimum scope bound the impact of a leaked credential; measure credential-expiry and scope-denial tests after simulated token compromise at the release gate; prefer tightly scoped, expiring credentials with secure rotation and audited issuance for the precise operation.
Incorrect Answers
A:User-context tests verify enforcement of the source system's individual access rules; carry an appropriate user-bound authorization context to downstream services and enforce record-level access per request; this tests zero cross-user CRM record disclosures in permission-matrix tests; the actual release condition is credential-expiry and scope-denial tests after simulated token compromise.
B:Object-level access checks close the horizontal privilege gap that prompt filtering misses; check the requested resource against the authenticated tenant and role at the data service rather than trusting user-provided identifiers; this tests denial of cross-tenant identifier enumeration attempts in direct tool tests; the actual release condition is credential-expiry and scope-denial tests after simulated token compromise.
D:Approval-path tests ensure the precise mutation was authorized before execution; bind high-impact actions to a separately verified approval and narrowly scoped execution credential; this tests zero critical network changes without a recorded action-specific approval; the actual release condition is credential-expiry and scope-denial tests after simulated token compromise.
Question 392
The true troubleshooting procedure appears in candidate results but outside the set sent to the model. What design correction resolves the reported failure?
- Rerank candidate passages for question relevance
- Retrieve parent context after finding a specific chunk
- Query structured data through authorized SQL or typed APIs
- Combine lexical and semantic candidates when queries mix both
Explanation
Correct Answer
A:Apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; rank-sensitive evaluation tests whether useful evidence reaches the final context window; verify the repair using relevant-procedure rank and end-to-end answer support rate after reranking.
Incorrect Answers
B:Passing isolated small chunks may preserve relevance while losing the context needed for a correct conclusion; use child-chunk retrieval for recall and fetch validated parent context or adjacent governing sections for interpretation; that control is assessed by conditional-rule accuracy with and without parent-document expansion; the incident instead requires relevant-procedure rank and end-to-end answer support rate after reranking.
C:Unstructured document retrieval cannot ensure completeness or mathematical correctness for database totals; use a governed structured query interface with parameter checks and business semantics for exact aggregates; that control is assessed by exact aggregate agreement with a trusted query on authorized records; the incident instead requires relevant-procedure rank and end-to-end answer support rate after reranking.
D:Either modality alone leaves a known class of mixed-format queries uncovered; hybrid retrieval joins exact-term and semantic evidence, then applies metadata filtering and ranking; that control is assessed by joint exact-identifier accuracy and semantic-symptom recall on mixed queries; the incident instead requires relevant-procedure rank and end-to-end answer support rate after reranking.
Question 393
A workplace safety classifier is evaluated on new descriptions that lie close to the boundary between an injury and a near miss. Which result would prove that its targeted examples generalize beyond the examples themselves?
- Correct separation of document claims and application instructions in adversarial input tests
- Recall on previously confused boundary examples outside the prompt set
- Label accuracy and token use versus a heavier example-based prompt
- Intermediate artifact validity and final score agreement with a policy-engine oracle
Explanation
Correct Answer
B:Held-out boundary performance shows whether examples teach the distinction rather than merely echoing training examples; measure recall on previously confused boundary examples outside the prompt set at the release gate; provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases.
Incorrect Answers
A:The acceptance set targets role confusion caused by mixed prompt sections; use clearly named sections or XML-style tags so Claude can distinguish task policy, input, and supporting documents; this tests correct separation of document claims and application instructions in adversarial input tests; the actual release condition is recall on previously confused boundary examples outside the prompt set.
C:Controlled comparisons show whether added prompt complexity yields a genuine benefit; a concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; this tests label accuracy and token use versus a heavier example-based prompt; the actual release condition is recall on previously confused boundary examples outside the prompt set.
D:Checkable milestones provide evidence of reasoning quality beyond rhetorical fluency; require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; this tests intermediate artifact validity and final score agreement with a policy-engine oracle; the actual release condition is recall on previously confused boundary examples outside the prompt set.
Question 394
A prompt change has been deployed to a small cohort with a rollback option. Which monitoring result would establish that new hallucination failures are attributable to that version rather than concealed in aggregate data?
- Detection of version-specific quality regressions in a staged rollout
- Safety precision/recall and legitimate-task completion before and after the change
- Detection lag for injected per-request token inflation
- Reproduction of sampled reviewer defects against their original configuration
- Quality delta between recent audited production samples and the launch baseline
Explanation
Correct Answer
A:Release segmentation isolates changes before broad impact; measure detection of version-specific quality regressions in a staged rollout at the release gate; partition traces and scored outcomes by deployed configuration, using release identifiers to support rollback and diagnosis.
Incorrect Answers
B:Paired monitoring distinguishes beneficial safety controls from excessive refusals; monitor both harmful-output violations and false-positive blocks by version and user segment; this tests safety precision/recall and legitimate-task completion before and after the change; the actual release condition is detection of version-specific quality regressions in a staged rollout.
C:Unit-level telemetry catches runaway expense before invoices provide delayed evidence; monitor input/output tokens, cache reads, per-stage costs, and cost-per-task anomalies with accountable ownership; this tests detection lag for injected per-request token inflation; the actual release condition is detection of version-specific quality regressions in a staged rollout.
D:Version-linked labels enable targeted, testable corrections; log consented feedback alongside prompt revision, model route, source version, and case context for reproducible analysis; this tests reproduction of sampled reviewer defects against their original configuration; the actual release condition is detection of version-specific quality regressions in a staged rollout.
E:Live sampling reveals degradation that unchanged historical tests cannot see; periodically sample representative production outputs for labeling, segment analysis, and drift signals without retaining unnecessary personal data; this tests quality delta between recent audited production samples and the launch baseline; the actual release condition is detection of version-specific quality regressions in a staged rollout.
Question 395
Two business units operate separate agents with their own tools and approval processes but need to exchange verified task artifacts. What action best fits the business and technical boundary?
- Use direct APIs for a narrow stable application integration
- Use agent-to-agent exchange for independently owned specialist systems
- Use MCP to expose reusable tools and resources across compatible clients
- Prefer managed server-side tools when appropriate to the platform
Explanation
Correct Answer
B:An agent-to-agent contract can exchange bounded tasks and outputs while preserving local autonomy, identity, and authority of each specialist system; boundary tests verify interoperability without merging authorities; evaluate the design with successful authenticated task exchange with each agent retaining its own permission checks.
Incorrect Answers
A:A direct typed API client is appropriate when the interface, arguments, and control flow are known and stable; direct API calls offer less automatic discovery when many heterogeneous tools must be used by several assistants; this alternative would measure integration error rate and p95 latency on the fixed status-check path; it does not establish successful authenticated task exchange with each agent retaining its own permission checks for the described use case.
C:A maintained MCP server can standardize discoverable tool and resource contracts while each client still enforces authentication and trust; mCP is an interface protocol, not a substitute for access policy or secure server implementation; this alternative would measure cross-client contract compatibility and unauthorized-action denial for the same MCP service; it does not establish successful authenticated task exchange with each agent retaining its own permission checks for the described use case.
D:A supported server-executed tool can offload execution to the provider while the application retains usage, privacy, and result-validation responsibilities; hosted tools do not automatically make external search content authoritative or safe from prompt injection; this alternative would measure search coverage, tool cost, and provenance checks with no application-owned execution endpoint; it does not establish successful authenticated task exchange with each agent retaining its own permission checks for the described use case.
Question 396
A settlement-drafting assistant can calculate proposed reimbursement amounts but cannot release funds. Transaction tests include attempts to bypass named approvers. Which result proves that the payment-control gate works?
- Measured correct escalation of ambiguous legal cases with reviewer attribution
- Defect-detection rate and reviewer workload under a risk-tiered sampling policy
- Recurrence rate of categorized, previously corrected error types
- Unauthorized field exposure and successful review accuracy after data minimization
- Zero disbursements without an auditable transaction-specific approver
Explanation
Correct Answer
E:Payment simulation tests prove approvals are bound to the exact side effect; measure zero disbursements without an auditable transaction-specific approver at the release gate; separate recommendation from execution and require an authenticated human approval bound to the exact amount, payee, and transaction.
Incorrect Answers
A:A consequence-aware test checks whether unsupported authoritative decisions are avoided; define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; this tests correct escalation of ambiguous legal cases with reviewer attribution; the actual release condition is zero disbursements without an auditable transaction-specific approver.
B:Sampling effectiveness must balance residual errors and human capacity; combine automated checks with representative human sampling and exception-based review scaled to risk; this tests defect-detection rate and reviewer workload under a risk-tiered sampling policy; the actual release condition is zero disbursements without an auditable transaction-specific approver.
C:Structured override trends reveal whether feedback improves the system over time; record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; this tests recurrence rate of categorized, previously corrected error types; the actual release condition is zero disbursements without an auditable transaction-specific approver.
D:Review workflows should preserve decision quality while protecting privacy; minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; this tests unauthorized field exposure and successful review accuracy after data minimization; the actual release condition is zero disbursements without an auditable transaction-specific approver.
Question 397
A company is considering an expensive enterprise-wide AI platform before confirming that the main use case has usable, authorized data. Another group must retire a widely called agent API without unexpectedly breaking dependent services. Which TWO lifecycle controls should govern the projects? Choose TWO.
- Use bounded pilots with exit criteria before expansion
- Set an evidence gate before moving beyond discovery
- Transfer durable ownership and operations before handoff
- Include security and governance checks during design
- Plan deprecation and rollback from the start
Explanation
Correct Answers
B:Require validated problem scope, value hypothesis, data authority, and feasibility evidence before full design commitment; stage gates prevent costly commitment to untested assumptions; its corresponding workstream should be validated by closure of high-risk feasibility questions before design approval.
E:Inventory consumers, define migration dates and fallback, preserve data obligations, and test the rollback or retirement path; dependency-aware migration tests verify that lifecycle closure is complete; its corresponding workstream should be validated by successful retirement with zero unplanned dependent-system failures.
Incorrect Answers
A:Define pilot size, critical acceptance metrics, escalation paths, and scale-up criteria before broad deployment; more launch enthusiasm does not replace production-like testing under load and edge conditions; its natural acceptance measure is pilot gate pass rate with defined performance and safety thresholds, which does not satisfy the separate workstreams being assessed.
C:Establish production owners for reliability, security, spend, data, prompts, and escalation, with verified access and runbooks; a completed repository is not proof of ongoing service operability; its natural acceptance measure is named owner coverage and support-drill completion at handoff, which does not satisfy the separate workstreams being assessed.
D:Incorporate threat modeling, authorization, data processing, and human oversight into architecture acceptance before implementation hardens; post-build security sign-off is costly and may not undo prior unauthorized processing; its natural acceptance measure is resolution of high-severity architecture risks before implementation approval, which does not satisfy the separate workstreams being assessed.
Question 398
A finance copilot promises auditable quarter-by-quarter totals from a filtered ledger. Which acceptance result would demonstrate that its numeric output comes from complete, authorized structured data rather than narrative similarity?
- Exact aggregate agreement with a trusted query on authorized records
- Relevant-procedure rank and end-to-end answer support rate after reranking
- Exact-ID hit rate at the top retrieval ranks
- Recall on human-authored paraphrases absent from manual headings
Explanation
Correct Answer
A:A structured oracle detects missing rows and arithmetic mistakes; measure exact aggregate agreement with a trusted query on authorized records at the release gate; use a governed structured query interface with parameter checks and business semantics for exact aggregates.
Incorrect Answers
B:Rank-sensitive evaluation tests whether useful evidence reaches the final context window; apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; this tests relevant-procedure rank and end-to-end answer support rate after reranking; the actual release condition is exact aggregate agreement with a trusted query on authorized records.
C:Identifier-focused tests reveal errors that a semantic relevance score may ignore; keyword or lexical matching is effective for codes, exact phrases, and terms whose characters carry the meaning; this tests exact-ID hit rate at the top retrieval ranks; the actual release condition is exact aggregate agreement with a trusted query on authorized records.
D:Paraphrase benchmark cases test the capability for which embeddings are chosen; vector retrieval can match conceptual similarity between user wording and document text, followed by evidence verification; this tests recall on human-authored paraphrases absent from manual headings; the actual release condition is exact aggregate agreement with a trusted query on authorized records.
Question 399
A commerce agent initially exposes only the capability family relevant to the user's intent. Test traffic alternates between catalog, returns, and orders. Which metric would show that unrelated actions are hidden without harming completion?
- Irrelevant tool exposure per intent with no loss of required task completion
- Schema-token consumption and invalid-argument rate after staged loading
- Cache-read tokens and total context cost while changing selected tool subsets
- Correct endpoint contract use with lower documentation-token overhead
- Intent-classification accuracy before any tool disclosure or call
Explanation
Correct Answer
A:Exposure and completion metrics together reveal whether dynamic loading is too aggressive; measure irrelevant tool exposure per intent with no loss of required task completion at the release gate; select a bounded task-specific capability group after intent routing with conservative fallback for ambiguous requests.
Incorrect Answers
B:The test balances context savings against input-contract reliability; keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; this tests schema-token consumption and invalid-argument rate after staged loading; the actual release condition is irrelevant tool exposure per intent with no loss of required task completion.
C:The combined test reveals interaction between discovery strategy and prompt-prefix economics; isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary; this tests cache-read tokens and total context cost while changing selected tool subsets; the actual release condition is irrelevant tool exposure per intent with no loss of required task completion.
D:Targeted documentation fetches should improve focus without losing necessary details; discover resource summaries first and fetch the specific authoritative documentation section at the point of need; this tests correct endpoint contract use with lower documentation-token overhead; the actual release condition is irrelevant tool exposure per intent with no loss of required task completion.
E:Phase separation verifies that the assistant understands the task before acting; separate goal interpretation and policy decisions from a downstream staged capability-discovery step; this tests intent-classification accuracy before any tool disclosure or call; the actual release condition is irrelevant tool exposure per intent with no loss of required task completion.
Question 400
A user bypasses a prompt-only restriction and submits an unauthorized tool request. How should the design be corrected?
- Handle retrieved instructions as untrusted data
- Allowlist generated fields and destinations before external calls
- Screen unsafe outputs in proportion to the use case
- Apply controls at input, tool execution, and output boundaries
Explanation
Correct Answer
D:Layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority; end-to-end adversarial cases assess the combined defenses and their failure isolation; verify the repair using denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
Incorrect Answers
A:Assuming retrieved text is authoritative because it came from a connector opens a prompt-injection path; keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; that control is assessed by attack success rate on adversarial web pages and zero unauthorized egress calls; the incident instead requires denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
B:Formatting the request as JSON does not make arbitrary field values safe to transmit; validate destinations, argument types, sensitive fields, and authorization before dispatch; reject values outside the approved contract; that control is assessed by zero unapproved recipients or fields accepted by the outbound service; the incident instead requires denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
C:An input-only filter cannot detect every unsafe completion, especially when harm arises from the generated detail; evaluate output safety against a documented risk rubric and use appropriate escalation or refusal for high-risk claims; that control is assessed by harmful-completion detection on labeled outputs with false-positive monitoring; the incident instead requires denial of unauthorized actions and harmful outputs across multi-stage red-team paths.