Question 30
A test corpus of only friendly documents gives a false sense of safety before hostile uploads are accepted. What design correction resolves the reported failure?
- Use deterministic code checks for objective output contracts
- Measure human annotator consistency before trusting labels
- Use expert rubrics for nuanced qualitative outputs
- Include hostile prompt-injection and no-answer probes
Explanation
Correct Answer
D:Add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; red-team cases make trust-boundary performance observable; verify the repair using attack success rate and safe handling of intentionally unanswerable prompts.
Incorrect Answers
A:Subjective model grading is unnecessary and less reliable for an objectively checkable parser contract; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; that control is assessed by zero schema violations accepted by the downstream consumer across generated cases; the incident instead requires attack success rate and safe handling of intentionally unanswerable prompts.
B:More test volume does not fix unreliable or undefined labels; use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; that control is assessed by inter-annotator agreement and unresolved-label rate after adjudication; the incident instead requires attack success rate and safe handling of intentionally unanswerable prompts.
C:Pure string comparison is unsuitable when many substantively correct phrasings exist; use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; that control is assessed by expert-rated factual fidelity and calibrated grader agreement on sampled explanations; the incident instead requires attack success rate and safe handling of intentionally unanswerable prompts.
Question 31
A large commerce deployment produces more inference traces than the log pipeline can retain economically. Which intervention is most appropriate for this use case?
- Sample complete traces strategically under heavy traffic
- Separate retrieval, provider, and tool latency budgets
- Correlate model, retrieval, and tool spans in distributed traces
- Segment reliability metrics by tenant and workload
Explanation
Correct Answer
A:Retain error and high-risk traces at a higher rate while sampling successful routine traffic with privacy controls; risk-aware sampling preserves evidence for rare faults without uncontrolled storage growth; evaluate the design with capture rate of seeded critical failures at a bounded telemetry cost.
Incorrect Answers
B:Instrument per-stage timing and compare each segment against its allocated latency budget before changing model configuration; optimizing generation tokens cannot repair a slow connector that dominates the critical path; this alternative would measure stage-specific p95 deltas identifying injected bottlenecks; it does not establish capture rate of seeded critical failures at a bounded telemetry cost for the described use case.
C:End-to-end traces with consistent request and span identifiers identify which service and step consumed time; a model-only latency chart cannot explain retrieval or downstream network bottlenecks; this alternative would measure trace completeness and correct bottleneck attribution on instrumented fault injections; it does not establish capture rate of seeded critical failures at a bounded telemetry cost for the described use case.
D:Break down latency, errors, and tool success by permission-safe tenant and workload dimensions with cardinality controls; overall averages can hide concentrated failures without meaningful dimensional analysis; this alternative would measure detection time for an injected single-tenant degradation; it does not establish capture rate of seeded critical failures at a bounded telemetry cost for the described use case.
Question 32
A coding team needs to invoke reproducible local build and test commands in a sandboxed engineering environment. Which intervention is most appropriate for this use case?
- Use agent-to-agent exchange for independently owned specialist systems
- Use direct APIs for a narrow stable application integration
- Use a controlled CLI interface for developer automation
- Use MCP to expose reusable tools and resources across compatible clients
Explanation
Correct Answer
C:A bounded CLI execution environment can reuse standard developer tools with explicit directory and command permissions; execution tests check both the developer outcome and the sandbox boundary; evaluate the design with reproducible build results with denied commands outside the approved allowlist.
Incorrect Answers
A:An agent-to-agent contract can exchange bounded tasks and outputs while preserving local autonomy, identity, and authority of each specialist system; messaging between agents does not make their outputs trustworthy or authorize cross-system actions automatically; this alternative would measure successful authenticated task exchange with each agent retaining its own permission checks; it does not establish reproducible build results with denied commands outside the approved allowlist for the described use case.
B:A direct typed API client is appropriate when the interface, arguments, and control flow are known and stable; direct API calls offer less automatic discovery when many heterogeneous tools must be used by several assistants; this alternative would measure integration error rate and p95 latency on the fixed status-check path; it does not establish reproducible build results with denied commands outside the approved allowlist for the described use case.
D:A maintained MCP server can standardize discoverable tool and resource contracts while each client still enforces authentication and trust; mCP is an interface protocol, not a substitute for access policy or secure server implementation; this alternative would measure cross-client contract compatibility and unauthorized-action denial for the same MCP service; it does not establish reproducible build results with denied commands outside the approved allowlist for the described use case.
Question 33
A supplier-records assistant is about to gain write access through a controlled service interface. The test plan includes bad field names, unauthorized category values, and interrupted transactions. What evidence best shows safe input handling?
- Repeated and interrupted test runs yielding exactly one committed credit per approved case
- Invalid write payloads rejected before execution, with zero partial supplier mutations
- Fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition
- Percentage of ingested documents meeting schema checks before any Claude call
Explanation
Correct Answer
B:Pre-write enforcement proves that the model cannot bypass the target system's integrity contract; measure invalid write payloads rejected before execution, with zero partial supplier mutations at the release gate; use structured output parsing, semantic validation, and a separate authorized command handler so generated text is never treated directly as a write instruction.
Incorrect Answers
A:Replay tests directly exercise the failure window that typically creates duplicate writes; persist state transitions and use idempotency keys at side-effect boundaries to avoid duplicate transactions across replays; this tests repeated and interrupted test runs yielding exactly one committed credit per approved case; the actual release condition is invalid write payloads rejected before execution, with zero partial supplier mutations.
C:End-to-end provenance lets reviewers investigate disputed outcomes without assuming the final text is self-evidencing; correlation IDs, source versions, and stage logs establish end-to-end lineage across extraction, retrieval, inference, and review; this tests fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition; the actual release condition is invalid write payloads rejected before execution, with zero partial supplier mutations.
D:A schema-conformance gate reveals extraction failures before they can contaminate reasoning; an input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values; this tests percentage of ingested documents meeting schema checks before any Claude call; the actual release condition is invalid write payloads rejected before execution, with zero partial supplier mutations.
Question 34
A safety console has a hard response deadline and uses optional descriptive enrichment. During delayed-service tests, which result shows that the core warning remains timely and accurately labeled?
- Job completion reliability and duplicate-submission rate during slow-system simulations
- Repeated-query latency with zero stale or cross-scope cache responses after updates
- Deadline compliance and correct indication of omitted enrichment
- Accuracy by difficulty segment and unit cost under p95 constraints
- End-to-end p95 improvement with all required sources still present
Explanation
Correct Answer
C:Timed fault injection confirms graceful degradation without inventing evidence; measure deadline compliance and correct indication of omitted enrichment at the release gate; keep the critical verified signal independent and return a clearly limited result when optional enrichment exceeds its budget.
Incorrect Answers
A:Durable job identity decouples user interaction time from background work duration; acknowledge an asynchronous job, persist its state, and let the client poll or receive authorized completion notifications; this tests job completion reliability and duplicate-submission rate during slow-system simulations; the actual release condition is deadline compliance and correct indication of omitted enrichment.
B:Freshness and authorization checks are essential alongside speed gains; use a cache keyed by query, permissions where relevant, and content version, with invalidation tied to source updates; this tests repeated-query latency with zero stale or cross-scope cache responses after updates; the actual release condition is deadline compliance and correct indication of omitted enrichment.
D:Joint segmented reporting reveals whether expensive processing is spent where it helps; use an evaluated routing policy that sends straightforward lookups to a light path and escalates uncertain cases to more capable reasoning; this tests accuracy by difficulty segment and unit cost under p95 constraints; the actual release condition is deadline compliance and correct indication of omitted enrichment.
E:Concurrency wins only if the final synthesis retains complete inputs; execute independent lookups concurrently with bounded timeouts and aggregate the results after all necessary dependencies complete; this tests end-to-end p95 improvement with all required sources still present; the actual release condition is deadline compliance and correct indication of omitted enrichment.
Question 35
A mortgage processing organization is choosing between broad exploratory AI projects and a repetitive matching process with bounded exceptions. At a separate benefits office, managers want model recommendations but must preserve human authority over binding eligibility denials. Which TWO architecture decisions fit the respective needs? Choose TWO.
- Define which decisions remain with people
- Prioritize a repetitive step with bounded exceptions
- Measure the current manual task before choosing automation
- Translate vague accuracy demands into task-specific criteria
- Map available data and its permitted uses before solution selection
Explanation
Correct Answers
A:Explicit decision-rights boundaries distinguish advisory generation from actions requiring a credentialed human or deterministic policy engine; an auditable decision trail demonstrates that AI outputs remained advisory where policy requires it; its corresponding workstream should be validated by share of binding decisions showing the required named approver and rationale.
B:Select a high-frequency, well-described task with measurable work reduction and explicit escalation for ambiguous records; counting both throughput and escalations prevents a fast happy path from hiding a costly exception queue; its corresponding workstream should be validated by accepted cases per reviewer-hour with exception share tracked separately.
Incorrect Answers
C:A baseline separates real business improvement from attractive demos; use completed cases, reviewer effort, errors, and exception handling as measurable comparators; process measurement establishes value, not the identity, data-permission, or runtime architecture of a Claude application; its natural acceptance measure is change in verified minutes and rework per adjudicated claim versus the documented baseline, which does not satisfy the separate workstreams being assessed.
D:Convert vague requests into observable task outcomes, rubric-based quality dimensions, and failure thresholds that stakeholders can agree to test; a clear success rubric evaluates outcomes but cannot on its own implement retrieval access controls; its natural acceptance measure is agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set, which does not satisfy the separate workstreams being assessed.
E:Data-readiness and usage-rights mapping identifies legitimate inputs, owners, quality gaps, and prohibited transfers before committing to an implementation; a data inventory does not alone identify target SLOs or decide whether an orchestration loop is necessary; its natural acceptance measure is percentage of required fields with validated owners, quality checks, and approved processing purposes, which does not satisfy the separate workstreams being assessed.
Question 36
A deductions assistant has split eligibility checks from the client-facing narrative stage. The release test deliberately includes unsupported deductions. Which outcome would establish the prerequisite gate?
- Intermediate artifact validity and final score agreement with a policy-engine oracle
- Correct type selection on unseen evidence-versus-action pairs
- Label accuracy and token use versus a heavier example-based prompt
- Explanations generated only for records that passed the eligibility stage
Explanation
Correct Answer
D:A prerequisite gate catches unvalidated reasoning before downstream text is produced; measure explanations generated only for records that passed the eligibility stage at the release gate; use a staged prompt chain where a verified intermediate classification is required before the next generation step.
Incorrect Answers
A:Checkable milestones provide evidence of reasoning quality beyond rhetorical fluency; require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; this tests intermediate artifact validity and final score agreement with a policy-engine oracle; the actual release condition is explanations generated only for records that passed the eligibility stage.
B:Contrastive tests verify the semantic boundary, not just the presence of a familiar template; show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; this tests correct type selection on unseen evidence-versus-action pairs; the actual release condition is explanations generated only for records that passed the eligibility stage.
C:Controlled comparisons show whether added prompt complexity yields a genuine benefit; a concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; this tests label accuracy and token use versus a heavier example-based prompt; the actual release condition is explanations generated only for records that passed the eligibility stage.
Question 37
A board sponsor needs to choose whether to fund a Claude rollout but receives a dense model-parameter presentation. Which choice most directly satisfies the documented goal?
- Keep a versioned decision and change history
- Present executive decisions through outcomes, cost, and risk
- Record alternatives and rationale in an architecture decision record
- Maintain a risk register with controls and accountable owners
- Demonstrate known limitations alongside successful workflows
Explanation
Correct Answer
B:Summarize business value, alternatives, cost envelope, measured evidence, and requested decisions in concise nontechnical terms; decision-focused review checks whether the audience can act on the information; evaluate the design with sponsor comprehension of expected outcomes, decision options, and risk ownership.
Incorrect Answers
A:Maintain a version-controlled decision log linked to implementation and review milestones, including superseded guidance; chat history and scattered slides rarely provide an authoritative current decision state; this alternative would measure correct retrieval of active decisions and their supersession chain during review; it does not establish sponsor comprehension of expected outcomes, decision options, and risk ownership for the described use case.
C:Use an ADR recording context, considered options, decision, trade-offs, implications, and conditions for revisiting it; a final diagram alone often omits rejected alternatives and the reasoning behind the choice; this alternative would measure reconstructability of key design decisions after a simulated staff handoff; it does not establish sponsor comprehension of expected outcomes, decision options, and risk ownership for the described use case.
D:Record probability and impact where appropriate, control owners, evidence, residual risk, and required approvals; a high-level statement that AI has risks does not create an actionable governance plan; this alternative would measure completion and ownership of high-severity mitigations before the release gate; it does not establish sponsor comprehension of expected outcomes, decision options, and risk ownership for the described use case.
E:Use representative happy-path and failure-path demonstrations with explicit fallback, uncertainty, and human ownership; a demo focused solely on best cases can mask residual risks and create unrealistic expectations; this alternative would measure stakeholder identification of boundaries and fallback responsibilities after the demo; it does not establish sponsor comprehension of expected outcomes, decision options, and risk ownership for the described use case.
Question 38
A developer assistant reports a missing file, but fault injection shows the repository search operation sometimes times out. Which diagnostic outcome would establish that the team identifies the faulty component rather than compensating with a larger model?
- Correct identification of enforced deny source without allowing prohibited commands
- Bug reproduction on sanitized fixtures with no sensitive token disclosure
- Correct root-cause attribution under injected search timeouts
- Time to restore normal operation using an approved previous release
- Reproduction rate of sampled incidents from stored sanitized diagnostic metadata
Explanation
Correct Answer
C:Controlled fault injection tests whether the team diagnoses the failing layer; measure correct root-cause attribution under injected search timeouts at the release gate; classify tool errors separately from model responses, validate return codes, and probe the failing integration before changing prompts.
Incorrect Answers
A:Precedence tests prevent insecure workarounds that bypass governance intent; review organization, project, and local permission layers in the documented precedence order before weakening security controls; this tests correct identification of enforced deny source without allowing prohibited commands; the actual release condition is correct root-cause attribution under injected search timeouts.
B:Safe replay demonstrates that privacy and diagnostic value can coexist; build redacted, access-controlled replay fixtures preserving relevant syntax and error conditions without unnecessary secrets or personal data; this tests bug reproduction on sanitized fixtures with no sensitive token disclosure; the actual release condition is correct root-cause attribution under injected search timeouts.
D:A rollback drill verifies that recovery does not depend on the original developer being online; maintain versioned releases and documented rollback steps for hooks, prompts, model routes, and connector configuration; this tests time to restore normal operation using an approved previous release; the actual release condition is correct root-cause attribution under injected search timeouts.
E:Reproducibility is a prerequisite for a trustworthy fix; collect privacy-safe traces, request IDs, environment details, and exact configuration revisions to establish a reproducible failure case; this tests reproduction rate of sampled incidents from stored sanitized diagnostic metadata; the actual release condition is correct root-cause attribution under injected search timeouts.
Question 39
A university personnel assistant must answer questions from a handbook whose authorized version changes each semester. Another HR team wants a fast, faithful summary of a transcript already supplied in the request, with no external operations. Which TWO patterns suit these different tasks? Choose TWO.
- Use a bounded tool-using agent for exploratory investigation
- Use a deterministic workflow for known ordered stages
- Ground dynamic policy answers with retrieval
- Use a single augmented-LLM call for a bounded text transformation
- Embed a small agent inside a controlled workflow
Explanation
Correct Answers
C:Retrieval from a versioned, access-filtered source grounds answers in current documents and can expose the authoritative citation; version-aware citation checks show whether answers rely on authoritative current evidence; its corresponding workstream should be validated by correct citations to currently effective policy versions across update tests.
D:For a fixed transformation, a prompt enriched with only needed context is simpler, cheaper, and easier to evaluate than an agentic loop; a bounded call should meet the content rubric without introducing an unnecessary action surface; its corresponding workstream should be validated by summary fidelity and p95 completion time without any unapproved tool invocation.
Incorrect Answers
A:Bounded agentic planning chooses follow-up tools from observed evidence while enforcing time, budget, and tool permissions; agent autonomy adds cost and uncertainty to tasks whose steps are already predictable and linear; its natural acceptance measure is resolution of unseen diagnostic branches within a specified tool-call and time budget, which does not satisfy the separate workstreams being assessed.
B:Explicit state transitions preserve required order, testing points, and deterministic branch conditions; Claude can supply judgment inside selected stages; a rigid sequence becomes cumbersome when the next action is genuinely unknown until exploratory results arrive; its natural acceptance measure is rate of recommendations backed by all required completed stage checks, which does not satisfy the separate workstreams being assessed.
E:Place bounded exploratory reasoning inside one stage of a deterministic state machine; enforce approval and write permissions outside the agent; a fully autonomous controller is not needed when most transitions and side effects are governed by policy; its natural acceptance measure is disputed-case resolution with every refund still passing the identity and approval gates, which does not satisfy the separate workstreams being assessed.
Question 40
A staged extraction workflow is upgraded one component at a time while downstream integrations stay on their prior versions. Which measure shows that independent deployments preserve the expected contracts?
- Correct merged clause counts across repeated parallel execution races
- Contract test pass rate for independently upgraded stages and invalid payload rejection
- Stage-specific defect localization and repair without redoing previously validated stages
- Successful recovery after injected worker failures without duplicated writes or stages
Explanation
Correct Answer
B:Versioned interfaces expose breaking changes before a multi-team deployment; measure contract test pass rate for independently upgraded stages and invalid payload rejection at the release gate; typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions.
Incorrect Answers
A:Deterministic merge checks measure integrity rather than only wall-clock speed; isolate independent tasks, retain separate results, then join only when required dependencies have completed; this tests correct merged clause counts across repeated parallel execution races; the actual release condition is contract test pass rate for independently upgraded stages and invalid payload rejection.
C:A traceable artifact boundary confines error analysis to the failed operation; separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries; this tests stage-specific defect localization and repair without redoing previously validated stages; the actual release condition is contract test pass rate for independently upgraded stages and invalid payload rejection.
D:Replay testing verifies the precise recovery behavior the pipeline requires; durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects; this tests successful recovery after injected worker failures without duplicated writes or stages; the actual release condition is contract test pass rate for independently upgraded stages and invalid payload rejection.