Question 430
A legal-review workflow is tested with agreements whose risk hinges on exceptions spread over several exhibits. Before changing its reasoning-model route, which evidence would demonstrate that the selected capacity handles those dependencies?
- Correct equipment identification and recommendation accuracy on labeled photographs
- Per-segment acceptance rate across language, length, and noisy-input strata
- No contract regressions and acceptable quality/latency deltas on the proposed release
- Correct resolution of cross-exhibit conflicts at the required latency and review cost
Explanation
Correct Answer
D:A targeted complex-case benchmark tests the capability that routine classification metrics miss; measure correct resolution of cross-exhibit conflicts at the required latency and review cost at the release gate; escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies.
Incorrect Answers
A:A multimodal acceptance set measures the capability absent from text-only trials; confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; this tests correct equipment identification and recommendation accuracy on labeled photographs; the actual release condition is correct resolution of cross-exhibit conflicts at the required latency and review cost.
B:Stratified scores expose failures that an overall aggregate conceals; representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; this tests per-segment acceptance rate across language, length, and noisy-input strata; the actual release condition is correct resolution of cross-exhibit conflicts at the required latency and review cost.
C:Version-aware tests make improvement claims meaningful for the deployed application; treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; this tests no contract regressions and acceptable quality/latency deltas on the proposed release; the actual release condition is correct resolution of cross-exhibit conflicts at the required latency and review cost.
Question 431
Developers add dozens of examples and complex reasoning steps, increasing tokens without improving category agreement. Which intervention targets the defect rather than its symptoms?
- Decouple instructions and source regions with clear tags
- Start with clear zero-shot instructions for unambiguous tasks
- Use contrastive examples when two outputs look similar
- Chain prompts when one result is a prerequisite for another
Explanation
Correct Answer
B:A concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; controlled comparisons show whether added prompt complexity yields a genuine benefit; verify the repair using label accuracy and token use versus a heavier example-based prompt.
Incorrect Answers
A:Tagged presentation cannot replace authenticating a source or enforcing repository access rights; use clearly named sections or XML-style tags so Claude can distinguish task policy, input, and supporting documents; that control is assessed by correct separation of document claims and application instructions in adversarial input tests; the incident instead requires label accuracy and token use versus a heavier example-based prompt.
C:Adding examples without highlighting the differentiating condition can reinforce superficial formatting only; show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; that control is assessed by correct type selection on unseen evidence-versus-action pairs; the incident instead requires label accuracy and token use versus a heavier example-based prompt.
D:Chaining every trivial substep increases latency without improving reliability for simple independent tasks; use a staged prompt chain where a verified intermediate classification is required before the next generation step; that control is assessed by explanations generated only for records that passed the eligibility stage; the incident instead requires label accuracy and token use versus a heavier example-based prompt.
Question 432
Before accepting uploaded documents, a retrieval application is tested with hostile instructions and intentionally unanswerable requests. Which measurement would demonstrate safety on negative cases rather than only friendly prompts?
- Holdout quality stability across successive prompt versions
- Expert-rated factual fidelity and calibrated grader agreement on sampled explanations
- Zero schema violations accepted by the downstream consumer across generated cases
- Attack success rate and safe handling of intentionally unanswerable prompts
Explanation
Correct Answer
D:Red-team cases make trust-boundary performance observable; measure attack success rate and safe handling of intentionally unanswerable prompts at the release gate; add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior.
Incorrect Answers
A:Untouched data estimates whether improvements transfer beyond the examples used during tuning; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; this tests holdout quality stability across successive prompt versions; the actual release condition is attack success rate and safe handling of intentionally unanswerable prompts.
B:A semantic rubric tests meaning while allowing legitimate variation in language; use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; this tests expert-rated factual fidelity and calibrated grader agreement on sampled explanations; the actual release condition is attack success rate and safe handling of intentionally unanswerable prompts.
C:Executable contract tests deliver reproducible feedback at scale; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; this tests zero schema violations accepted by the downstream consumer across generated cases; the actual release condition is attack success rate and safe handling of intentionally unanswerable prompts.
Question 433
Disabling it without a compatibility plan breaks automated customer-service workflows. Which change should the architect recommend first?
- Set an evidence gate before moving beyond discovery
- Transfer durable ownership and operations before handoff
- Use bounded pilots with exit criteria before expansion
- Plan deprecation and rollback from the start
Explanation
Correct Answer
D:Inventory consumers, define migration dates and fallback, preserve data obligations, and test the rollback or retirement path; dependency-aware migration tests verify that lifecycle closure is complete; verify the repair using successful retirement with zero unplanned dependent-system failures.
Incorrect Answers
A:A procurement timeline does not eliminate uncertainty about whether the problem is solvable; require validated problem scope, value hypothesis, data authority, and feasibility evidence before full design commitment; that control is assessed by closure of high-risk feasibility questions before design approval; the incident instead requires successful retirement with zero unplanned dependent-system failures.
B:A completed repository is not proof of ongoing service operability; establish production owners for reliability, security, spend, data, prompts, and escalation, with verified access and runbooks; that control is assessed by named owner coverage and support-drill completion at handoff; the incident instead requires successful retirement with zero unplanned dependent-system failures.
C:More launch enthusiasm does not replace production-like testing under load and edge conditions; define pilot size, critical acceptance metrics, escalation paths, and scale-up criteria before broad deployment; that control is assessed by pilot gate pass rate with defined performance and safety thresholds; the incident instead requires successful retirement with zero unplanned dependent-system failures.
Question 434
Launch discussions revisit known threats without deciding mitigation or escalation responsibility. What design correction resolves the reported failure?
- Demonstrate known limitations alongside successful workflows
- Maintain a risk register with controls and accountable owners
- Present executive decisions through outcomes, cost, and risk
- Document alternatives and rationale in an architecture decision record
Explanation
Correct Answer
B:Record probability and impact where appropriate, control owners, evidence, residual risk, and required approvals; a control register makes risk decisions reviewable rather than rhetorical; verify the repair using completion and ownership of high-severity mitigations before the release gate.
Incorrect Answers
A:A demo focused solely on best cases can mask residual risks and create unrealistic expectations; use representative happy-path and failure-path demonstrations with explicit fallback, uncertainty, and human ownership; that control is assessed by stakeholder identification of boundaries and fallback responsibilities after the demo; the incident instead requires completion and ownership of high-severity mitigations before the release gate.
C:A detailed API design is valuable to implementers but not sufficient for a funding decision; summarize business value, alternatives, cost envelope, measured evidence, and requested decisions in concise nontechnical terms; that control is assessed by sponsor comprehension of expected outcomes, decision options, and risk ownership; the incident instead requires completion and ownership of high-severity mitigations before the release gate.
D:A final diagram alone often omits rejected alternatives and the reasoning behind the choice; use an ADR recording context, considered options, decision, trade-offs, implications, and conditions for revisiting it; that control is assessed by reconstructability of key design decisions after a simulated staff handoff; the incident instead requires completion and ownership of high-severity mitigations before the release gate.
Question 435
A customer agent uses account tools and generates sensitive recommendations. A red-team drill combines abusive prompts with forged tool inputs across several stages. Which result would prove that the layered safety design protects both output and action boundaries?
- Forbidden file and network access attempts blocked in execution tests
- Denial of unauthorized actions and harmful outputs across multi-stage red-team paths
- Attack success rate on adversarial web pages and zero unauthorized egress calls
- Zero unapproved recipients or fields accepted by the outbound service
Explanation
Correct Answer
B:End-to-end adversarial cases assess the combined defenses and their failure isolation; measure denial of unauthorized actions and harmful outputs across multi-stage red-team paths at the release gate; layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority.
Incorrect Answers
A:Sandbox escapes and denial logs are more relevant than the assistant's stated intentions; isolate execution with restricted filesystem, network, credentials, resources, and observable policy denials; this tests forbidden file and network access attempts blocked in execution tests; the actual release condition is denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
C:Simulated hostile documents test the trust boundary directly; keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; this tests attack success rate on adversarial web pages and zero unauthorized egress calls; the actual release condition is denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
D:Contract-level checks demonstrate control over the actual data leaving the system; validate destinations, argument types, sensitive fields, and authorization before dispatch; reject values outside the approved contract; this tests zero unapproved recipients or fields accepted by the outbound service; the actual release condition is denial of unauthorized actions and harmful outputs across multi-stage red-team paths.
Question 436
An identity-management assistant prepares change plans for later approval. A red-team exercise tries to make the planning phase perform a real account update. Which release result verifies the privilege boundary?
- Unauthorized mutations during plan-only tests, with a target of zero
- Stage-specific defect localization and repair without redoing previously validated stages
- Conflict detection recall and citation coverage across independently processed source sets
- Correct merged clause counts across repeated parallel execution races
- Contract test pass rate for independently upgraded stages and invalid payload rejection
Explanation
Correct Answer
A:This tests enforcement at the action boundary rather than relying on prose instructions; measure unauthorized mutations during plan-only tests, with a target of zero at the release gate; use a planning artifact that is reviewed and handed to a distinct authorized executor; never treat proposed tool calls as permission to mutate state.
Incorrect Answers
B:A traceable artifact boundary confines error analysis to the failed operation; separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries; this tests stage-specific defect localization and repair without redoing previously validated stages; the actual release condition is unauthorized mutations during plan-only tests, with a target of zero.
C:The final synthesis must expose disagreements rather than average them away; analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; this tests conflict detection recall and citation coverage across independently processed source sets; the actual release condition is unauthorized mutations during plan-only tests, with a target of zero.
D:Deterministic merge checks measure integrity rather than only wall-clock speed; isolate independent tasks, retain separate results, then join only when required dependencies have completed; this tests correct merged clause counts across repeated parallel execution races; the actual release condition is unauthorized mutations during plan-only tests, with a target of zero.
E:Versioned interfaces expose breaking changes before a multi-team deployment; typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions; this tests contract test pass rate for independently upgraded stages and invalid payload rejection; the actual release condition is unauthorized mutations during plan-only tests, with a target of zero.
Question 437
Following an API refactor, a team regenerates endpoint documentation and example requests from the current implementation. Which evidence would prove the examples match what the reviewed build actually accepts?
- CI pass reproducibility on clean builds with locked dependencies
- Documented endpoint examples executing successfully against the reviewed build
- Passing unit and property tests including specified rounding boundaries
- Checklist completeness across independent sessions using the reviewed Skill version
Explanation
Correct Answer
B:Executable examples reveal fabricated or obsolete instructions; measure documented endpoint examples executing successfully against the reviewed build at the release gate; ground documentation generation in current code, schemas, and tests, then validate examples and have owners review changes.
Incorrect Answers
A:Independent automation prevents environmental drift from masquerading as code correctness; execute standardized linting, type checks, tests, and security scanning in versioned CI environments before merges; this tests cI pass reproducibility on clean builds with locked dependencies; the actual release condition is documented endpoint examples executing successfully against the reviewed build.
C:Test-first evidence anchors generated code in independently defined behavior; define executable tests and expected behavior before relying on AI-generated implementation, then review failures and iterate; this tests passing unit and property tests including specified rounding boundaries; the actual release condition is documented endpoint examples executing successfully against the reviewed build.
D:Reuse tests verify that essential steps are consistently available; maintain task-scoped reusable Skills or playbooks with versioning, ownership, and documented prerequisites; this tests checklist completeness across independent sessions using the reviewed Skill version; the actual release condition is documented endpoint examples executing successfully against the reviewed build.
Question 438
Aggregate dashboards approve the release despite a serious segment-specific decline. Which corrective change best addresses the root cause?
- Inspect segment-level effects rather than only global lift
- Use shadow evaluation before enabling new side effects
- Randomize consistently at the user or account level
- Require an explicit rollback trigger for harmful regressions
Explanation
Correct Answer
A:Compare treatment effects across predeclared critical segments and require safeguards where harm would be unacceptable; segment-aware decisions expose concentrated regressions before full rollout; verify the repair using resolution and safety deltas by channel and accessibility segment.
Incorrect Answers
B:A normal A/B test is inappropriate if experimental behavior can cause unreviewed irreversible changes; replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; that control is assessed by shadow-policy agreement and zero write side effects across replayed billing cases; the incident instead requires resolution and safety deltas by channel and accessibility segment.
C:Request-by-request randomization can be invalid when outcomes depend on ongoing user interactions; assign a stable randomized unit aligned to the behavior being measured and maintain consistent treatment during the experiment; that control is assessed by balanced treatment cohorts and clean user-level exposure histories; the incident instead requires resolution and safety deltas by channel and accessibility segment.
D:Monitoring without an action threshold can document harm while allowing it to continue; establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; that control is assessed by time from triggered guardrail breach to verified rollback; the incident instead requires resolution and safety deltas by channel and accessibility segment.
Question 439
A contact-center leader attributes improving first-contact resolution entirely to a new Claude assistant amid a staffing and policy change. Which intervention is most appropriate for this use case?
- Translate response-time expectations into a percentile SLO
- Measure productivity after correction and escalation
- Design a degraded mode that protects the value proposition
- Use a control group to establish incremental business impact
- Choose model and architecture tiers by case impact
Explanation
Correct Answer
D:Use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; a defensible counterfactual is required to credit the architecture with the observed benefit; evaluate the design with incremental resolution lift compared with a comparable non-assisted cohort.
Incorrect Answers
A:Define a p95 or p99 target against real end-to-end requests, including retrieval and tools, rather than reporting an average model time; a percentile SLO does not alone reveal whether the answers are factually correct; this alternative would measure end-to-end p95 latency and error rate against the agreed service objective; it does not establish incremental resolution lift compared with a comparable non-assisted cohort for the described use case.
B:Measure net time saved after review, appeals, and escalations, segmented by complexity, before claiming productivity; counting drafts as completed outcomes overlooks downstream human effort and defect consequences; this alternative would measure end-to-end completed cases per paid work-hour with rework and appeal volumes; it does not establish incremental resolution lift compared with a comparable non-assisted cohort for the described use case.
C:Separate critical deterministic operations from AI enhancement and provide explicit fallback and manual paths when model service is unavailable; high answer accuracy in normal mode says little about process continuity during dependency failures; this alternative would measure successful completion of critical workflows during injected model outages; it does not establish incremental resolution lift compared with a comparable non-assisted cohort for the described use case.
E:Route by validated task complexity and consequence, using stronger reasoning or human review where evidence justifies the incremental spend; a uniformly premium model can raise cost without proportionate value on simple, low-risk tasks; this alternative would measure quality and total cost segmented by request risk tier under a fixed accuracy floor; it does not establish incremental resolution lift compared with a comparable non-assisted cohort for the described use case.
Question 440
An enterprise helpdesk gets thirty plausible retrieved articles for each query, many covering obsolete or adjacent workflows. Which approach best meets the stated requirement?
- Rerank candidate passages for question relevance
- Use semantic vector search for paraphrased knowledge questions
- Combine lexical and semantic candidates when queries mix both
- Query structured data through authorized SQL or typed APIs
Explanation
Correct Answer
A:Apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; rank-sensitive evaluation tests whether useful evidence reaches the final context window; evaluate the design with relevant-procedure rank and end-to-end answer support rate after reranking.
Incorrect Answers
B:Vector retrieval can match conceptual similarity between user wording and document text, followed by evidence verification; semantic search is not guaranteed to respect exact version strings, numerals, or access boundaries without additional controls; this alternative would measure recall on human-authored paraphrases absent from manual headings; it does not establish relevant-procedure rank and end-to-end answer support rate after reranking for the described use case.
C:Hybrid retrieval joins exact-term and semantic evidence, then applies metadata filtering and ranking; either modality alone leaves a known class of mixed-format queries uncovered; this alternative would measure joint exact-identifier accuracy and semantic-symptom recall on mixed queries; it does not establish relevant-procedure rank and end-to-end answer support rate after reranking for the described use case.
D:Use a governed structured query interface with parameter checks and business semantics for exact aggregates; unstructured document retrieval cannot ensure completeness or mathematical correctness for database totals; this alternative would measure exact aggregate agreement with a trusted query on authorized records; it does not establish relevant-procedure rank and end-to-end answer support rate after reranking for the described use case.