Question 650
An invoice coder returns fluent category descriptions but the business requires exact tax-code accuracy. What should the solution architect recommend?
- Assess whether abstention and escalation are calibrated
- Use task-correctness metrics tied to verified labels
- Use tail-latency indicators for interactive promises
- Test whether the tool layer denies unauthorized requests
- Measure rate and severity of safety policy violations
Explanation
Correct Answer
B:Measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; label-based metrics catch mistakes that surface-level satisfaction may conceal; evaluate the design with critical-class precision and recall on independently verified tax-code cases.
Incorrect Answers
A:Measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; high refusal frequency alone can appear safe while making the assistant operationally useless; this alternative would measure correct-answer rate at each supported coverage threshold including no-answer cases; it does not establish critical-class precision and recall on independently verified tax-code cases for the described use case.
C:Report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; mean latency underweights rare long waits and omits retrieval and external-tool delays; this alternative would measure p95 time from user submission to fully validated answer under load; it does not establish critical-class precision and recall on independently verified tax-code cases for the described use case.
D:Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; model refusal scores can be high even when the underlying API is vulnerable; this alternative would measure unauthorized API requests rejected under role, tenant, and resource attack matrices; it does not establish critical-class precision and recall on independently verified tax-code cases for the described use case.
E:Define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; general task correctness does not measure policy-conformance failures on dangerous requests; this alternative would measure severity-weighted policy violation incidence on a reviewed adversarial test set; it does not establish critical-class precision and recall on independently verified tax-code cases for the described use case.
Question 651
A troubleshooting agent accumulates old logs and superseded metrics during a long incident. Which intervention is most appropriate for this use case?
- Remove obsolete tool output while preserving current evidence
- Persist operational decisions outside transient chat history
- Allocate input and output token budgets per workflow step
- Attach provenance metadata to selected context
Explanation
Correct Answer
A:Prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; freshness-aware tests expose whether stale tool observations override newer facts; evaluate the design with current-state citation accuracy across log-update sequences.
Incorrect Answers
B:Store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger; long context alone cannot provide transactional durability or a trustworthy authorization trail; this alternative would measure correct recovery of approvals and pending actions after forced session reset; it does not establish current-state citation accuracy across log-update sequences for the described use case.
C:Reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; merely expanding every context window can increase cost while leaving output truncation unaddressed; this alternative would measure complete mandatory report sections under the worst supported input size; it does not establish current-state citation accuracy across log-update sequences for the described use case.
D:Supply source identity, version, effective date, and retrieval relevance alongside each chunk so the model can attribute evidence correctly; provenance metadata does not guarantee the upstream search selected the right document in the first place; this alternative would measure citation correctness by effective-date version in temporal regression questions; it does not establish current-state citation accuracy across log-update sequences for the described use case.
Question 652
Reviewers see compliant conversation text but a direct tool call with a forged tenant ID succeeds. Which intervention targets the defect rather than its symptoms?
- Track cost per verified successful task
- Measure rate and severity of safety policy violations
- Evaluate whether abstention and escalation are calibrated
- Test whether the tool layer denies unauthorized requests
- Use task-correctness metrics tied to verified labels
Explanation
Correct Answer
D:Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; direct penetration-style checks assess the enforceable security boundary; verify the repair using unauthorized API requests rejected under role, tenant, and resource attack matrices.
Incorrect Answers
A:Cost per token can improve while cost per correct business outcome deteriorates; relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; that control is assessed by fully loaded cost per correctly completed document; the incident instead requires unauthorized API requests rejected under role, tenant, and resource attack matrices.
B:General task correctness does not measure policy-conformance failures on dangerous requests; define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; that control is assessed by severity-weighted policy violation incidence on a reviewed adversarial test set; the incident instead requires unauthorized API requests rejected under role, tenant, and resource attack matrices.
C:High refusal frequency alone can appear safe while making the assistant operationally useless; measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; that control is assessed by correct-answer rate at each supported coverage threshold including no-answer cases; the incident instead requires unauthorized API requests rejected under role, tenant, and resource attack matrices.
E:Positive conversational tone does not establish that a legally or financially meaningful label is correct; measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; that control is assessed by critical-class precision and recall on independently verified tax-code cases; the incident instead requires unauthorized API requests rejected under role, tenant, and resource attack matrices.
Question 653
Launch is delayed because stakeholders disagree over what 'done' means and which cases remain manual. How should the design be corrected?
- Obtain sign-off on explicit acceptance criteria and exclusions
- Interview each role that owns a distinct outcome or risk
- Capture external dependencies and acceptance risks explicitly
- Record nonfunctional requirements as measurable thresholds
Explanation
Correct Answer
A:Document in-scope outcomes, exclusions, metrics, responsibilities, and go-live gates for accountable approval; an explicit agreement limits scope drift and creates a common release standard; verify the repair using signed acceptance criteria mapped to demonstrable tests and named owners.
Incorrect Answers
B:A sponsor's high-level vision does not fully describe operational controls or edge cases; use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders; that control is assessed by coverage of approved requirements traced to affected stakeholder groups; the incident instead requires signed acceptance criteria mapped to demonstrable tests and named owners.
C:An AI model benchmark cannot predict every upstream system's production behavior; maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; that control is assessed by verified readiness of high-impact external dependencies against test quotas and outage cases; the incident instead requires signed acceptance criteria mapped to demonstrable tests and named owners.
D:Functional task lists do not tell designers what constitutes acceptable production operation; specify capacity, tail latency, availability, privacy, and recovery objectives with measurement windows and ownership; that control is assessed by agreed p95, availability, and load thresholds exercised in an acceptance environment; the incident instead requires signed acceptance criteria mapped to demonstrable tests and named owners.
Question 654
A text-only processing step drops image information and gives incomplete equipment recommendations. Which action best removes the underlying failure mode?
- Assess model choice against the real input distribution
- Gate model-version updates on regression tests
- Use stronger reasoning capacity for genuinely complex decisions
- Select a model path compatible with required modalities
Explanation
Correct Answer
D:Confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; a multimodal acceptance set measures the capability absent from text-only trials; verify the repair using correct equipment identification and recommendation accuracy on labeled photographs.
Incorrect Answers
A:Vendor benchmark rankings do not establish suitability for a domain-specific and skewed workload; representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; that control is assessed by per-segment acceptance rate across language, length, and noisy-input strata; the incident instead requires correct equipment identification and recommendation accuracy on labeled photographs.
B:Newer model branding alone cannot ensure identical tool-use or output behavior in an existing workflow; treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; that control is assessed by no contract regressions and acceptable quality/latency deltas on the proposed release; the incident instead requires correct equipment identification and recommendation accuracy on labeled photographs.
C:High reasoning capacity is wasteful if the task only extracts a known field or matches a stable label; escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; that control is assessed by correct resolution of cross-exhibit conflicts at the required latency and review cost; the incident instead requires correct equipment identification and recommendation accuracy on labeled photographs.
Question 655
A clinician-facing knowledge lookup has a tight service target for almost all requests. Which approach best meets the stated requirement?
- Evaluate whether abstention and escalation are calibrated
- Use task-correctness metrics tied to verified labels
- Measure rate and severity of safety policy violations
- Track cost per verified successful task
- Use tail-latency indicators for interactive promises
- Test whether the tool layer denies unauthorized requests
Explanation
Correct Answer
E:Report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; measuring the completed workflow captures actual end-user waiting, including downstream services; evaluate the design with p95 time from user submission to fully validated answer under load.
Incorrect Answers
A:Measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; high refusal frequency alone can appear safe while making the assistant operationally useless; this alternative would measure correct-answer rate at each supported coverage threshold including no-answer cases; it does not establish p95 time from user submission to fully validated answer under load for the described use case.
B:Measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; positive conversational tone does not establish that a legally or financially meaningful label is correct; this alternative would measure critical-class precision and recall on independently verified tax-code cases; it does not establish p95 time from user submission to fully validated answer under load for the described use case.
C:Define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; general task correctness does not measure policy-conformance failures on dangerous requests; this alternative would measure severity-weighted policy violation incidence on a reviewed adversarial test set; it does not establish p95 time from user submission to fully validated answer under load for the described use case.
D:Relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; cost per token can improve while cost per correct business outcome deteriorates; this alternative would measure fully loaded cost per correctly completed document; it does not establish p95 time from user submission to fully validated answer under load for the described use case.
F:Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; model refusal scores can be high even when the underlying API is vulnerable; this alternative would measure unauthorized API requests rejected under role, tenant, and resource attack matrices; it does not establish p95 time from user submission to fully validated answer under load for the described use case.
Question 656
Several repositories use a shared, version-controlled release Skill that includes security checks. Which result across separate coding sessions would demonstrate that the standardized procedure is followed consistently?
- Review time and defect localization per incremental change set
- CI pass reproducibility on clean builds with locked dependencies
- Detection of seeded authorization regressions during peer review and CI
- Checklist completeness across independent sessions using the reviewed Skill version
Explanation
Correct Answer
D:Reuse tests verify that essential steps are consistently available; measure checklist completeness across independent sessions using the reviewed Skill version at the release gate; maintain task-scoped reusable Skills or playbooks with versioning, ownership, and documented prerequisites.
Incorrect Answers
A:Small-batch evidence shows whether modernization remains controlled and reversible; use small scoped changes, compatibility checkpoints, and separate pull requests tied to specific tests; this tests review time and defect localization per incremental change set; the actual release condition is checklist completeness across independent sessions using the reviewed Skill version.
B:Independent automation prevents environmental drift from masquerading as code correctness; execute standardized linting, type checks, tests, and security scanning in versioned CI environments before merges; this tests cI pass reproducibility on clean builds with locked dependencies; the actual release condition is checklist completeness across independent sessions using the reviewed Skill version.
C:Diff-focused review finds changes that ordinary output fluency cannot reveal; require focused code review of the actual diff, threat-relevant tests, and approval before merging sensitive changes; this tests detection of seeded authorization regressions during peer review and CI; the actual release condition is checklist completeness across independent sessions using the reviewed Skill version.
Question 657
The writer incorporates a stale research snapshot after an analyst has corrected the latest figures. How should the design be corrected?
- Version the handoff artifacts between agents
- Use a coordinator to route distinct specialist tasks
- Separate independent critique from generation
- Run independent specialists in parallel with a merge contract
Explanation
Correct Answer
A:Immutable, versioned handoff artifacts and typed schemas prevent one agent from consuming ambiguous or mismatched intermediate state; version-aware handoffs can be tested deterministically under racing writes; verify the repair using zero stale-artifact merges across concurrent revision and retry scenarios.
Incorrect Answers
B:Specialist routing adds handoffs and can be wasteful for one small, homogeneous transformation; a coordinator can dispatch scoped subagents and merge their typed outputs while restricting each specialist's context and tools; that control is assessed by completed reports with correctly scoped evidence and no cross-specialty data leakage; the incident instead requires zero stale-artifact merges across concurrent revision and retry scenarios.
C:A critic agent does not repair missing source documents or guarantee that all facts are verifiable; a distinct verifier with a rubric and evidence access can challenge unsupported assertions before an external document is released; that control is assessed by verified claim support rate after independent critique on seeded unsupported drafts; the incident instead requires zero stale-artifact merges across concurrent revision and retry scenarios.
D:Parallel fan-out is unsafe when tasks mutate the same state or depend on each other's evolving outputs; parallel fan-out followed by deterministic aggregation can shorten wall-clock time if each branch has its own contract and errors are handled; that control is assessed by p95 end-to-end latency and deterministic merge completeness under one-branch failures; the incident instead requires zero stale-artifact merges across concurrent revision and retry scenarios.
Question 658
A shipment-resolution process invokes Claude and several order APIs under intermittent network interruptions. What action best fits the business and technical boundary?
- Capture reviewer corrections as labeled feedback
- Normalize incoming documents into a validated input contract
- Persist workflow state and make retries idempotent
- Link each output to source and processing trace IDs
Explanation
Correct Answer
C:Persist state transitions and use idempotency keys at side-effect boundaries to avoid duplicate transactions across replays; replay tests directly exercise the failure window that typically creates duplicate writes; evaluate the design with repeated and interrupted test runs yielding exactly one committed credit per approved case.
Incorrect Answers
A:Preserve authorized reviewer labels linked to case versions and turn them into regression examples with data-retention controls; a feedback dataset improves future evaluation; it does not guarantee that the current response is timely or properly permissioned; this alternative would measure regression pass rate on previously corrected clause-omission cases; it does not establish repeated and interrupted test runs yielding exactly one committed credit per approved case for the described use case.
B:An input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values; input normalization addresses upstream ambiguity, not the authorization needed before writing to a finance ledger; this alternative would measure percentage of ingested documents meeting schema checks before any Claude call; it does not establish repeated and interrupted test runs yielding exactly one committed credit per approved case for the described use case.
D:Correlation IDs, source versions, and stage logs establish end-to-end lineage across extraction, retrieval, inference, and review; traceability explains decisions after the fact; it is not a replacement for preventing unauthorized source access; this alternative would measure fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition; it does not establish repeated and interrupted test runs yielding exactly one committed credit per approved case for the described use case.
Question 659
A user-controlled request is concatenated with higher-priority operational rules, making user text look like trusted policy. Which prompt-construction boundary should the application enforce?
- Use a machine-readable output contract and validator
- Write explicit roles, scope, and prohibited operations
- Keep invariant policy instructions distinct from per-request input
- Delimit retrieved content as untrusted evidence
Explanation
Correct Answer
C:Separate stable system-level policy from user content and tool results, retaining consistent privilege boundaries across requests; boundary-specific tests confirm that changing user content cannot rewrite the application contract; verify the repair using rate of user messages that fail to override defined task boundaries in regression tests.
Incorrect Answers
A:Human-readable examples alone do not guarantee that all runtime outputs satisfy machine contracts; request a documented structured format and validate schema, allowed values, and business semantics before downstream use; that control is assessed by valid schema rate and zero unauthorized enum values reaching the consumer; the incident instead requires rate of user messages that fail to override defined task boundaries in regression tests.
B:Role instructions improve behavior but cannot grant or restrict database permissions on their own; define the assistant's permitted tasks, out-of-scope handling, and escalation behavior in stable application-level instructions, backed by independent access enforcement; that control is assessed by out-of-scope refusal and handoff accuracy on a policy-boundary test set; the incident instead requires rate of user messages that fail to override defined task boundaries in regression tests.
D:Delimiters aid instruction hierarchy but do not sanitize malicious destinations or replace output security controls; clearly separate lower-trust source material from application instructions and enforce permissions at the tool layer as well; that control is assessed by prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use; the incident instead requires rate of user messages that fail to override defined task boundaries in regression tests.
Question 660
An insurance-policy transformation pipeline now emits independently validated extraction, comparison, and drafting artifacts. When seeded defects are introduced at different stages, which result would prove that debugging and repair are isolated?
- Stage-specific defect localization and repair without redoing previously validated stages
- Contract test pass rate for independently upgraded stages and invalid payload rejection
- Conflict detection recall and citation coverage across independently processed source sets
- Successful recovery after injected worker failures without duplicated writes or stages
Explanation
Correct Answer
A:A traceable artifact boundary confines error analysis to the failed operation; measure stage-specific defect localization and repair without redoing previously validated stages at the release gate; separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries.
Incorrect Answers
B:Versioned interfaces expose breaking changes before a multi-team deployment; typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions; this tests contract test pass rate for independently upgraded stages and invalid payload rejection; the actual release condition is stage-specific defect localization and repair without redoing previously validated stages.
C:The final synthesis must expose disagreements rather than average them away; analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; this tests conflict detection recall and citation coverage across independently processed source sets; the actual release condition is stage-specific defect localization and repair without redoing previously validated stages.
D:Replay testing verifies the precise recovery behavior the pipeline requires; durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects; this tests successful recovery after injected worker failures without duplicated writes or stages; the actual release condition is stage-specific defect localization and repair without redoing previously validated stages.