Question 90
Applicants using an automated scholarship screening service can challenge suspected factual errors. Which measured outcome would demonstrate that review and correction are accessible and accountable?
- Task completion by users of tested assistive technologies on representative flows
- Time to resolve substantiated challenges and proportion of cases with documented review
- Quality and escalation parity across predeclared language segments
- Successful intervention before seeded high-impact incorrect actions are committed
Explanation
Correct Answer
B:Remedy effectiveness measures whether affected people can obtain meaningful reconsideration; measure time to resolve substantiated challenges and proportion of cases with documented review at the release gate; provide a contest and correction process with accountable human ownership for significant impacts.
Incorrect Answers
A:Direct accessibility testing measures practical use rather than formal content correctness only; include accessibility requirements, assistive-technology testing, and inclusive content rubrics in evaluation and design; this tests task completion by users of tested assistive technologies on representative flows; the actual release condition is time to resolve substantiated challenges and proportion of cases with documented review.
C:Group-specific evaluation reveals concentrated harm masked by pooled scores; evaluate suitable subgroup error rates and access outcomes, with privacy-aware sampling and appropriate fairness review; this tests quality and escalation parity across predeclared language segments; the actual release condition is time to resolve substantiated challenges and proportion of cases with documented review.
D:Pre-commit override drills test real controllability rather than retrospective documentation; preserve an authorized override and recovery route scaled to decision consequences, with feedback recorded for evaluation; this tests successful intervention before seeded high-impact incorrect actions are committed; the actual release condition is time to resolve substantiated challenges and proportion of cases with documented review.
Question 91
A procurement assistant suggests supplier master-data updates, but downstream APIs accept only approved fields and valid category codes. Which choice most directly satisfies the documented goal?
- Normalize incoming documents into a validated input contract
- Use a queue and asynchronous workers for bursty workloads
- Capture reviewer corrections as labeled feedback
- Validate model output against a business schema before side effects
- Link each output to source and processing trace IDs
Explanation
Correct Answer
D:Use structured output parsing, semantic validation, and a separate authorized command handler so generated text is never treated directly as a write instruction; pre-write enforcement proves that the model cannot bypass the target system's integrity contract; evaluate the design with invalid write payloads rejected before execution, with zero partial supplier mutations.
Incorrect Answers
A:An input-adapter layer validates document type, field types, provenance, and required metadata before the model reasons over values; input normalization addresses upstream ambiguity, not the authorization needed before writing to a finance ledger; this alternative would measure percentage of ingested documents meeting schema checks before any Claude call; it does not establish invalid write payloads rejected before execution, with zero partial supplier mutations for the described use case.
B:Decouple intake from processing through durable queues, bounded workers, and status callbacks rather than keeping user connections open; queueing absorbs bursts but does not improve a weak answer rubric or justify a higher-privilege tool set; this alternative would measure job completion rate and queue age during the peak intake window; it does not establish invalid write payloads rejected before execution, with zero partial supplier mutations for the described use case.
C:Preserve authorized reviewer labels linked to case versions and turn them into regression examples with data-retention controls; a feedback dataset improves future evaluation; it does not guarantee that the current response is timely or properly permissioned; this alternative would measure regression pass rate on previously corrected clause-omission cases; it does not establish invalid write payloads rejected before execution, with zero partial supplier mutations for the described use case.
E:Correlation IDs, source versions, and stage logs establish end-to-end lineage across extraction, retrieval, inference, and review; traceability explains decisions after the fact; it is not a replacement for preventing unauthorized source access; this alternative would measure fraction of sampled outputs reproducibly linked to input, model configuration, and reviewer disposition; it does not establish invalid write payloads rejected before execution, with zero partial supplier mutations for the described use case.
Question 92
An audit team is unable to recreate a previous assistant answer because release-specific prompts and tool definitions were lost. A different operations team faces an outdated-source incident but lacks clear containment commands. Which TWO documentation measures address reproducibility and incident response? Choose TWO.
- Maintain an end-to-end data-flow and trust-boundary diagram
- Specify tool schemas, errors, and authorization contracts
- Version prompts, model routes, tools, and evaluation artifacts together
- Publish an incident runbook with containment and rollback steps
- Document decisions with assumptions and reversal triggers
Explanation
Correct Answers
C:Maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; versioned provenance enables defensible debugging and rollback; its corresponding workstream should be validated by reproducibility of a sampled historical answer using its full manifest.
D:Maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; a runbook is effective when a different on-call engineer can perform the recovery; its corresponding workstream should be validated by successful containment time in a simulated stale-index incident.
Incorrect Answers
A:A current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; a feature list does not reveal crossing of sensitive trust boundaries; its natural acceptance measure is completeness of sensitive-data flow paths identified during security review, which does not satisfy the separate workstreams being assessed.
B:Define typed inputs, outputs, error semantics, idempotency, required scopes, and audit obligations for tool boundaries; aPI examples without failure and permission semantics leave dangerous integration ambiguity; its natural acceptance measure is consumer contract pass rate under timeouts, invalid arguments, and permission denials, which does not satisfy the separate workstreams being assessed.
E:Record architectural context, alternatives, decision rationale, constraints, and explicit conditions for revisiting choices; configuration exports show what exists but rarely why it was chosen; its natural acceptance measure is ability to identify affected decisions after simulated policy and workload changes, which does not satisfy the separate workstreams being assessed.
Question 93
A development assistant loads full tool schemas only after a compact tool search selects a candidate. Which paired result would prove reduced context pressure without an increase in malformed calls?
- Correct endpoint contract use with lower documentation-token overhead
- Schema-token consumption and invalid-argument rate after staged loading
- Intent-classification accuracy before any tool disclosure or call
- Cache-read tokens and total context cost while changing selected tool subsets
Explanation
Correct Answer
B:The test balances context savings against input-contract reliability; measure schema-token consumption and invalid-argument rate after staged loading at the release gate; keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable.
Incorrect Answers
A:Targeted documentation fetches should improve focus without losing necessary details; discover resource summaries first and fetch the specific authoritative documentation section at the point of need; this tests correct endpoint contract use with lower documentation-token overhead; the actual release condition is schema-token consumption and invalid-argument rate after staged loading.
C:Phase separation verifies that the assistant understands the task before acting; separate goal interpretation and policy decisions from a downstream staged capability-discovery step; this tests intent-classification accuracy before any tool disclosure or call; the actual release condition is schema-token consumption and invalid-argument rate after staged loading.
D:The combined test reveals interaction between discovery strategy and prompt-prefix economics; isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary; this tests cache-read tokens and total context cost while changing selected tool subsets; the actual release condition is schema-token consumption and invalid-argument rate after staged loading.
Question 94
A service-management copilot resumes an approved task after its conversation context expires. Which fault-recovery result would prove that external state and approval records survived the reset?
- Correct recovery of approvals and pending actions after forced session reset
- Complete mandatory report sections under the worst supported input size
- Citation correctness by effective-date version in temporal regression questions
- Answer support and retrieved-token volume on a sampled policy-question set
Explanation
Correct Answer
A:Durable-state testing proves behavior survives context eviction; measure correct recovery of approvals and pending actions after forced session reset at the release gate; store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger.
Incorrect Answers
B:Budgeted completion tests directly reveal truncation at maximum expected load; reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; this tests complete mandatory report sections under the worst supported input size; the actual release condition is correct recovery of approvals and pending actions after forced session reset.
C:Metadata-aware checks distinguish legitimate historic references from current-policy answers; supply source identity, version, effective date, and retrieval relevance alongside each chunk so the model can attribute evidence correctly; this tests citation correctness by effective-date version in temporal regression questions; the actual release condition is correct recovery of approvals and pending actions after forced session reset.
D:Relevance per token measures whether context compression preserves the needed evidence; apply query-driven retrieval and evidence selection before prompting Claude, retaining citations and permission filters; this tests answer support and retrieved-token volume on a sampled policy-question set; the actual release condition is correct recovery of approvals and pending actions after forced session reset.
Question 95
A medical knowledge lookup is tested under simultaneous requests from busy clinics. The service objective applies to the completed, checked answer seen by the clinician. Which measurement is appropriate?
- Fully loaded cost per correctly completed document
- Severity-weighted policy violation incidence on a reviewed adversarial test set
- Unauthorized API requests rejected under role, tenant, and resource attack matrices
- p95 time from user submission to fully validated answer under load
Explanation
Correct Answer
D:Measuring the completed workflow captures actual end-user waiting, including downstream services; measure p95 time from user submission to fully validated answer under load at the release gate; report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime.
Incorrect Answers
A:This metric includes rework rather than rewarding low-quality cheap drafts; relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; this tests fully loaded cost per correctly completed document; the actual release condition is p95 time from user submission to fully validated answer under load.
B:Risk-weighted scoring detects rare critical violations that a single average hides; define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; this tests severity-weighted policy violation incidence on a reviewed adversarial test set; the actual release condition is p95 time from user submission to fully validated answer under load.
C:Direct penetration-style checks assess the enforceable security boundary; treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; this tests unauthorized API requests rejected under role, tenant, and resource attack matrices; the actual release condition is p95 time from user submission to fully validated answer under load.
Question 96
AI-generated refund logic passes one demonstration but mishandles negative line-item adjustments. Which development practice should establish the expected edge-case behavior before accepting the implementation?
- Run reproducible CI gates on assisted code
- Package reusable engineering instructions as governed Skills
- Use test-first acceptance cases for generated changes
- Review AI-generated diffs for logic and security regressions
Explanation
Correct Answer
C:Define executable tests and expected behavior before relying on AI-generated implementation, then review failures and iterate; test-first evidence anchors generated code in independently defined behavior; verify the repair using passing unit and property tests including specified rounding boundaries.
Incorrect Answers
A:Local success can be environment-dependent and should not be the only release assurance; execute standardized linting, type checks, tests, and security scanning in versioned CI environments before merges; that control is assessed by cI pass reproducibility on clean builds with locked dependencies; the incident instead requires passing unit and property tests including specified rounding boundaries.
B:Shared instructions improve consistency but do not replace enforced approval or automated tests; maintain task-scoped reusable Skills or playbooks with versioning, ownership, and documented prerequisites; that control is assessed by checklist completeness across independent sessions using the reviewed Skill version; the incident instead requires passing unit and property tests including specified rounding boundaries.
D:Successful compilation does not establish preservation of security controls; require focused code review of the actual diff, threat-relevant tests, and approval before merging sensitive changes; that control is assessed by detection of seeded authorization regressions during peer review and CI; the incident instead requires passing unit and property tests including specified rounding boundaries.
Question 97
A policy change invalidates those assumptions but no one knows what would trigger redesign. Which corrective change best addresses the root cause?
- Publish an incident runbook with containment and rollback steps
- Document decisions with assumptions and reversal triggers
- Version prompts, model routes, tools, and evaluation artifacts together
- Maintain an end-to-end data-flow and trust-boundary diagram
Explanation
Correct Answer
B:Record architectural context, alternatives, decision rationale, constraints, and explicit conditions for revisiting choices; assumption-aware ADRs support controlled adaptation rather than accidental drift; verify the repair using ability to identify affected decisions after simulated policy and workload changes.
Incorrect Answers
A:A high-level diagram alone is not an executable incident response procedure; maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; that control is assessed by successful containment time in a simulated stale-index incident; the incident instead requires ability to identify affected decisions after simulated policy and workload changes.
C:Recording only the model name omits other behavior-changing dependencies; maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; that control is assessed by reproducibility of a sampled historical answer using its full manifest; the incident instead requires ability to identify affected decisions after simulated policy and workload changes.
D:A feature list does not reveal crossing of sensitive trust boundaries; a current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; that control is assessed by completeness of sensitive-data flow paths identified during security review; the incident instead requires ability to identify affected decisions after simulated policy and workload changes.
Question 98
The team deploys its largest available model for routine tags and exceeds its cost budget without a measurable quality benefit. What design correction resolves the reported failure?
- Use stronger reasoning capacity for genuinely complex decisions
- Benchmark a smaller model for routine classification
- Gate model-version updates on regression tests
- Select a model path compatible with required modalities
Explanation
Correct Answer
B:Compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; joint quality and price measurement prevents choosing a cheap model that fails costly minority categories; verify the repair using macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic.
Incorrect Answers
A:High reasoning capacity is wasteful if the task only extracts a known field or matches a stable label; escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; that control is assessed by correct resolution of cross-exhibit conflicts at the required latency and review cost; the incident instead requires macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic.
C:Newer model branding alone cannot ensure identical tool-use or output behavior in an existing workflow; treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; that control is assessed by no contract regressions and acceptable quality/latency deltas on the proposed release; the incident instead requires macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic.
D:Model selection based solely on text benchmark accuracy does not prove image interpretation reliability; confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; that control is assessed by correct equipment identification and recommendation accuracy on labeled photographs; the incident instead requires macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic.
Question 99
A shared incident-writing tool is being adopted by teams that have different client libraries. Tests deliberately send malformed inputs, duplicate requests, and expired permissions. Which result would demonstrate contract compatibility?
- Completeness of sensitive-data flow paths identified during security review
- Reproducibility of a sampled historical answer using its full manifest
- Successful completion of standard support drills by staff who did not build the system
- Consumer contract pass rate under timeouts, invalid arguments, and permission denials
Explanation
Correct Answer
D:Contract tests validate the documented behavior under non-happy paths; measure consumer contract pass rate under timeouts, invalid arguments, and permission denials at the release gate; define typed inputs, outputs, error semantics, idempotency, required scopes, and audit obligations for tool boundaries.
Incorrect Answers
A:Architecture diagrams are useful when they expose where protection and authority change; a current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; this tests completeness of sensitive-data flow paths identified during security review; the actual release condition is consumer contract pass rate under timeouts, invalid arguments, and permission denials.
B:Versioned provenance enables defensible debugging and rollback; maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; this tests reproducibility of a sampled historical answer using its full manifest; the actual release condition is consumer contract pass rate under timeouts, invalid arguments, and permission denials.
C:Independent operator readiness is stronger evidence than architect-led demonstrations; provide system overview, access setup, troubleshooting flowcharts, monitoring thresholds, and named support responsibilities; this tests successful completion of standard support drills by staff who did not build the system; the actual release condition is consumer contract pass rate under timeouts, invalid arguments, and permission denials.
Question 100
A report generator frequently reaches output limits before its mandatory findings section is complete. What is the most defensible next step?
- Allocate input and output token budgets per workflow step
- Remove obsolete tool output while preserving current evidence
- Persist operational decisions outside transient chat history
- Compact extended sessions without discarding commitments
Explanation
Correct Answer
A:Reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; budgeted completion tests directly reveal truncation at maximum expected load; evaluate the design with complete mandatory report sections under the worst supported input size.
Incorrect Answers
B:Prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; pruning without version markers can erase the evidence needed to explain the incident's evolution; this alternative would measure current-state citation accuracy across log-update sequences; it does not establish complete mandatory report sections under the worst supported input size for the described use case.
C:Store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger; long context alone cannot provide transactional durability or a trustworthy authorization trail; this alternative would measure correct recovery of approvals and pending actions after forced session reset; it does not establish complete mandatory report sections under the worst supported input size for the described use case.
D:Store a structured, reviewable summary of decisions, open issues, and provenance; inject it at checkpoints rather than replaying every chat turn; blind summarization can lose critical exceptions unless durable commitments are explicitly represented; this alternative would measure retention of signed-off requirements after long simulated conversations; it does not establish complete mandatory report sections under the worst supported input size for the described use case.