Question 170
A manufacturer tests a model on clean internal samples but production includes multilingual, noisy, and exceptionally long reports. What action best fits the business and technical boundary?
- Benchmark a smaller model for routine classification
- Evaluate model choice against the real input distribution
- Gate model-version updates on regression tests
- Select a model path compatible with required modalities
Explanation
Correct Answer
B:Representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; stratified scores expose failures that an overall aggregate conceals; evaluate the design with per-segment acceptance rate across language, length, and noisy-input strata.
Incorrect Answers
A:Compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; a lightweight route should not be assumed suitable for complex multi-step decisions without a representative evaluation; this alternative would measure macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic; it does not establish per-segment acceptance rate across language, length, and noisy-input strata for the described use case.
C:Treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; newer model branding alone cannot ensure identical tool-use or output behavior in an existing workflow; this alternative would measure no contract regressions and acceptable quality/latency deltas on the proposed release; it does not establish per-segment acceptance rate across language, length, and noisy-input strata for the described use case.
D:Confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; model selection based solely on text benchmark accuracy does not prove image interpretation reliability; this alternative would measure correct equipment identification and recommendation accuracy on labeled photographs; it does not establish per-segment acceptance rate across language, length, and noisy-input strata for the described use case.
Question 171
Reviewer accounts receive unrelated patient records that are not needed for judging the assistant's answer. What is the most direct remediation?
- Escalate binding legal interpretations to qualified reviewers
- Collect reviewer overrides as structured improvement signals
- Use risk-based sampling for low-impact decisions
- Give reviewers only necessary sensitive information
Explanation
Correct Answer
D:Minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; review workflows should preserve decision quality while protecting privacy; verify the repair using unauthorized field exposure and successful review accuracy after data minimization.
Incorrect Answers
A:A high model confidence estimate cannot replace delegated legal decision authority; define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; that control is assessed by correct escalation of ambiguous legal cases with reviewer attribution; the incident instead requires unauthorized field exposure and successful review accuracy after data minimization.
B:Counting approval clicks alone does not describe why humans disagreed with the AI; record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; that control is assessed by recurrence rate of categorized, previously corrected error types; the incident instead requires unauthorized field exposure and successful review accuracy after data minimization.
C:Universal manual review may be disproportionate when actions are reversible and errors are easily detected; combine automated checks with representative human sampling and exception-based review scaled to risk; that control is assessed by defect-detection rate and reviewer workload under a risk-tiered sampling policy; the incident instead requires unauthorized field exposure and successful review accuracy after data minimization.
Question 172
An employee assistant is offered a large enterprise connector library through demand-based tool discovery. Which evaluation result would show that selective discovery improves tool selection and reduces prompt overhead?
- Irrelevant tool exposure per intent with no loss of required task completion
- Tool-choice accuracy and context-token savings on varied employee requests
- Cache-read tokens and total context cost while changing selected tool subsets
- Schema-token consumption and invalid-argument rate after staged loading
Explanation
Correct Answer
B:Discovery quality must preserve necessary capabilities while shrinking unused schema context; measure tool-choice accuracy and context-token savings on varied employee requests at the release gate; expose a lean search or discovery interface and load relevant detailed tools only after intent is established.
Incorrect Answers
A:Exposure and completion metrics together reveal whether dynamic loading is too aggressive; select a bounded task-specific capability group after intent routing with conservative fallback for ambiguous requests; this tests irrelevant tool exposure per intent with no loss of required task completion; the actual release condition is tool-choice accuracy and context-token savings on varied employee requests.
C:The combined test reveals interaction between discovery strategy and prompt-prefix economics; isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary; this tests cache-read tokens and total context cost while changing selected tool subsets; the actual release condition is tool-choice accuracy and context-token savings on varied employee requests.
D:The test balances context savings against input-contract reliability; keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; this tests schema-token consumption and invalid-argument rate after staged loading; the actual release condition is tool-choice accuracy and context-token savings on varied employee requests.
Question 173
A broad rollout creates incompatible exception paths and no isolatable explanation for a spike in rework. Which corrective change best addresses the root cause?
- Pilot one bounded workflow with acceptance gates
- Map available data and its permitted uses before solution selection
- Define which decisions remain with people
- Prioritize a repetitive step with bounded exceptions
Explanation
Correct Answer
A:A narrow pilot validates one representative flow and pre-agreed quality, latency, and escalation thresholds before scaling to other workflows; a bounded deployment can prove business fitness before a large enterprise-wide dependency is created; verify the repair using passing the acceptance gate on a representative pilot cohort including difficult exceptions.
Incorrect Answers
B:A data inventory does not alone identify target SLOs or decide whether an orchestration loop is necessary; data-readiness and usage-rights mapping identifies legitimate inputs, owners, quality gaps, and prohibited transfers before committing to an implementation; that control is assessed by percentage of required fields with validated owners, quality checks, and approved processing purposes; the incident instead requires passing the acceptance gate on a representative pilot cohort including difficult exceptions.
C:Decision ownership is an authorization and accountability control; it does not fix missing input records or prompt-token waste; explicit decision-rights boundaries distinguish advisory generation from actions requiring a credentialed human or deterministic policy engine; that control is assessed by share of binding decisions showing the required named approver and rationale; the incident instead requires passing the acceptance gate on a representative pilot cohort including difficult exceptions.
D:Prioritization improves initial return and feasibility; it does not directly supply output-validation code or tenant isolation; select a high-frequency, well-described task with measurable work reduction and explicit escalation for ambiguous records; that control is assessed by accepted cases per reviewer-hour with exception share tracked separately; the incident instead requires passing the acceptance gate on a representative pilot cohort including difficult exceptions.
Question 174
Developers want live A/B execution even though the new strategy might produce unauthorized writes. Which change should the architect recommend first?
- Use shadow evaluation before enabling new side effects
- Inspect segment-level effects rather than only global lift
- Predefine success criteria and sample-size assumptions
- Randomize consistently at the user or account level
- Require an explicit rollback trigger for harmful regressions
Explanation
Correct Answer
A:Replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; non-mutating evidence reduces risk before enabling operational authority; verify the repair using shadow-policy agreement and zero write side effects across replayed billing cases.
Incorrect Answers
B:A positive overall average does not guarantee every relevant user group benefits; compare treatment effects across predeclared critical segments and require safeguards where harm would be unacceptable; that control is assessed by resolution and safety deltas by channel and accessibility segment; the incident instead requires shadow-policy agreement and zero write side effects across replayed billing cases.
C:Repeatedly peeking and stopping at a favorable fluctuation inflates false-positive decisions; define primary metrics, minimal meaningful effect, sample-size method, and stopping rules before examining results; that control is assessed by pre-registered experiment compliance and interval estimates at the planned sample size; the incident instead requires shadow-policy agreement and zero write side effects across replayed billing cases.
D:Request-by-request randomization can be invalid when outcomes depend on ongoing user interactions; assign a stable randomized unit aligned to the behavior being measured and maintain consistent treatment during the experiment; that control is assessed by balanced treatment cohorts and clean user-level exposure histories; the incident instead requires shadow-policy agreement and zero write side effects across replayed billing cases.
E:Monitoring without an action threshold can document harm while allowing it to continue; establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; that control is assessed by time from triggered guardrail breach to verified rollback; the incident instead requires shadow-policy agreement and zero write side effects across replayed billing cases.
Question 175
A merger-analysis task spans hundreds of vendor filings with overlapping names and inconsistent reporting dates. What is the most targeted architecture choice?
- Checkpoint long-running work for replay and recovery
- Partition research by source and reconcile conflicts explicitly
- Define input and output contracts for each subtask
- Divide work along independently verifiable artifacts
Explanation
Correct Answer
B:Analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; the final synthesis must expose disagreements rather than average them away; evaluate the design with conflict detection recall and citation coverage across independently processed source sets.
Incorrect Answers
A:Durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects; checkpointing reduces replay cost but cannot correct a defective classification rubric; this alternative would measure successful recovery after injected worker failures without duplicated writes or stages; it does not establish conflict detection recall and citation coverage across independently processed source sets for the described use case.
C:Typed versioned contracts and compatibility tests let stages evolve while preserving their callers' assumptions; contracts define allowed shapes and semantics but cannot ensure a source is truthful; this alternative would measure contract test pass rate for independently upgraded stages and invalid payload rejection; it does not establish conflict detection recall and citation coverage across independently processed source sets for the described use case.
D:Separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries; artifact decomposition controls work structure but does not set the organization-wide release SLA; this alternative would measure stage-specific defect localization and repair without redoing previously validated stages; it does not establish conflict detection recall and citation coverage across independently processed source sets for the described use case.
Question 176
Sending all schemas up front crowds out the case evidence the agent needs to reason over. What is the most direct remediation?
- Fetch detailed schemas only for shortlisted actions
- Use searchable tool discovery for large catalogs
- Load capability groups after intent classification
- Retrieve detailed reference material only when requested
Explanation
Correct Answer
A:Keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; the test balances context savings against input-contract reliability; verify the repair using schema-token consumption and invalid-argument rate after staged loading.
Incorrect Answers
B:Up-front registration of every schema does not improve a request that needs only one domain-specific operation; expose a lean search or discovery interface and load relevant detailed tools only after intent is established; that control is assessed by tool-choice accuracy and context-token savings on varied employee requests; the incident instead requires schema-token consumption and invalid-argument rate after staged loading.
C:Capability-group routing cannot replace policy enforcement once a sensitive tool is selected; select a bounded task-specific capability group after intent routing with conservative fallback for ambiguous requests; that control is assessed by irrelevant tool exposure per intent with no loss of required task completion; the incident instead requires schema-token consumption and invalid-argument rate after staged loading.
D:Reference discovery without source-version checks can still supply outdated interface information; discover resource summaries first and fetch the specific authoritative documentation section at the point of need; that control is assessed by correct endpoint contract use with lower documentation-token overhead; the incident instead requires schema-token consumption and invalid-argument rate after staged loading.
Question 177
A public educational assistant may return sensitive health advice in response to ambiguous wellness requests. What action best fits the business and technical boundary?
- Treat retrieved instructions as untrusted data
- Enforce least privilege where tool actions execute
- Screen unsafe outputs in proportion to the use case
- Apply controls at input, tool execution, and output boundaries
Explanation
Correct Answer
C:Evaluate output safety against a documented risk rubric and use appropriate escalation or refusal for high-risk claims; output-layer tests show whether the actual delivered material meets policy constraints; evaluate the design with harmful-completion detection on labeled outputs with false-positive monitoring.
Incorrect Answers
A:Keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; assuming retrieved text is authoritative because it came from a connector opens a prompt-injection path; this alternative would measure attack success rate on adversarial web pages and zero unauthorized egress calls; it does not establish harmful-completion detection on labeled outputs with false-positive monitoring for the described use case.
B:Enforce role and scope in the backend execution service using credentials that cannot perform the forbidden action; natural-language reminders do not remove capabilities exposed by the API or bearer token; this alternative would measure zero payment writes when tested with the analytics identity and forged instructions; it does not establish harmful-completion detection on labeled outputs with false-positive monitoring for the described use case.
D:Layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority; a single content filter cannot enforce account privileges or validate downstream business transactions; this alternative would measure denial of unauthorized actions and harmful outputs across multi-stage red-team paths; it does not establish harmful-completion detection on labeled outputs with false-positive monitoring for the described use case.
Question 178
A legal knowledge assistant continues operating after launch while policy versions and user questions evolve. Which metric would show that detected quality drift leads to tested corrections without weakening safeguards?
- Successful retirement with zero unplanned dependent-system failures
- Closure of high-risk feasibility questions before design approval
- Resolution of high-severity architecture risks before implementation approval
- Cycle time from detected drift to validated improvement without safety regression
Explanation
Correct Answer
D:Continuous assessment links real usage to controlled change; measure cycle time from detected drift to validated improvement without safety regression at the release gate; feed production samples and feedback into versioned evals, prioritize changes, and release improvements behind regression gates.
Incorrect Answers
A:Dependency-aware migration tests verify that lifecycle closure is complete; inventory consumers, define migration dates and fallback, preserve data obligations, and test the rollback or retirement path; this tests successful retirement with zero unplanned dependent-system failures; the actual release condition is cycle time from detected drift to validated improvement without safety regression.
B:Stage gates prevent costly commitment to untested assumptions; require validated problem scope, value hypothesis, data authority, and feasibility evidence before full design commitment; this tests closure of high-risk feasibility questions before design approval; the actual release condition is cycle time from detected drift to validated improvement without safety regression.
C:Early controls reduce late redesign and exposure; incorporate threat modeling, authorization, data processing, and human oversight into architecture acceptance before implementation hardens; this tests resolution of high-severity architecture risks before implementation approval; the actual release condition is cycle time from detected drift to validated improvement without safety regression.
Question 179
Teams blame the model even though the connector's network round trips account for the additional delay. What design correction resolves the reported failure?
- Sample complete traces strategically under heavy traffic
- Separate retrieval, provider, and tool latency budgets
- Segment reliability metrics by tenant and workload
- Alert on unusual high-impact tool activity
- Correlate model, retrieval, and tool spans in distributed traces
Explanation
Correct Answer
B:Instrument per-stage timing and compare each segment against its allocated latency budget before changing model configuration; budget attribution points remediation at the stage that truly regressed; verify the repair using stage-specific p95 deltas identifying injected bottlenecks.
Incorrect Answers
A:Keeping every raw trace is costly and may violate retention principles without improving detection proportionally; retain error and high-risk traces at a higher rate while sampling successful routine traffic with privacy controls; that control is assessed by capture rate of seeded critical failures at a bounded telemetry cost; the incident instead requires stage-specific p95 deltas identifying injected bottlenecks.
C:Overall averages can hide concentrated failures without meaningful dimensional analysis; break down latency, errors, and tool success by permission-safe tenant and workload dimensions with cardinality controls; that control is assessed by detection time for an injected single-tenant degradation; the incident instead requires stage-specific p95 deltas identifying injected bottlenecks.
D:General chat satisfaction metrics are too delayed and indirect to detect dangerous mutation spikes; monitor write volume, denials, unusual resource patterns, and action-specific thresholds with fast containment routing; that control is assessed by time to detect and stop a simulated unauthorized bulk-write burst; the incident instead requires stage-specific p95 deltas identifying injected bottlenecks.
E:A model-only latency chart cannot explain retrieval or downstream network bottlenecks; end-to-end traces with consistent request and span identifiers identify which service and step consumed time; that control is assessed by trace completeness and correct bottleneck attribution on instrumented fault injections; the incident instead requires stage-specific p95 deltas identifying injected bottlenecks.
Question 180
A support application repeatedly sends an unchanged reference handbook along with short, unrelated customer questions. After changing cache placement, which measurement would demonstrate actual prefix reuse and responsiveness?
- Cache-read input tokens and end-to-end latency over repeated requests
- Task success with reduced irrelevant instruction tokens across mixed workflows
- Cross-application prompt-version alignment and regression pass rate
- Prefix reuse ratio when request metadata changes across calls
Explanation
Correct Answer
A:Cache-hit telemetry verifies the optimization rather than relying on estimated token savings; measure cache-read input tokens and end-to-end latency over repeated requests at the release gate; put reusable instructions and reference material in a consistent prefix and use supported prompt-cache controls while placing variable requests afterward.
Incorrect Answers
B:On-demand loading must retain quality while avoiding unnecessary context overhead; keep Skills discoverable and load task-relevant instructions or resources on demand, maintaining scope and permission constraints; this tests task success with reduced irrelevant instruction tokens across mixed workflows; the actual release condition is cache-read input tokens and end-to-end latency over repeated requests.
C:A shared module succeeds when consuming workflows inherit the same reviewed rule set; maintain a versioned reusable prompt module with explicit parameters and regression tests across consumers; this tests cross-application prompt-version alignment and regression pass rate; the actual release condition is cache-read input tokens and end-to-end latency over repeated requests.
D:Controlled request pairs reveal avoidable cache invalidation; move changing metadata after the stable prefix so cache keys remain reusable across otherwise similar requests; this tests prefix reuse ratio when request metadata changes across calls; the actual release condition is cache-read input tokens and end-to-end latency over repeated requests.