Question 470
A customer-support change increases total resolution but worsens outcomes for users on an accessibility channel. What should the solution architect recommend?
- Predefine success criteria and sample-size assumptions
- Inspect segment-level effects rather than only global lift
- Use shadow evaluation before enabling new side effects
- Release canaries with guardrail metrics and rollback conditions
Explanation
Correct Answer
B:Compare treatment effects across predeclared critical segments and require safeguards where harm would be unacceptable; segment-aware decisions expose concentrated regressions before full rollout; evaluate the design with resolution and safety deltas by channel and accessibility segment.
Incorrect Answers
A:Define primary metrics, minimal meaningful effect, sample-size method, and stopping rules before examining results; repeatedly peeking and stopping at a favorable fluctuation inflates false-positive decisions; this alternative would measure pre-registered experiment compliance and interval estimates at the planned sample size; it does not establish resolution and safety deltas by channel and accessibility segment for the described use case.
C:Replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; a normal A/B test is inappropriate if experimental behavior can cause unreviewed irreversible changes; this alternative would measure shadow-policy agreement and zero write side effects across replayed billing cases; it does not establish resolution and safety deltas by channel and accessibility segment for the described use case.
D:Deploy to a bounded cohort while observing quality, errors, and tail latency, with predefined stop and rollback thresholds; a single offline score cannot show all production interactions or blast radius; this alternative would measure canary threshold compliance and time to revert a seeded regression; it does not establish resolution and safety deltas by channel and accessibility segment for the described use case.
Question 471
A customer-facing service agreement emphasizes fast responses for nearly every request, not just a good average. During a realistic load test, which evidence should determine whether the service meets its promise?
- Incremental resolution lift compared with a comparable non-assisted cohort
- Successful completion of critical workflows during injected model outages
- End-to-end completed cases per paid work-hour with rework and appeal volumes
- End-to-end p95 latency and error rate against the agreed service objective
Explanation
Correct Answer
D:Tail measurement detects the bad experiences that averages can obscure; measure end-to-end p95 latency and error rate against the agreed service objective at the release gate; define a p95 or p99 target against real end-to-end requests, including retrieval and tools, rather than reporting an average model time.
Incorrect Answers
A:A defensible counterfactual is required to credit the architecture with the observed benefit; use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; this tests incremental resolution lift compared with a comparable non-assisted cohort; the actual release condition is end-to-end p95 latency and error rate against the agreed service objective.
B:Fallback drills demonstrate that the business process is not hostage to one generative dependency; separate critical deterministic operations from AI enhancement and provide explicit fallback and manual paths when model service is unavailable; this tests successful completion of critical workflows during injected model outages; the actual release condition is end-to-end p95 latency and error rate against the agreed service objective.
C:Net throughput reflects business output rather than the volume of AI text produced; measure net time saved after review, appeals, and escalations, segmented by complexity, before claiming productivity; this tests end-to-end completed cases per paid work-hour with rework and appeal volumes; the actual release condition is end-to-end p95 latency and error rate against the agreed service objective.
Question 472
A commerce copilot supports orders, returns, catalog, and billing but each task uses only one capability family. Which strategy should be prioritized for this requirement?
- Fetch detailed schemas only for shortlisted actions
- Retrieve detailed reference material only when requested
- Load capability groups after intent classification
- Keep orchestrator intent and policy separate from tool inventories
Explanation
Correct Answer
C:Select a bounded task-specific capability group after intent routing with conservative fallback for ambiguous requests; exposure and completion metrics together reveal whether dynamic loading is too aggressive; evaluate the design with irrelevant tool exposure per intent with no loss of required task completion.
Incorrect Answers
A:Keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; schema-on-demand still needs strict parameter validation at execution time; this alternative would measure schema-token consumption and invalid-argument rate after staged loading; it does not establish irrelevant tool exposure per intent with no loss of required task completion for the described use case.
B:Discover resource summaries first and fetch the specific authoritative documentation section at the point of need; reference discovery without source-version checks can still supply outdated interface information; this alternative would measure correct endpoint contract use with lower documentation-token overhead; it does not establish irrelevant tool exposure per intent with no loss of required task completion for the described use case.
D:Separate goal interpretation and policy decisions from a downstream staged capability-discovery step; tool search does not supply missing user intent or approval; it is only a mechanism for selecting capabilities; this alternative would measure intent-classification accuracy before any tool disclosure or call; it does not establish irrelevant tool exposure per intent with no loss of required task completion for the described use case.
Question 473
A contract library contains numbered sections where a definition and its exception must be read together. What is the most targeted architecture choice?
- Filter retrieval by access control before model exposure
- Assess retrieval quality independently of generation
- Refresh the index when source content changes
- Chunk documents along meaningful structural boundaries
- Preserve citation metadata through retrieval and synthesis
Explanation
Correct Answer
D:Chunk by headings and semantic sections with deliberate overlap where necessary while maintaining parent-document references; structure-aware test cases measure preservation of legally relevant context; evaluate the design with retrieval of both governing clause and exception for labeled contract questions.
Incorrect Answers
A:Apply document permissions to retrieval candidates before material enters model context or logs; a model instruction to ignore confidential items is not a data-access boundary once the text is supplied; this alternative would measure unauthorized chunk exposure rate under staff permission matrices, targeting zero; it does not establish retrieval of both governing clause and exception for labeled contract questions for the described use case.
B:Measure retrieval recall, ranking relevance, and evidence coverage on labeled query-document pairs before optimizing generation; a stronger model cannot reliably cite evidence that retrieval never supplies; this alternative would measure recall at k and evidence-coverage rate for known-answer queries; it does not establish retrieval of both governing clause and exception for labeled contract questions for the described use case.
C:Build a monitored ingestion path for adds, updates, deletes, and effective dates, with freshness checks and removal of superseded chunks; a larger context window cannot compensate for documents missing from the index; this alternative would measure publication-to-search freshness lag and absence of withdrawn clauses in results; it does not establish retrieval of both governing clause and exception for labeled contract questions for the described use case.
E:Store source IDs, locations, dates, and stable links with chunks and propagate them to answer citation validation; a similarity score is not a source identifier and cannot support a traceable citation; this alternative would measure fraction of material claims linked to resolvable source locations; it does not establish retrieval of both governing clause and exception for labeled contract questions for the described use case.
Question 474
Executives repeat a precise number from a confident AI report, but no retrieved source supports that number. Which safeguard should be applied to factual numeric claims?
- Use claim verification to mitigate confident fabrication
- Cap agent steps and resource expenditure
- Constrain outbound channels to prevent data exfiltration
- Validate tool-result provenance and semantics
Explanation
Correct Answer
A:Require support from retrieved source passages and provide an explicit unknown path for unsupported claims; grounding tests detect confident fabrication at the statement level; verify the repair using proportion of numeric claims supported by independently inspected references.
Incorrect Answers
B:Asking the model to 'be efficient' does not guarantee bounded autonomous execution; set per-task step, token, cost, and elapsed-time budgets with a clear abort and escalation state; that control is assessed by maximum resource usage and orderly termination on deliberately unsolvable tasks; the incident instead requires proportion of numeric claims supported by independently inspected references.
C:Redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; that control is assessed by blocked unauthorized destinations and absence of confidential fields in egress logs; the incident instead requires proportion of numeric claims supported by independently inspected references.
D:A successful HTTP status does not mean returned data has decision authority; preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; that control is assessed by zero external-result fields promoted to internal approvals in test cases; the incident instead requires proportion of numeric claims supported by independently inspected references.
Question 475
Two document-extraction designs have different token charges and downstream correction loads. Finance needs a unit economics comparison based on usable completed work. Which measure should decide the release?
- p95 time from user submission to fully validated answer under load
- Severity-weighted policy violation incidence on a reviewed adversarial test set
- Fully loaded cost per correctly completed document
- Critical-class precision and recall on independently verified tax-code cases
Explanation
Correct Answer
C:This metric includes rework rather than rewarding low-quality cheap drafts; measure fully loaded cost per correctly completed document at the release gate; relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted.
Incorrect Answers
A:Measuring the completed workflow captures actual end-user waiting, including downstream services; report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; this tests p95 time from user submission to fully validated answer under load; the actual release condition is fully loaded cost per correctly completed document.
B:Risk-weighted scoring detects rare critical violations that a single average hides; define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; this tests severity-weighted policy violation incidence on a reviewed adversarial test set; the actual release condition is fully loaded cost per correctly completed document.
D:Label-based metrics catch mistakes that surface-level satisfaction may conceal; measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; this tests critical-class precision and recall on independently verified tax-code cases; the actual release condition is fully loaded cost per correctly completed document.
Question 476
Authors trust the fluent wording despite citations that do not contain the asserted numbers. What is the most direct remediation?
- Test for model capability mismatch on complex reasoning
- Separate external tool faults from model decisions
- Verify whether claims are supported by supplied sources
- Investigate retrieval recall before changing the language model
- Trace malformed outputs to prompt or schema boundaries
Explanation
Correct Answer
C:Compare material claims against cited evidence and flag unsupported statements for abstention or review; claim-level verification targets hallucinations that ordinary coherence checks miss; verify the repair using supported-claim rate on source-checked summaries with seeded unsupported numbers.
Incorrect Answers
A:Rebuilding retrieval is unlikely to help when required evidence was already consistently supplied; hold evidence constant and compare validated model capabilities or staged reasoning designs on the failed complexity class; that control is assessed by correct conflict resolution with fixed retrieved evidence across candidate reasoning configurations; the incident instead requires supported-claim rate on source-checked summaries with seeded unsupported numbers.
B:Prompt changes alone cannot restore a failing service or make an empty placeholder a valid business result; propagate typed tool errors, retries, and unknown states instead of converting transport failures into negative facts; that control is assessed by correct 'unknown' handling and no false denials during injected API failures; the incident instead requires supported-claim rate on source-checked summaries with seeded unsupported numbers.
D:Model upgrades cannot recover information that never reaches the answer context; inspect search coverage, index freshness, filters, and chunk retrieval before attributing the regression to Claude's reasoning; that control is assessed by recall of newly published policy passages and removal of stale hits; the incident instead requires supported-claim rate on source-checked summaries with seeded unsupported numbers.
E:Switching to a more expensive model may leave the consumer's missing validation problem intact; inspect instruction clarity, output-schema validation, and edge-case examples before assuming a core model reasoning failure; that control is assessed by schema error rate across multiline, null, and unusual-field edge cases; the incident instead requires supported-claim rate on source-checked summaries with seeded unsupported numbers.
Question 477
A prompt engineering team has repeatedly optimized against a visible development set. Before publishing another revision, which evidence would show that improvements hold on previously unseen examples?
- Zero schema violations accepted by the downstream consumer across generated cases
- Inter-annotator agreement and unresolved-label rate after adjudication
- Holdout quality stability across successive prompt versions
- Per-stratum pass rates with no uncovered critical scenario classes
Explanation
Correct Answer
C:Untouched data estimates whether improvements transfer beyond the examples used during tuning; measure holdout quality stability across successive prompt versions at the release gate; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls.
Incorrect Answers
A:Executable contract tests deliver reproducible feedback at scale; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; this tests zero schema violations accepted by the downstream consumer across generated cases; the actual release condition is holdout quality stability across successive prompt versions.
B:Label reliability is necessary before model score differences can be interpreted; use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; this tests inter-annotator agreement and unresolved-label rate after adjudication; the actual release condition is holdout quality stability across successive prompt versions.
D:Stratified coverage prevents abundant easy cases from hiding unsupported workflows; build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; this tests per-stratum pass rates with no uncovered critical scenario classes; the actual release condition is holdout quality stability across successive prompt versions.
Question 478
A business-critical approvals service has a manual path for outages. In a controlled resilience drill, the model and retrieval backends become unavailable. Which result demonstrates acceptable continuity?
- End-to-end completed cases per paid work-hour with rework and appeal volumes
- Successful completion of critical workflows during injected model outages
- Quality and total cost segmented by request risk tier under a fixed accuracy floor
- Incremental resolution lift compared with a comparable non-assisted cohort
Explanation
Correct Answer
B:Fallback drills demonstrate that the business process is not hostage to one generative dependency; measure successful completion of critical workflows during injected model outages at the release gate; separate critical deterministic operations from AI enhancement and provide explicit fallback and manual paths when model service is unavailable.
Incorrect Answers
A:Net throughput reflects business output rather than the volume of AI text produced; measure net time saved after review, appeals, and escalations, segmented by complexity, before claiming productivity; this tests end-to-end completed cases per paid work-hour with rework and appeal volumes; the actual release condition is successful completion of critical workflows during injected model outages.
C:Segmented evaluation reveals whether additional reasoning spend is targeted at cases that benefit; route by validated task complexity and consequence, using stronger reasoning or human review where evidence justifies the incremental spend; this tests quality and total cost segmented by request risk tier under a fixed accuracy floor; the actual release condition is successful completion of critical workflows during injected model outages.
D:A defensible counterfactual is required to credit the architecture with the observed benefit; use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; this tests incremental resolution lift compared with a comparable non-assisted cohort; the actual release condition is successful completion of critical workflows during injected model outages.
Question 479
A workflow planner needs to decide the user goal before exploring available execution capabilities. What action best fits the business and technical boundary?
- Keep orchestrator intent and policy separate from tool inventories
- Preserve stable prefixes while dynamically discovering tools
- Fetch detailed schemas only for shortlisted actions
- Adopt searchable tool discovery for large catalogs
Explanation
Correct Answer
A:Separate goal interpretation and policy decisions from a downstream staged capability-discovery step; phase separation verifies that the assistant understands the task before acting; evaluate the design with intent-classification accuracy before any tool disclosure or call.
Incorrect Answers
B:Isolate stable instructions from changing discovery payloads and measure cache behavior when tool inventories vary; progressive discovery can lose its token advantage if dynamic tool definitions repeatedly destroy cache reuse; this alternative would measure cache-read tokens and total context cost while changing selected tool subsets; it does not establish intent-classification accuracy before any tool disclosure or call for the described use case.
C:Keep concise discovery metadata initially and retrieve the precise argument schema when a tool becomes actionable; schema-on-demand still needs strict parameter validation at execution time; this alternative would measure schema-token consumption and invalid-argument rate after staged loading; it does not establish intent-classification accuracy before any tool disclosure or call for the described use case.
D:Expose a lean search or discovery interface and load relevant detailed tools only after intent is established; up-front registration of every schema does not improve a request that needs only one domain-specific operation; this alternative would measure tool-choice accuracy and context-token savings on varied employee requests; it does not establish intent-classification accuracy before any tool disclosure or call for the described use case.
Question 480
An enterprise has proposed an expensive automation program but is still uncertain about upstream data access and extraction feasibility. Which discovery-gate result would justify committing to detailed solution design?
- Named owner coverage and support-drill completion at handoff
- Successful retirement with zero unplanned dependent-system failures
- Cycle time from detected drift to validated improvement without safety regression
- Closure of high-risk feasibility questions before design approval
Explanation
Correct Answer
D:Stage gates prevent costly commitment to untested assumptions; measure closure of high-risk feasibility questions before design approval at the release gate; require validated problem scope, value hypothesis, data authority, and feasibility evidence before full design commitment.
Incorrect Answers
A:Ownership evidence shows the service can be maintained after consultants leave; establish production owners for reliability, security, spend, data, prompts, and escalation, with verified access and runbooks; this tests named owner coverage and support-drill completion at handoff; the actual release condition is closure of high-risk feasibility questions before design approval.
B:Dependency-aware migration tests verify that lifecycle closure is complete; inventory consumers, define migration dates and fallback, preserve data obligations, and test the rollback or retirement path; this tests successful retirement with zero unplanned dependent-system failures; the actual release condition is closure of high-risk feasibility questions before design approval.
C:Continuous assessment links real usage to controlled change; feed production samples and feedback into versioned evals, prioritize changes, and release improvements behind regression gates; this tests cycle time from detected drift to validated improvement without safety regression; the actual release condition is closure of high-risk feasibility questions before design approval.