Question 360
A subscription service reports temporary upstream errors during eligibility checks. The test suite distinguishes missing facts from unavailable services. Which result would prove that technical failures do not become false customer denials?
- Supported-claim rate on source-checked summaries with seeded unsupported numbers
- Recall of newly published policy passages and removal of stale hits
- Correct conflict resolution with fixed retrieved evidence across candidate reasoning configurations
- Correct 'unknown' handling and no false denials during injected API failures
Explanation
Correct Answer
D:Fault injection distinguishes tool availability from genuine absence of data; measure correct 'unknown' handling and no false denials during injected API failures at the release gate; propagate typed tool errors, retries, and unknown states instead of converting transport failures into negative facts.
Incorrect Answers
A:Claim-level verification targets hallucinations that ordinary coherence checks miss; compare material claims against cited evidence and flag unsupported statements for abstention or review; this tests supported-claim rate on source-checked summaries with seeded unsupported numbers; the actual release condition is correct 'unknown' handling and no false denials during injected API failures.
B:A retrieval-specific test isolates the component that changed; inspect search coverage, index freshness, filters, and chunk retrieval before attributing the regression to Claude's reasoning; this tests recall of newly published policy passages and removal of stale hits; the actual release condition is correct 'unknown' handling and no false denials during injected API failures.
C:Controlled component isolation determines whether the bottleneck is reasoning rather than retrieval; hold evidence constant and compare validated model capabilities or staged reasoning designs on the failed complexity class; this tests correct conflict resolution with fixed retrieved evidence across candidate reasoning configurations; the actual release condition is correct 'unknown' handling and no false denials during injected API failures.
Question 361
An analytics assistant has a service identity intended only to read reconciliation data. Testers inject transfer requests into the retrieved source content. Which result would demonstrate that the payment endpoint remains unreachable?
- Denial of unauthorized actions and harmful outputs across multi-stage red-team paths
- Attack success rate on adversarial web pages and zero unauthorized egress calls
- Zero payment writes when tested with the analytics identity and forged instructions
- Zero unapproved recipients or fields accepted by the outbound service
Explanation
Correct Answer
C:Direct permission tests verify security independent of model obedience; measure zero payment writes when tested with the analytics identity and forged instructions at the release gate; enforce role and scope in the backend execution service using credentials that cannot perform the forbidden action.
Incorrect Answers
A:End-to-end adversarial cases assess the combined defenses and their failure isolation; layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority; this tests denial of unauthorized actions and harmful outputs across multi-stage red-team paths; the actual release condition is zero payment writes when tested with the analytics identity and forged instructions.
B:Simulated hostile documents test the trust boundary directly; keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; this tests attack success rate on adversarial web pages and zero unauthorized egress calls; the actual release condition is zero payment writes when tested with the analytics identity and forged instructions.
D:Contract-level checks demonstrate control over the actual data leaving the system; validate destinations, argument types, sensitive fields, and authorization before dispatch; reject values outside the approved contract; this tests zero unapproved recipients or fields accepted by the outbound service; the actual release condition is zero payment writes when tested with the analytics identity and forged instructions.
Question 362
The golden dataset contains conflicting answer keys and every model looks unreliable on the ambiguous rows. What is the most direct remediation?
- Protect a separate holdout set for regression decisions
- Use deterministic code checks for objective output contracts
- Measure human annotator consistency before trusting labels
- Include hostile prompt-injection and no-answer probes
Explanation
Correct Answer
C:Use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; label reliability is necessary before model score differences can be interpreted; verify the repair using inter-annotator agreement and unresolved-label rate after adjudication.
Incorrect Answers
A:Reusing a public tuning set as the sole release gate invites overfitting rather than generalization; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; that control is assessed by holdout quality stability across successive prompt versions; the incident instead requires inter-annotator agreement and unresolved-label rate after adjudication.
B:Subjective model grading is unnecessary and less reliable for an objectively checkable parser contract; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; that control is assessed by zero schema violations accepted by the downstream consumer across generated cases; the incident instead requires inter-annotator agreement and unresolved-label rate after adjudication.
D:Testing normal prompts alone cannot estimate resilience to malicious or unsatisfiable requests; add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; that control is assessed by attack success rate and safe handling of intentionally unanswerable prompts; the incident instead requires inter-annotator agreement and unresolved-label rate after adjudication.
Question 363
A generalized multi-agent connector stack creates unnecessary negotiation and operational complexity for a single call. How should the design be corrected?
- Adopt agent-to-agent exchange for independently owned specialist systems
- Use MCP to expose reusable tools and resources across compatible clients
- Use direct APIs for a narrow stable application integration
- Use a controlled CLI interface for developer automation
Explanation
Correct Answer
C:A direct typed API client is appropriate when the interface, arguments, and control flow are known and stable; a stable-contract benchmark confirms that extra orchestration is unnecessary; verify the repair using integration error rate and p95 latency on the fixed status-check path.
Incorrect Answers
A:Messaging between agents does not make their outputs trustworthy or authorize cross-system actions automatically; an agent-to-agent contract can exchange bounded tasks and outputs while preserving local autonomy, identity, and authority of each specialist system; that control is assessed by successful authenticated task exchange with each agent retaining its own permission checks; the incident instead requires integration error rate and p95 latency on the fixed status-check path.
B:MCP is an interface protocol, not a substitute for access policy or secure server implementation; a maintained MCP server can standardize discoverable tool and resource contracts while each client still enforces authentication and trust; that control is assessed by cross-client contract compatibility and unauthorized-action denial for the same MCP service; the incident instead requires integration error rate and p95 latency on the fixed status-check path.
D:General CLI execution is dangerous when exposed to untrusted instructions without sandboxing and command policy; a bounded CLI execution environment can reuse standard developer tools with explicit directory and command permissions; that control is assessed by reproducible build results with denied commands outside the approved allowlist; the incident instead requires integration error rate and p95 latency on the fixed status-check path.
Question 364
A claims AI initiative has a sponsor's efficiency target but no input from frontline adjusters, legal reviewers, or data owners. What should the solution architect recommend?
- Validate risky assumptions through narrow experiments
- Capture external dependencies and acceptance risks explicitly
- Interview each role that owns a distinct outcome or risk
- Map the owner and authority of every required data source
Explanation
Correct Answer
C:Use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders; traceable requirements reveal whether important operational perspectives were omitted; evaluate the design with coverage of approved requirements traced to affected stakeholder groups.
Incorrect Answers
A:Identify high-risk assumptions and design small feasibility spikes with explicit success and stop criteria; detailed architecture diagrams do not prove uncertain input quality or model capability; this alternative would measure resolution rate of prioritized unknowns through measured prototypes; it does not establish coverage of approved requirements traced to affected stakeholder groups for the described use case.
B:Maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; an AI model benchmark cannot predict every upstream system's production behavior; this alternative would measure verified readiness of high-impact external dependencies against test quotas and outage cases; it does not establish coverage of approved requirements traced to affected stakeholder groups for the described use case.
D:Create a source inventory with accountable owners, sensitivity, update rate, and approval dependencies before committing to the design; a list of desired features cannot grant access or establish source truthfulness; this alternative would measure percentage of required sources with approved owners, legal uses, and integration paths; it does not establish coverage of approved requirements traced to affected stakeholder groups for the described use case.
Question 365
Claude cites a pre-fix error count even though newer diagnostics are available in the same swollen context. What design correction resolves the reported failure?
- Attach provenance metadata to selected context
- Remove obsolete tool output while preserving current evidence
- Allocate input and output token budgets per workflow step
- Retrieve only relevant material instead of full archives
Explanation
Correct Answer
B:Prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; freshness-aware tests expose whether stale tool observations override newer facts; verify the repair using current-state citation accuracy across log-update sequences.
Incorrect Answers
A:Provenance metadata does not guarantee the upstream search selected the right document in the first place; supply source identity, version, effective date, and retrieval relevance alongside each chunk so the model can attribute evidence correctly; that control is assessed by citation correctness by effective-date version in temporal regression questions; the incident instead requires current-state citation accuracy across log-update sequences.
C:Merely expanding every context window can increase cost while leaving output truncation unaddressed; reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; that control is assessed by complete mandatory report sections under the worst supported input size; the incident instead requires current-state citation accuracy across log-update sequences.
D:Broad context stuffing can degrade relevance and cost even when a large context window is available; apply query-driven retrieval and evidence selection before prompting Claude, retaining citations and permission filters; that control is assessed by answer support and retrieved-token volume on a sampled policy-question set; the incident instead requires current-state citation accuracy across log-update sequences.
Question 366
A benefits advisor should answer policy-supported questions but avoid guesses when the documents contain no applicable rule. A separate consumer helpdesk must measure serious unsafe responses under hostile prompts. Which TWO evaluation measures address calibrated coverage and safety violations? Choose TWO.
- Measure rate and severity of safety policy violations
- Use task-correctness metrics tied to verified labels
- Test whether the tool layer denies unauthorized requests
- Use tail-latency indicators for interactive promises
- Evaluate whether abstention and escalation are calibrated
Explanation
Correct Answers
A:Define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; risk-weighted scoring detects rare critical violations that a single average hides; its corresponding workstream should be validated by severity-weighted policy violation incidence on a reviewed adversarial test set.
E:Measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; coverage-risk curves expose the tradeoff between reliability and unnecessary handoffs; its corresponding workstream should be validated by correct-answer rate at each supported coverage threshold including no-answer cases.
Incorrect Answers
B:Measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; positive conversational tone does not establish that a legally or financially meaningful label is correct; its natural acceptance measure is critical-class precision and recall on independently verified tax-code cases, which does not satisfy the separate workstreams being assessed.
C:Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; model refusal scores can be high even when the underlying API is vulnerable; its natural acceptance measure is unauthorized API requests rejected under role, tenant, and resource attack matrices, which does not satisfy the separate workstreams being assessed.
D:Report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; mean latency underweights rare long waits and omits retrieval and external-tool delays; its natural acceptance measure is p95 time from user submission to fully validated answer under load, which does not satisfy the separate workstreams being assessed.
Question 367
A security support agent has thousands of easy helpdesk cases and a few high-impact threat scenarios. The evaluation plan must prevent strong common-case results from masking missing risk classes. Which test coverage result matters most?
- Expert-rated factual fidelity and calibrated grader agreement on sampled explanations
- Per-stratum pass rates with no uncovered critical scenario classes
- Inter-annotator agreement and unresolved-label rate after adjudication
- Holdout quality stability across successive prompt versions
Explanation
Correct Answer
B:Stratified coverage prevents abundant easy cases from hiding unsupported workflows; measure per-stratum pass rates with no uncovered critical scenario classes at the release gate; build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases.
Incorrect Answers
A:A semantic rubric tests meaning while allowing legitimate variation in language; use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; this tests expert-rated factual fidelity and calibrated grader agreement on sampled explanations; the actual release condition is per-stratum pass rates with no uncovered critical scenario classes.
C:Label reliability is necessary before model score differences can be interpreted; use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; this tests inter-annotator agreement and unresolved-label rate after adjudication; the actual release condition is per-stratum pass rates with no uncovered critical scenario classes.
D:Untouched data estimates whether improvements transfer beyond the examples used during tuning; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; this tests holdout quality stability across successive prompt versions; the actual release condition is per-stratum pass rates with no uncovered critical scenario classes.
Question 368
A team upgrades its model and prompt instructions but retains unlabeled output metrics from older versions. What should the solution architect recommend?
- Keep an end-to-end data-flow and trust-boundary diagram
- Version prompts, model routes, tools, and evaluation artifacts together
- Document decisions with assumptions and reversal triggers
- Publish an incident runbook with containment and rollback steps
Explanation
Correct Answer
B:Maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; versioned provenance enables defensible debugging and rollback; evaluate the design with reproducibility of a sampled historical answer using its full manifest.
Incorrect Answers
A:A current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; a feature list does not reveal crossing of sensitive trust boundaries; this alternative would measure completeness of sensitive-data flow paths identified during security review; it does not establish reproducibility of a sampled historical answer using its full manifest for the described use case.
C:Record architectural context, alternatives, decision rationale, constraints, and explicit conditions for revisiting choices; configuration exports show what exists but rarely why it was chosen; this alternative would measure ability to identify affected decisions after simulated policy and workload changes; it does not establish reproducibility of a sampled historical answer using its full manifest for the described use case.
D:Maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; a high-level diagram alone is not an executable incident response procedure; this alternative would measure successful containment time in a simulated stale-index incident; it does not establish reproducibility of a sampled historical answer using its full manifest for the described use case.
Question 369
The team keeps applying ad hoc fixes in production without a reliable prior configuration to restore. What is the most direct remediation?
- Roll back to a known-good tool and prompt release
- Distinguish model selection faults from tool execution failures
- Produce incident learning actions with accountable follow-up
- Inspect permission precedence when a tool is unexpectedly denied
Explanation
Correct Answer
A:Maintain versioned releases and documented rollback steps for hooks, prompts, model routes, and connector configuration; a rollback drill verifies that recovery does not depend on the original developer being online; verify the repair using time to restore normal operation using an approved previous release.
Incorrect Answers
B:A more capable model cannot read data from a tool that never returns successfully; classify tool errors separately from model responses, validate return codes, and probe the failing integration before changing prompts; that control is assessed by correct root-cause attribution under injected search timeouts; the incident instead requires time to restore normal operation using an approved previous release.
C:Restarting the service restores availability but does not reduce the recurrence probability; run blameless root-cause analysis, link corrective actions to owners, and add regression or operational safeguards before closing the incident; that control is assessed by recurrence rate of the diagnosed failure and completion of preventive actions; the incident instead requires time to restore normal operation using an approved previous release.
D:Editing a user preference cannot override a stronger centrally managed deny rule; review organization, project, and local permission layers in the documented precedence order before weakening security controls; that control is assessed by correct identification of enforced deny source without allowing prohibited commands; the incident instead requires time to restore normal operation using an approved previous release.
Question 370
Reviewers cannot isolate behavior changes and rollbacks become unsafe when one small contract test fails. Which action best removes the underlying failure mode?
- Run reproducible CI gates on assisted code
- Break large refactors into independently reviewable changes
- Package reusable engineering instructions as governed Skills
- Use test-first acceptance cases for generated changes
Explanation
Correct Answer
B:Use small scoped changes, compatibility checkpoints, and separate pull requests tied to specific tests; small-batch evidence shows whether modernization remains controlled and reversible; verify the repair using review time and defect localization per incremental change set.
Incorrect Answers
A:Local success can be environment-dependent and should not be the only release assurance; execute standardized linting, type checks, tests, and security scanning in versioned CI environments before merges; that control is assessed by cI pass reproducibility on clean builds with locked dependencies; the incident instead requires review time and defect localization per incremental change set.
C:Shared instructions improve consistency but do not replace enforced approval or automated tests; maintain task-scoped reusable Skills or playbooks with versioning, ownership, and documented prerequisites; that control is assessed by checklist completeness across independent sessions using the reviewed Skill version; the incident instead requires review time and defect localization per incremental change set.
D:Merely reading generated code for plausibility does not replace tests of business invariants; define executable tests and expected behavior before relying on AI-generated implementation, then review failures and iterate; that control is assessed by passing unit and property tests including specified rounding boundaries; the incident instead requires review time and defect localization per incremental change set.