Question 670
A business owner assumes the model provider will manage its proprietary knowledge-base permissions and human escalation. What is the most targeted architecture choice?
- Route user feedback into a triaged, owned backlog
- Communicate release behavior changes before rollout
- Establish a documented escalation path for service incidents
- Clarify shared responsibility for the service boundary
Explanation
Correct Answer
D:Define provider, platform, application, security, and business-owner responsibilities in an explicit operating model; responsibility tests prevent gaps between contractual expectations and implemented controls; evaluate the design with ownership coverage for failures across model, application, data, and tool layers.
Incorrect Answers
A:Classify feedback by severity and root-cause type, assign owners, link trace IDs, and feed validated cases into regression tests; unstructured rating totals cannot by themselves direct a safe corrective action; this alternative would measure time from actionable feedback to triage, owner assignment, and verified fix; it does not establish ownership coverage for failures across model, application, data, and tool layers for the described use case.
B:Define release notes, behavior-change notices, affected audiences, and support channels proportional to the deployment's impact; quiet deployment may reduce coordination overhead but damages trust when workflows or permissions change; this alternative would measure affected-user awareness and support-ticket volume following staged changes; it does not establish ownership coverage for failures across model, application, data, and tool layers for the described use case.
C:Define incident severity, on-call ownership, manual fallback authority, and customer communication triggers; generic 'contact support' guidance lacks accountable timing and action for critical failures; this alternative would measure time to appropriate escalation and fallback activation during drills; it does not establish ownership coverage for failures across model, application, data, and tool layers for the described use case.
Question 671
An untrusted service labels a result 'approved' and the assistant treats it as an internal authorization decision. Which action best removes the underlying failure mode?
- Use claim verification to mitigate confident fabrication
- Cap agent steps and resource expenditure
- Constrain outbound channels to prevent data exfiltration
- Counter automation bias with visible uncertainty and review
- Validate tool-result provenance and semantics
Explanation
Correct Answer
E:Preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; provenance-sensitive checks stop evidence laundering through connector outputs; verify the repair using zero external-result fields promoted to internal approvals in test cases.
Incorrect Answers
A:Fluency and citation formatting cannot establish factual support on their own; require support from retrieved source passages and provide an explicit unknown path for unsupported claims; that control is assessed by proportion of numeric claims supported by independently inspected references; the incident instead requires zero external-result fields promoted to internal approvals in test cases.
B:Asking the model to 'be efficient' does not guarantee bounded autonomous execution; set per-task step, token, cost, and elapsed-time budgets with a clear abort and escalation state; that control is assessed by maximum resource usage and orderly termination on deliberately unsolvable tasks; the incident instead requires zero external-result fields promoted to internal approvals in test cases.
C:Redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; that control is assessed by blocked unauthorized destinations and absence of confidential fields in egress logs; the incident instead requires zero external-result fields promoted to internal approvals in test cases.
D:A nominal human checkpoint is weak when the interface encourages blind confirmation; design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; that control is assessed by reviewer error interception on seeded incorrect recommendations; the incident instead requires zero external-result fields promoted to internal approvals in test cases.
Question 672
After a retrieval pipeline upgrade, the answer team wants to diagnose whether failures arise before or after Claude sees the evidence. Which measurement would establish that relevant supporting passages are reaching the generation stage?
- Retrieval recall before and after a complete embedding-index migration
- Retrieval of both governing clause and exception for labeled contract questions
- Recall at k and evidence-coverage rate for known-answer queries
- Publication-to-search freshness lag and absence of withdrawn clauses in results
- Unauthorized chunk exposure rate under staff permission matrices, targeting zero
Explanation
Correct Answer
C:Separating retrieval metrics identifies which pipeline stage needs repair; measure recall at k and evidence-coverage rate for known-answer queries at the release gate; measure retrieval recall, ranking relevance, and evidence coverage on labeled query-document pairs before optimizing generation.
Incorrect Answers
A:Version-aligned search tests detect vector-compatibility regressions; record embedding model and preprocessing versions, re-embed consistently, and use controlled index migration; this tests retrieval recall before and after a complete embedding-index migration; the actual release condition is recall at k and evidence-coverage rate for known-answer queries.
B:Structure-aware test cases measure preservation of legally relevant context; chunk by headings and semantic sections with deliberate overlap where necessary while maintaining parent-document references; this tests retrieval of both governing clause and exception for labeled contract questions; the actual release condition is recall at k and evidence-coverage rate for known-answer queries.
D:Update/delete tests prove the index tracks the authoritative corpus; build a monitored ingestion path for adds, updates, deletes, and effective dates, with freshness checks and removal of superseded chunks; this tests publication-to-search freshness lag and absence of withdrawn clauses in results; the actual release condition is recall at k and evidence-coverage rate for known-answer queries.
E:The security test must inspect retrieved inputs, not just final visible answers; apply document permissions to retrieval candidates before material enters model context or logs; this tests unauthorized chunk exposure rate under staff permission matrices, targeting zero; the actual release condition is recall at k and evidence-coverage rate for known-answer queries.
Question 673
A global accuracy score remains high despite serious failures for the rare incident category. What is the most direct remediation?
- Use expert rubrics for nuanced qualitative outputs
- Use deterministic code checks for objective output contracts
- Protect a separate holdout set for regression decisions
- Stratify a golden dataset by real workload segments
Explanation
Correct Answer
D:Build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; stratified coverage prevents abundant easy cases from hiding unsupported workflows; verify the repair using per-stratum pass rates with no uncovered critical scenario classes.
Incorrect Answers
A:Pure string comparison is unsuitable when many substantively correct phrasings exist; use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; that control is assessed by expert-rated factual fidelity and calibrated grader agreement on sampled explanations; the incident instead requires per-stratum pass rates with no uncovered critical scenario classes.
B:Subjective model grading is unnecessary and less reliable for an objectively checkable parser contract; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; that control is assessed by zero schema violations accepted by the downstream consumer across generated cases; the incident instead requires per-stratum pass rates with no uncovered critical scenario classes.
C:Reusing a public tuning set as the sole release gate invites overfitting rather than generalization; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; that control is assessed by holdout quality stability across successive prompt versions; the incident instead requires per-stratum pass rates with no uncovered critical scenario classes.
Question 674
Several support teams maintain copied but diverging versions of the same troubleshooting prompt. What should the solution architect recommend?
- Load specialized Skills only when the task needs them
- Trace cache effectiveness with token usage counters
- Keep dynamic fields out of cached prefix content
- Modularize shared task instructions for consistent reuse
Explanation
Correct Answer
D:Maintain a versioned reusable prompt module with explicit parameters and regression tests across consumers; a shared module succeeds when consuming workflows inherit the same reviewed rule set; evaluate the design with cross-application prompt-version alignment and regression pass rate.
Incorrect Answers
A:Keep Skills discoverable and load task-relevant instructions or resources on demand, maintaining scope and permission constraints; skills provide reusable guidance, not a separate security boundary or a substitute for approved tool permissions; this alternative would measure task success with reduced irrelevant instruction tokens across mixed workflows; it does not establish cross-application prompt-version alignment and regression pass rate for the described use case.
B:Compare cache-creation and cache-read token telemetry and inspect prefix ordering, eligible size, and reuse timing; a faster model might improve latency but would not explain a missing cache hit on identical prompts; this alternative would measure sustained cache-read token share under repeated workload replay; it does not establish cross-application prompt-version alignment and regression pass rate for the described use case.
C:Move changing metadata after the stable prefix so cache keys remain reusable across otherwise similar requests; a cache breakpoint cannot compensate for changes to earlier content in the hashed prefix; this alternative would measure prefix reuse ratio when request metadata changes across calls; it does not establish cross-application prompt-version alignment and regression pass rate for the described use case.
Question 675
A manufacturing assistant depends on a vendor ERP API that has strict rate limits and periodic maintenance windows. Which strategy should be prioritized for this requirement?
- Document nonfunctional requirements as measurable thresholds
- Map the owner and authority of every required data source
- Interview each role that owns a distinct outcome or risk
- Capture external dependencies and acceptance risks explicitly
Explanation
Correct Answer
D:Maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; dependency validation closes gaps between prototype success and operational feasibility; evaluate the design with verified readiness of high-impact external dependencies against test quotas and outage cases.
Incorrect Answers
A:Specify capacity, tail latency, availability, privacy, and recovery objectives with measurement windows and ownership; functional task lists do not tell designers what constitutes acceptable production operation; this alternative would measure agreed p95, availability, and load thresholds exercised in an acceptance environment; it does not establish verified readiness of high-impact external dependencies against test quotas and outage cases for the described use case.
B:Create a source inventory with accountable owners, sensitivity, update rate, and approval dependencies before committing to the design; a list of desired features cannot grant access or establish source truthfulness; this alternative would measure percentage of required sources with approved owners, legal uses, and integration paths; it does not establish verified readiness of high-impact external dependencies against test quotas and outage cases for the described use case.
C:Use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders; a sponsor's high-level vision does not fully describe operational controls or edge cases; this alternative would measure coverage of approved requirements traced to affected stakeholder groups; it does not establish verified readiness of high-impact external dependencies against test quotas and outage cases for the described use case.
Question 676
Several internal Claude applications need the same customer-directory lookup and resource definitions. Which strategy should be prioritized for this requirement?
- Prefer managed server-side tools when appropriate to the platform
- Adopt event notifications to drive state changes rather than constant polling
- Use MCP to expose reusable tools and resources across compatible clients
- Use direct APIs for a narrow stable application integration
- Use a controlled CLI interface for developer automation
Explanation
Correct Answer
C:A maintained MCP server can standardize discoverable tool and resource contracts while each client still enforces authentication and trust; testing both interoperability and permission enforcement captures the two essential concerns; evaluate the design with cross-client contract compatibility and unauthorized-action denial for the same MCP service.
Incorrect Answers
A:A supported server-executed tool can offload execution to the provider while the application retains usage, privacy, and result-validation responsibilities; hosted tools do not automatically make external search content authoritative or safe from prompt injection; this alternative would measure search coverage, tool cost, and provenance checks with no application-owned execution endpoint; it does not establish cross-client contract compatibility and unauthorized-action denial for the same MCP service for the described use case.
B:An authenticated event or webhook integration can trigger bounded processing only when upstream state changes; events still require deduplication, retries, and authorization checks to avoid duplicate downstream effects; this alternative would measure event-to-index update delay and duplicate-event idempotency under replay; it does not establish cross-client contract compatibility and unauthorized-action denial for the same MCP service for the described use case.
D:A direct typed API client is appropriate when the interface, arguments, and control flow are known and stable; direct API calls offer less automatic discovery when many heterogeneous tools must be used by several assistants; this alternative would measure integration error rate and p95 latency on the fixed status-check path; it does not establish cross-client contract compatibility and unauthorized-action denial for the same MCP service for the described use case.
E:A bounded CLI execution environment can reuse standard developer tools with explicit directory and command permissions; general CLI execution is dangerous when exposed to untrusted instructions without sandboxing and command policy; this alternative would measure reproducible build results with denied commands outside the approved allowlist; it does not establish cross-client contract compatibility and unauthorized-action denial for the same MCP service for the described use case.
Question 677
An audit assistant must distinguish a remediation recommendation from a statement of observed control evidence. What should the solution architect recommend?
- Begin with direct zero-shot instructions for unambiguous tasks
- Add few-shot examples for difficult boundary cases
- Break multi-step reasoning into checkable intermediate outputs
- Use contrastive examples when two outputs look similar
Explanation
Correct Answer
D:Show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; contrastive tests verify the semantic boundary, not just the presence of a familiar template; evaluate the design with correct type selection on unseen evidence-versus-action pairs.
Incorrect Answers
A:A concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value; zero-shot prompting may fail where domain edge cases or specialized formatting rules are not expressible clearly in prose; this alternative would measure label accuracy and token use versus a heavier example-based prompt; it does not establish correct type selection on unseen evidence-versus-action pairs for the described use case.
B:Provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases; examples are not a substitute for a clean class definition or representative test labels; this alternative would measure recall on previously confused boundary examples outside the prompt set; it does not establish correct type selection on unseen evidence-versus-action pairs for the described use case.
C:Require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; requesting longer explanations without validation can make unsupported conclusions appear more persuasive; this alternative would measure intermediate artifact validity and final score agreement with a policy-engine oracle; it does not establish correct type selection on unseen evidence-versus-action pairs for the described use case.
Question 678
An early investigation message triggers the write-off tool before the user explicitly requests a transaction. Which corrective change best addresses the root cause?
- Write unambiguous tool descriptions and input schemas
- Remove duplicate and deprecated tools from the catalog
- Stage privileged capabilities only after explicit intent confirmation
- Offer each agent only the tools required for its role
Explanation
Correct Answer
C:Reveal high-impact tools only within a confirmed, authorized transaction stage; maintain server-side gating independently of model choice; stage-bound tests prove that tool exposure follows authorization and intent; verify the repair using absence of privileged tool calls before the confirmed transaction stage.
Incorrect Answers
A:A richer schema improves selection but does not override a backend that ignores caller authorization; define task-specific descriptions, typed arguments, required fields, and clear side-effect semantics so tool selection is meaningful; that control is assessed by correct tool-selection rate for ambiguous intents and invalid-argument rejection; the incident instead requires absence of privileged tool calls before the confirmed transaction stage.
B:More tool variants do not add genuine capability when they expose the same action with conflicting contracts; consolidate overlapping operations behind a maintained interface and retire obsolete tool definitions after migration testing; that control is assessed by single correct connector selection and consistent ticket states across repeated requests; the incident instead requires absence of privileged tool calls before the confirmed transaction stage.
D:Monitoring a broad tool catalog does not remove the authority to invoke dangerous operations; give the agent a least-capability tool set mapped to the exact tasks it may perform, reducing accidental and adversarial action paths; that control is assessed by unauthorized tool availability and invocation count in hostile transcript tests; the incident instead requires absence of privileged tool calls before the confirmed transaction stage.
Question 679
A retrieved instruction makes it send a sensitive customer list to an unknown external endpoint. Which corrective change best addresses the root cause?
- Cap agent steps and resource expenditure
- Constrain outbound channels to prevent data exfiltration
- Validate tool-result provenance and semantics
- Threat-model prompt injection across lower-trust inputs
- Use claim verification to mitigate confident fabrication
Explanation
Correct Answer
B:Restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; network- and tool-level evidence catches leakage not visible in user-facing text; verify the repair using blocked unauthorized destinations and absence of confidential fields in egress logs.
Incorrect Answers
A:Asking the model to 'be efficient' does not guarantee bounded autonomous execution; set per-task step, token, cost, and elapsed-time budgets with a clear abort and escalation state; that control is assessed by maximum resource usage and orderly termination on deliberately unsolvable tasks; the incident instead requires blocked unauthorized destinations and absence of confidential fields in egress logs.
C:A successful HTTP status does not mean returned data has decision authority; preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; that control is assessed by zero external-result fields promoted to internal approvals in test cases; the incident instead requires blocked unauthorized destinations and absence of confidential fields in egress logs.
D:Stronger wording in the system prompt is not a complete defense against an authorized dangerous tool; model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; that control is assessed by unauthorized tool-action success rate over injected emails and documents; the incident instead requires blocked unauthorized destinations and absence of confidential fields in egress logs.
E:Fluency and citation formatting cannot establish factual support on their own; require support from retrieved source passages and provide an explicit unknown path for unsupported claims; that control is assessed by proportion of numeric claims supported by independently inspected references; the incident instead requires blocked unauthorized destinations and absence of confidential fields in egress logs.
Question 680
A customer-support connector is exercised with forged roles, altered tenant identifiers, and unauthorized resource targets. Which result would demonstrate that backend authorization operates independently of polite model responses?
- Fully loaded cost per correctly completed document
- Unauthorized API requests rejected under role, tenant, and resource attack matrices
- Correct-answer rate at each supported coverage threshold including no-answer cases
- p95 time from user submission to fully validated answer under load
Explanation
Correct Answer
B:Direct penetration-style checks assess the enforceable security boundary; measure unauthorized API requests rejected under role, tenant, and resource attack matrices at the release gate; treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording.
Incorrect Answers
A:This metric includes rework rather than rewarding low-quality cheap drafts; relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; this tests fully loaded cost per correctly completed document; the actual release condition is unauthorized API requests rejected under role, tenant, and resource attack matrices.
C:Coverage-risk curves expose the tradeoff between reliability and unnecessary handoffs; measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; this tests correct-answer rate at each supported coverage threshold including no-answer cases; the actual release condition is unauthorized API requests rejected under role, tenant, and resource attack matrices.
D:Measuring the completed workflow captures actual end-user waiting, including downstream services; report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; this tests p95 time from user submission to fully validated answer under load; the actual release condition is unauthorized API requests rejected under role, tenant, and resource attack matrices.