Question 410
A large repository needs independent code search, test analysis, and documentation review without letting every helper deploy changes. What is the most targeted architecture choice?
- Use specialized subagents with bounded tools and instructions
- Version team-wide CLAUDE.md guidance with the repository
- Use deterministic hooks to enforce required developer gates
- Keep credentials out of prompts and versioned configuration
- Scope MCP connections by project and user identity
- Use managed configuration to constrain dangerous permissions
Explanation
Correct Answer
A:Give each delegated role explicit objectives, context limits, and tool permissions with clear return artifacts; role-isolation tests prove that decomposition also restricts authority; evaluate the design with subagent task completion with zero out-of-scope tool access.
Incorrect Answers
B:Put reviewed repository-specific guidance in a shared CLAUDE.md file with appropriate scope and code review; a documentation file guides behavior but cannot enforce operating-system permissions or replace CI tests; this alternative would measure consistency of generated changes with repository build and test conventions across fresh sessions; it does not establish subagent task completion with zero out-of-scope tool access for the described use case.
C:Configure supported hooks for deterministic checks at appropriate lifecycle points, with clear failure behavior and maintenance ownership; a written instruction alone is best-effort and does not guarantee enforcement of a critical command gate; this alternative would measure hook execution and blocked noncompliant operations under simulated bypass attempts; it does not establish subagent task completion with zero out-of-scope tool access for the described use case.
D:Use approved secret storage, scoped environment access, and redaction while avoiding credentials in repositories, prompts, or logs; masking output after committing a secret cannot undo credential exposure or grant least privilege; this alternative would measure secret-scanning results and blocked exfiltration on seeded repository credentials; it does not establish subagent task completion with zero out-of-scope tool access for the described use case.
E:Choose appropriate configuration scope and per-user or service authorization for MCP servers, avoiding shared cross-project secrets; a working connection does not prove the assistant is restricted to the user's intended project resources; this alternative would measure access denial for cross-project MCP resources under user identity tests; it does not establish subagent task completion with zero out-of-scope tool access for the described use case.
F:Use organization-managed policy and supported permission configuration so enforced restrictions cannot be casually overridden by per-user convenience settings; personal preference files are not a trustworthy substitute for centrally governed high-risk permissions; this alternative would measure denial of prohibited commands after attempts to override local and project configuration; it does not establish subagent task completion with zero out-of-scope tool access for the described use case.
Question 411
A long-running engineering report can resume after an interruption, including one that occurs after an external side effect has completed. Which fault-injection result most directly verifies durable recovery?
- Stage-specific defect localization and repair without redoing previously validated stages
- Conflict detection recall and citation coverage across independently processed source sets
- Correct merged clause counts across repeated parallel execution races
- Successful recovery after injected worker failures without duplicated writes or stages
Explanation
Correct Answer
D:Replay testing verifies the precise recovery behavior the pipeline requires; measure successful recovery after injected worker failures without duplicated writes or stages at the release gate; durable checkpoints record validated intermediate artifacts and resume at safe boundaries with idempotent side effects.
Incorrect Answers
A:A traceable artifact boundary confines error analysis to the failed operation; separate stages with typed artifacts allow validation and targeted reruns at meaningful boundaries; this tests stage-specific defect localization and repair without redoing previously validated stages; the actual release condition is successful recovery after injected worker failures without duplicated writes or stages.
B:The final synthesis must expose disagreements rather than average them away; analyze source partitions separately, retain provenance, and reconcile contradictions through a dedicated synthesis stage; this tests conflict detection recall and citation coverage across independently processed source sets; the actual release condition is successful recovery after injected worker failures without duplicated writes or stages.
C:Deterministic merge checks measure integrity rather than only wall-clock speed; isolate independent tasks, retain separate results, then join only when required dependencies have completed; this tests correct merged clause counts across repeated parallel execution races; the actual release condition is successful recovery after injected worker failures without duplicated writes or stages.
Question 412
Dense semantic search returns similar part descriptions but not the exact requested identifier. Which change should the architect recommend first?
- Use semantic vector search for paraphrased knowledge questions
- Use lexical search for exact identifiers and literals
- Combine lexical and semantic candidates when queries mix both
- Rerank candidate passages for question relevance
- Retrieve parent context after finding a specific chunk
Explanation
Correct Answer
B:Keyword or lexical matching is effective for codes, exact phrases, and terms whose characters carry the meaning; identifier-focused tests reveal errors that a semantic relevance score may ignore; verify the repair using exact-ID hit rate at the top retrieval ranks.
Incorrect Answers
A:Semantic search is not guaranteed to respect exact version strings, numerals, or access boundaries without additional controls; vector retrieval can match conceptual similarity between user wording and document text, followed by evidence verification; that control is assessed by recall on human-authored paraphrases absent from manual headings; the incident instead requires exact-ID hit rate at the top retrieval ranks.
C:Either modality alone leaves a known class of mixed-format queries uncovered; hybrid retrieval joins exact-term and semantic evidence, then applies metadata filtering and ranking; that control is assessed by joint exact-identifier accuracy and semantic-symptom recall on mixed queries; the incident instead requires exact-ID hit rate at the top retrieval ranks.
D:A bigger first-stage candidate set without selection can dilute context and increase cost; apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; that control is assessed by relevant-procedure rank and end-to-end answer support rate after reranking; the incident instead requires exact-ID hit rate at the top retrieval ranks.
E:Passing isolated small chunks may preserve relevance while losing the context needed for a correct conclusion; use child-chunk retrieval for recall and fetch validated parent context or adjacent governing sections for interpretation; that control is assessed by conditional-rule accuracy with and without parent-document expansion; the incident instead requires exact-ID hit rate at the top retrieval ranks.
Question 413
Debuggers copy live transcripts into an open testing environment to reproduce a rare parsing failure. Which change should the architect recommend first?
- Replay failures safely with sanitized inputs
- Inspect permission precedence when a tool is unexpectedly denied
- Produce incident learning actions with accountable follow-up
- Roll back to a known-good tool and prompt release
Explanation
Correct Answer
A:Build redacted, access-controlled replay fixtures preserving relevant syntax and error conditions without unnecessary secrets or personal data; safe replay demonstrates that privacy and diagnostic value can coexist; verify the repair using bug reproduction on sanitized fixtures with no sensitive token disclosure.
Incorrect Answers
B:Editing a user preference cannot override a stronger centrally managed deny rule; review organization, project, and local permission layers in the documented precedence order before weakening security controls; that control is assessed by correct identification of enforced deny source without allowing prohibited commands; the incident instead requires bug reproduction on sanitized fixtures with no sensitive token disclosure.
C:Restarting the service restores availability but does not reduce the recurrence probability; run blameless root-cause analysis, link corrective actions to owners, and add regression or operational safeguards before closing the incident; that control is assessed by recurrence rate of the diagnosed failure and completion of preventive actions; the incident instead requires bug reproduction on sanitized fixtures with no sensitive token disclosure.
D:Repeated untracked hotfixes can compound an outage and obscure the initial trigger; maintain versioned releases and documented rollback steps for hooks, prompts, model routes, and connector configuration; that control is assessed by time to restore normal operation using an approved previous release; the incident instead requires bug reproduction on sanitized fixtures with no sensitive token disclosure.
Question 414
An internal tool catalog now requires a reviewed manifest before a new connector can enter a tenant deployment. Which negative test would show the approval gate prevents privilege expansion?
- Zero mutation permissions for report-only identities across tool contract checks
- Changes rejected when requested tool scopes exceed the approved task manifest
- Correct tool-selection rate for ambiguous intents and invalid-argument rejection
- Single correct connector selection and consistent ticket states across repeated requests
Explanation
Correct Answer
B:Policy-based manifest validation detects excessive authority before runtime exposure; measure changes rejected when requested tool scopes exceed the approved task manifest at the release gate; maintain owner-approved manifests with risk classification, contract checks, permission reviews, and change-control gates.
Incorrect Answers
A:Permission tests reveal whether supposedly analytical agents can cause side effects; expose narrowly scoped read-only capabilities and gate changes through separate authenticated write paths with explicit approvals; this tests zero mutation permissions for report-only identities across tool contract checks; the actual release condition is changes rejected when requested tool scopes exceed the approved task manifest.
C:Confusion tests exercise the ambiguity that caused misrouting; define task-specific descriptions, typed arguments, required fields, and clear side-effect semantics so tool selection is meaningful; this tests correct tool-selection rate for ambiguous intents and invalid-argument rejection; the actual release condition is changes rejected when requested tool scopes exceed the approved task manifest.
D:Removing ambiguity should reduce routing variance without reducing supported tasks; consolidate overlapping operations behind a maintained interface and retire obsolete tool definitions after migration testing; this tests single correct connector selection and consistent ticket states across repeated requests; the actual release condition is changes rejected when requested tool scopes exceed the approved task manifest.
Question 415
Adoption and satisfaction metrics climb, but nobody can separate AI impact from concurrent operations changes. What design correction resolves the reported failure?
- Establish a degraded mode that protects the value proposition
- Track cost per accepted business outcome
- Translate response-time expectations into a percentile SLO
- Use a control group to establish incremental business impact
Explanation
Correct Answer
D:Use matched baselines, controlled rollout, or appropriate difference-in-differences analysis to isolate contribution under plausible confounders; a defensible counterfactual is required to credit the architecture with the observed benefit; verify the repair using incremental resolution lift compared with a comparable non-assisted cohort.
Incorrect Answers
B:Raw token prices do not capture productivity or failure-driven rework; combine inference, retrieval, tools, review, and rework costs into cost per accepted deliverable; that control is assessed by total operational cost divided by approved documents meeting the quality rubric; the incident instead requires incremental resolution lift compared with a comparable non-assisted cohort.
C:A percentile SLO does not alone reveal whether the answers are factually correct; define a p95 or p99 target against real end-to-end requests, including retrieval and tools, rather than reporting an average model time; that control is assessed by end-to-end p95 latency and error rate against the agreed service objective; the incident instead requires incremental resolution lift compared with a comparable non-assisted cohort.
A:High answer accuracy in normal mode says little about process continuity during dependency failures; separate critical deterministic operations from AI enhancement and provide explicit fallback and manual paths when model service is unavailable; that control is assessed by successful completion of critical workflows during injected model outages; the incident instead requires incremental resolution lift compared with a comparable non-assisted cohort.
Question 416
A back-office team has several possible uses for Claude, but only one offers repeatable inputs and a manageable exception queue. Which measured operational outcome would demonstrate that the chosen use case creates value?
- Accepted cases per reviewer-hour with exception share tracked separately
- Passing the acceptance gate on a representative pilot cohort including difficult exceptions
- Agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set
- Share of binding decisions showing the required named approver and rationale
Explanation
Correct Answer
A:Counting both throughput and escalations prevents a fast happy path from hiding a costly exception queue; measure accepted cases per reviewer-hour with exception share tracked separately at the release gate; select a high-frequency, well-described task with measurable work reduction and explicit escalation for ambiguous records.
Incorrect Answers
B:A bounded deployment can prove business fitness before a large enterprise-wide dependency is created; a narrow pilot validates one representative flow and pre-agreed quality, latency, and escalation thresholds before scaling to other workflows; this tests passing the acceptance gate on a representative pilot cohort including difficult exceptions; the actual release condition is accepted cases per reviewer-hour with exception share tracked separately.
C:Consistent scoring makes a go-live decision defensible rather than impressionistic; convert vague requests into observable task outcomes, rubric-based quality dimensions, and failure thresholds that stakeholders can agree to test; this tests agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set; the actual release condition is accepted cases per reviewer-hour with exception share tracked separately.
D:An auditable decision trail demonstrates that AI outputs remained advisory where policy requires it; explicit decision-rights boundaries distinguish advisory generation from actions requiring a credentialed human or deterministic policy engine; this tests share of binding decisions showing the required named approver and rationale; the actual release condition is accepted cases per reviewer-hour with exception share tracked separately.
Question 417
A platform group debates direct API integration versus MCP for a shared connector. Which strategy should be prioritized for this requirement?
- Record alternatives and rationale in an architecture decision record
- Keep a versioned decision and change history
- Present executive decisions through outcomes, cost, and risk
- Maintain a risk register with controls and accountable owners
Explanation
Correct Answer
A:Use an ADR recording context, considered options, decision, trade-offs, implications, and conditions for revisiting it; decision records preserve rationale beyond a particular architect's memory; evaluate the design with reconstructability of key design decisions after a simulated staff handoff.
Incorrect Answers
B:Maintain a version-controlled decision log linked to implementation and review milestones, including superseded guidance; chat history and scattered slides rarely provide an authoritative current decision state; this alternative would measure correct retrieval of active decisions and their supersession chain during review; it does not establish reconstructability of key design decisions after a simulated staff handoff for the described use case.
C:Summarize business value, alternatives, cost envelope, measured evidence, and requested decisions in concise nontechnical terms; a detailed API design is valuable to implementers but not sufficient for a funding decision; this alternative would measure sponsor comprehension of expected outcomes, decision options, and risk ownership; it does not establish reconstructability of key design decisions after a simulated staff handoff for the described use case.
D:Record probability and impact where appropriate, control owners, evidence, residual risk, and required approvals; a high-level statement that AI has risks does not create an actionable governance plan; this alternative would measure completion and ownership of high-severity mitigations before the release gate; it does not establish reconstructability of key design decisions after a simulated staff handoff for the described use case.
Question 418
Engineers need to debug failures but raw request logging copies patient details into general observability storage. How should the design be corrected?
- Log structured tool outcomes without exposing sensitive payloads
- Alert on unusual high-impact tool activity
- Separate retrieval, provider, and tool latency budgets
- Correlate model, retrieval, and tool spans in distributed traces
- Segment reliability metrics by tenant and workload
Explanation
Correct Answer
A:Record tool type, status, duration, correlation IDs, and carefully redacted diagnostic fields under controlled retention; redaction and observability must be validated together, not traded off blindly; verify the repair using error diagnosability with seeded sensitive fields absent from collected logs.
Incorrect Answers
B:General chat satisfaction metrics are too delayed and indirect to detect dangerous mutation spikes; monitor write volume, denials, unusual resource patterns, and action-specific thresholds with fast containment routing; that control is assessed by time to detect and stop a simulated unauthorized bulk-write burst; the incident instead requires error diagnosability with seeded sensitive fields absent from collected logs.
C:Optimizing generation tokens cannot repair a slow connector that dominates the critical path; instrument per-stage timing and compare each segment against its allocated latency budget before changing model configuration; that control is assessed by stage-specific p95 deltas identifying injected bottlenecks; the incident instead requires error diagnosability with seeded sensitive fields absent from collected logs.
D:A model-only latency chart cannot explain retrieval or downstream network bottlenecks; end-to-end traces with consistent request and span identifiers identify which service and step consumed time; that control is assessed by trace completeness and correct bottleneck attribution on instrumented fault injections; the incident instead requires error diagnosability with seeded sensitive fields absent from collected logs.
E:Overall averages can hide concentrated failures without meaningful dimensional analysis; break down latency, errors, and tool success by permission-safe tenant and workload dimensions with cardinality controls; that control is assessed by detection time for an injected single-tenant degradation; the incident instead requires error diagnosability with seeded sensitive fields absent from collected logs.
Question 419
A question-answering product gets unsupported answers after a new chunking scheme is deployed. Which intervention is most appropriate for this use case?
- Chunk documents along meaningful structural boundaries
- Filter retrieval by access control before model exposure
- Evaluate retrieval quality independently of generation
- Refresh the index when source content changes
Explanation
Correct Answer
C:Measure retrieval recall, ranking relevance, and evidence coverage on labeled query-document pairs before optimizing generation; separating retrieval metrics identifies which pipeline stage needs repair; evaluate the design with recall at k and evidence-coverage rate for known-answer queries.
Incorrect Answers
A:Chunk by headings and semantic sections with deliberate overlap where necessary while maintaining parent-document references; arbitrary equal-length chunks may split dependencies despite fitting an embedding model's input limit; this alternative would measure retrieval of both governing clause and exception for labeled contract questions; it does not establish recall at k and evidence-coverage rate for known-answer queries for the described use case.
B:Apply document permissions to retrieval candidates before material enters model context or logs; a model instruction to ignore confidential items is not a data-access boundary once the text is supplied; this alternative would measure unauthorized chunk exposure rate under staff permission matrices, targeting zero; it does not establish recall at k and evidence-coverage rate for known-answer queries for the described use case.
D:Build a monitored ingestion path for adds, updates, deletes, and effective dates, with freshness checks and removal of superseded chunks; a larger context window cannot compensate for documents missing from the index; this alternative would measure publication-to-search freshness lag and absence of withdrawn clauses in results; it does not establish recall at k and evidence-coverage rate for known-answer queries for the described use case.
Question 420
Increasing the raw top-k count improves recall yet overwhelms Claude with irrelevant policy fragments. What design correction resolves the reported failure?
- Rerank a short candidate set for ambiguous retrieval queries
- Cache safe, version-keyed retrieval responses
- Parallelize independent read-only tool calls
- Use a safe limited answer when the deadline would otherwise be missed
Explanation
Correct Answer
A:Retrieve a bounded candidate set and rerank against the actual question to concentrate useful evidence without feeding all candidates to the model; a quality-latency frontier shows whether an extra ranking pass is worth its overhead; verify the repair using grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths.
Incorrect Answers
B:An unversioned cache can improve speed while silently sacrificing freshness and access correctness; use a cache keyed by query, permissions where relevant, and content version, with invalidation tied to source updates; that control is assessed by repeated-query latency with zero stale or cross-scope cache responses after updates; the incident instead requires grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths.
C:Parallel execution offers no benefit when one call requires another's output or modifies shared state; execute independent lookups concurrently with bounded timeouts and aggregate the results after all necessary dependencies complete; that control is assessed by end-to-end p95 improvement with all required sources still present; the incident instead requires grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths.
D:Silently presenting an unverified enriched conclusion would violate accuracy requirements despite meeting a timer; keep the critical verified signal independent and return a clearly limited result when optional enrichment exceeds its budget; that control is assessed by deadline compliance and correct indication of omitted enrichment; the incident instead requires grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths.