Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 28

Question 270

An AI code review integration experiences recurring outages after untested connector changes. Which design decision directly addresses this constraint?

  1. Reproduce incidents using correlated logs and configuration versions
  2. Replay failures safely with sanitized inputs
  3. Roll back to a known-good tool and prompt release
  4. Produce incident learning actions with accountable follow-up

Explanation

Correct Answer

D:

Run blameless root-cause analysis, link corrective actions to owners, and add regression or operational safeguards before closing the incident; follow-up measures show whether operational learning survives the immediate repair; evaluate the design with recurrence rate of the diagnosed failure and completion of preventive actions.

Incorrect Answers

A:

Collect privacy-safe traces, request IDs, environment details, and exact configuration revisions to establish a reproducible failure case; a guessed prompt rewrite is unlikely to repair a fault that has not been isolated; this alternative would measure reproduction rate of sampled incidents from stored sanitized diagnostic metadata; it does not establish recurrence rate of the diagnosed failure and completion of preventive actions for the described use case.

B:

Build redacted, access-controlled replay fixtures preserving relevant syntax and error conditions without unnecessary secrets or personal data; raw transcript reuse can expose confidential inputs and violate data-handling policies; this alternative would measure bug reproduction on sanitized fixtures with no sensitive token disclosure; it does not establish recurrence rate of the diagnosed failure and completion of preventive actions for the described use case.

C:

Maintain versioned releases and documented rollback steps for hooks, prompts, model routes, and connector configuration; repeated untracked hotfixes can compound an outage and obscure the initial trigger; this alternative would measure time to restore normal operation using an approved previous release; it does not establish recurrence rate of the diagnosed failure and completion of preventive actions for the described use case.

Question 271

A patient education team needs to score clinical explanations that can be correct despite varied wording. An IT security helpdesk team needs its benchmark to include rare, high-consequence incidents rather than mostly routine resets. Which TWO test-design techniques best address these needs? Choose TWO.

  1. Measure human annotator consistency before trusting labels
  2. Use expert rubrics for nuanced qualitative outputs
  3. Protect a separate holdout set for regression decisions
  4. Stratify a golden dataset by real workload segments
  5. Include hostile prompt-injection and no-answer probes

Explanation

Correct Answers

B:

Use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; a semantic rubric tests meaning while allowing legitimate variation in language; its corresponding workstream should be validated by expert-rated factual fidelity and calibrated grader agreement on sampled explanations.

D:

Build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; stratified coverage prevents abundant easy cases from hiding unsupported workflows; its corresponding workstream should be validated by per-stratum pass rates with no uncovered critical scenario classes.

Incorrect Answers

A:

Use calibration sessions, disagreement analysis, adjudication rules, and documented label provenance before relying on the data as ground truth; more test volume does not fix unreliable or undefined labels; its natural acceptance measure is inter-annotator agreement and unresolved-label rate after adjudication, which does not satisfy the separate workstreams being assessed.

C:

Maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; reusing a public tuning set as the sole release gate invites overfitting rather than generalization; its natural acceptance measure is holdout quality stability across successive prompt versions, which does not satisfy the separate workstreams being assessed.

E:

Add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; testing normal prompts alone cannot estimate resilience to malicious or unsatisfiable requests; its natural acceptance measure is attack success rate and safe handling of intentionally unanswerable prompts, which does not satisfy the separate workstreams being assessed.

Question 272

A personnel team is piloting a bounded transcript summarizer. The release review must establish that it faithfully summarizes supplied content without making external calls or adding avoidable delay. Which result is most convincing?

  1. Rate of recommendations backed by all required completed stage checks
  2. Resolution of unseen diagnostic branches within a specified tool-call and time budget
  3. Correct citations to currently effective policy versions across update tests
  4. Summary fidelity and p95 completion time without any unapproved tool invocation

Explanation

Correct Answer

D:

A bounded call should meet the content rubric without introducing an unnecessary action surface; measure summary fidelity and p95 completion time without any unapproved tool invocation at the release gate; for a fixed transformation, a prompt enriched with only needed context is simpler, cheaper, and easier to evaluate than an agentic loop.

Incorrect Answers

A:

Stage-completion evidence verifies that mandatory controls were not left to agent discretion; explicit state transitions preserve required order, testing points, and deterministic branch conditions; Claude can supply judgment inside selected stages; this tests rate of recommendations backed by all required completed stage checks; the actual release condition is summary fidelity and p95 completion time without any unapproved tool invocation.

B:

The agent must demonstrate flexible discovery without escaping its operational guardrails; bounded agentic planning chooses follow-up tools from observed evidence while enforcing time, budget, and tool permissions; this tests resolution of unseen diagnostic branches within a specified tool-call and time budget; the actual release condition is summary fidelity and p95 completion time without any unapproved tool invocation.

C:

Version-aware citation checks show whether answers rely on authoritative current evidence; retrieval from a versioned, access-filtered source grounds answers in current documents and can expose the authoritative citation; this tests correct citations to currently effective policy versions across update tests; the actual release condition is summary fidelity and p95 completion time without any unapproved tool invocation.

Question 273

A factory deploys an inspection assistant to locations with different languages, scan quality, and document lengths. Which release score would prevent a clean-sample benchmark from hiding unacceptable performance in one subgroup?

  1. Per-segment acceptance rate across language, length, and noisy-input strata
  2. Measured macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic
  3. High-risk escalation recall and total cost under the required error ceiling
  4. Correct equipment identification and recommendation accuracy on labeled photographs

Explanation

Correct Answer

A:

Stratified scores expose failures that an overall aggregate conceals; measure per-segment acceptance rate across language, length, and noisy-input strata at the release gate; representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy.

Incorrect Answers

B:

Joint quality and price measurement prevents choosing a cheap model that fails costly minority categories; compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; this tests macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic; the actual release condition is per-segment acceptance rate across language, length, and noisy-input strata.

C:

Routing is valuable only if it captures difficult cases without needlessly escalating every simple request; combine an evaluated uncertainty trigger with escalation to a stronger model or reviewer, using explicit limits and post-route quality measurement; this tests high-risk escalation recall and total cost under the required error ceiling; the actual release condition is per-segment acceptance rate across language, length, and noisy-input strata.

D:

A multimodal acceptance set measures the capability absent from text-only trials; confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; this tests correct equipment identification and recommendation accuracy on labeled photographs; the actual release condition is per-segment acceptance rate across language, length, and noisy-input strata.

Question 274

A fast model produces plausible clause summaries but misses a three-document dependency that changes the legal risk assessment. Which intervention targets the defect rather than its symptoms?

  1. Use stronger reasoning capacity for genuinely complex decisions
  2. Select a model path compatible with required modalities
  3. Use confidence-aware routing with a bounded fallback
  4. Gate model-version updates on regression tests

Explanation

Correct Answer

A:

Escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; a targeted complex-case benchmark tests the capability that routine classification metrics miss; verify the repair using correct resolution of cross-exhibit conflicts at the required latency and review cost.

Incorrect Answers

B:

Model selection based solely on text benchmark accuracy does not prove image interpretation reliability; confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; that control is assessed by correct equipment identification and recommendation accuracy on labeled photographs; the incident instead requires correct resolution of cross-exhibit conflicts at the required latency and review cost.

C:

Blindly using one model tier on all cases neither controls the hardest error modes nor optimizes cost; combine an evaluated uncertainty trigger with escalation to a stronger model or reviewer, using explicit limits and post-route quality measurement; that control is assessed by high-risk escalation recall and total cost under the required error ceiling; the incident instead requires correct resolution of cross-exhibit conflicts at the required latency and review cost.

D:

Newer model branding alone cannot ensure identical tool-use or output behavior in an existing workflow; treat model replacements as versioned releases with task-specific eval gates, contract tests, canaries, and rollback capability; that control is assessed by no contract regressions and acceptable quality/latency deltas on the proposed release; the incident instead requires correct resolution of cross-exhibit conflicts at the required latency and review cost.

Question 275

An emergency coordination tool generates resource-allocation plans, but field leaders retain the ability to intervene. During a seeded high-impact error, which result would demonstrate that the override works before harm occurs?

  1. Successful intervention before seeded high-impact incorrect actions are committed
  2. Measured quality and escalation parity across predeclared language segments
  3. Time to resolve substantiated challenges and proportion of cases with documented review
  4. Task completion by users of tested assistive technologies on representative flows

Explanation

Correct Answer

A:

Pre-commit override drills test real controllability rather than retrospective documentation; measure successful intervention before seeded high-impact incorrect actions are committed at the release gate; preserve an authorized override and recovery route scaled to decision consequences, with feedback recorded for evaluation.

Incorrect Answers

B:

Group-specific evaluation reveals concentrated harm masked by pooled scores; evaluate suitable subgroup error rates and access outcomes, with privacy-aware sampling and appropriate fairness review; this tests quality and escalation parity across predeclared language segments; the actual release condition is successful intervention before seeded high-impact incorrect actions are committed.

C:

Remedy effectiveness measures whether affected people can obtain meaningful reconsideration; provide a contest and correction process with accountable human ownership for significant impacts; this tests time to resolve substantiated challenges and proportion of cases with documented review; the actual release condition is successful intervention before seeded high-impact incorrect actions are committed.

D:

Direct accessibility testing measures practical use rather than formal content correctness only; include accessibility requirements, assistive-technology testing, and inclusive content rubrics in evaluation and design; this tests task completion by users of tested assistive technologies on representative flows; the actual release condition is successful intervention before seeded high-impact incorrect actions are committed.

Question 276

A code-review service has repeated connector outages despite successful manual recoveries. Which operational measure would demonstrate that post-incident fixes address recurrence rather than only symptoms?

  1. Correct identification of enforced deny source without allowing prohibited commands
  2. Recurrence rate of the diagnosed failure and completion of preventive actions
  3. Bug reproduction on sanitized fixtures with no sensitive token disclosure
  4. Time to restore normal operation using an approved previous release
  5. Correct root-cause attribution under injected search timeouts

Explanation

Correct Answer

B:

Follow-up measures show whether operational learning survives the immediate repair; measure recurrence rate of the diagnosed failure and completion of preventive actions at the release gate; run blameless root-cause analysis, link corrective actions to owners, and add regression or operational safeguards before closing the incident.

Incorrect Answers

A:

Precedence tests prevent insecure workarounds that bypass governance intent; review organization, project, and local permission layers in the documented precedence order before weakening security controls; this tests correct identification of enforced deny source without allowing prohibited commands; the actual release condition is recurrence rate of the diagnosed failure and completion of preventive actions.

C:

Safe replay demonstrates that privacy and diagnostic value can coexist; build redacted, access-controlled replay fixtures preserving relevant syntax and error conditions without unnecessary secrets or personal data; this tests bug reproduction on sanitized fixtures with no sensitive token disclosure; the actual release condition is recurrence rate of the diagnosed failure and completion of preventive actions.

D:

A rollback drill verifies that recovery does not depend on the original developer being online; maintain versioned releases and documented rollback steps for hooks, prompts, model routes, and connector configuration; this tests time to restore normal operation using an approved previous release; the actual release condition is recurrence rate of the diagnosed failure and completion of preventive actions.

E:

Controlled fault injection tests whether the team diagnoses the failing layer; classify tool errors separately from model responses, validate return codes, and probe the failing integration before changing prompts; this tests correct root-cause attribution under injected search timeouts; the actual release condition is recurrence rate of the diagnosed failure and completion of preventive actions.

Question 277

A regulated claims team finds reviewers habitually approving confident model outputs without checking source contradictions. Separately, an exploratory agent repeatedly asks its tools the same unresolved question until its budget is exhausted. Which TWO safeguards address human oversight and resource control? Choose TWO.

  1. Cap agent steps and resource expenditure
  2. Threat-model prompt injection across lower-trust inputs
  3. Restrict outbound channels to prevent data exfiltration
  4. Validate tool-result provenance and semantics
  5. Counter automation bias with visible uncertainty and review

Explanation

Correct Answers

A:

Set per-task step, token, cost, and elapsed-time budgets with a clear abort and escalation state; budget stress tests show whether runaway loops fail safely; its corresponding workstream should be validated by maximum resource usage and orderly termination on deliberately unsolvable tasks.

E:

Design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; the review test measures meaningful human oversight, not click-through compliance; its corresponding workstream should be validated by reviewer error interception on seeded incorrect recommendations.

Incorrect Answers

B:

Model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; stronger wording in the system prompt is not a complete defense against an authorized dangerous tool; its natural acceptance measure is unauthorized tool-action success rate over injected emails and documents, which does not satisfy the separate workstreams being assessed.

C:

Restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; its natural acceptance measure is blocked unauthorized destinations and absence of confidential fields in egress logs, which does not satisfy the separate workstreams being assessed.

D:

Preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; a successful HTTP status does not mean returned data has decision authority; its natural acceptance measure is zero external-result fields promoted to internal approvals in test cases, which does not satisfy the separate workstreams being assessed.

Question 278

The team continues the experiment while waiting for customer complaints rather than acting on a pre-agreed safety limit. Which intervention targets the defect rather than its symptoms?

  1. Inspect segment-level effects rather than only global lift
  2. Require an explicit rollback trigger for harmful regressions
  3. Release canaries with guardrail metrics and rollback conditions
  4. Use shadow evaluation before enabling new side effects

Explanation

Correct Answer

B:

Establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; operational readiness includes the ability to stop unsafe experiments promptly; verify the repair using time from triggered guardrail breach to verified rollback.

Incorrect Answers

A:

A positive overall average does not guarantee every relevant user group benefits; compare treatment effects across predeclared critical segments and require safeguards where harm would be unacceptable; that control is assessed by resolution and safety deltas by channel and accessibility segment; the incident instead requires time from triggered guardrail breach to verified rollback.

C:

A single offline score cannot show all production interactions or blast radius; deploy to a bounded cohort while observing quality, errors, and tail latency, with predefined stop and rollback thresholds; that control is assessed by canary threshold compliance and time to revert a seeded regression; the incident instead requires time from triggered guardrail breach to verified rollback.

D:

A normal A/B test is inappropriate if experimental behavior can cause unreviewed irreversible changes; replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; that control is assessed by shadow-policy agreement and zero write side effects across replayed billing cases; the incident instead requires time from triggered guardrail breach to verified rollback.

Question 279

A compliance helper is tested on cases with ambiguous or conflicting jurisdictional guidance. Which measure would establish that legally consequential uncertainties reach an accountable specialist?

  1. Defect-detection rate and reviewer workload under a risk-tiered sampling policy
  2. Zero disbursements without an auditable transaction-specific approver
  3. Recurrence rate of categorized, previously corrected error types
  4. Unauthorized field exposure and successful review accuracy after data minimization
  5. Correct escalation of ambiguous legal cases with reviewer attribution

Explanation

Correct Answer

E:

A consequence-aware test checks whether unsupported authoritative decisions are avoided; measure correct escalation of ambiguous legal cases with reviewer attribution at the release gate; define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context.

Incorrect Answers

A:

Sampling effectiveness must balance residual errors and human capacity; combine automated checks with representative human sampling and exception-based review scaled to risk; this tests defect-detection rate and reviewer workload under a risk-tiered sampling policy; the actual release condition is correct escalation of ambiguous legal cases with reviewer attribution.

B:

Payment simulation tests prove approvals are bound to the exact side effect; separate recommendation from execution and require an authenticated human approval bound to the exact amount, payee, and transaction; this tests zero disbursements without an auditable transaction-specific approver; the actual release condition is correct escalation of ambiguous legal cases with reviewer attribution.

C:

Structured override trends reveal whether feedback improves the system over time; record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; this tests recurrence rate of categorized, previously corrected error types; the actual release condition is correct escalation of ambiguous legal cases with reviewer attribution.

D:

Review workflows should preserve decision quality while protecting privacy; minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; this tests unauthorized field exposure and successful review accuracy after data minimization; the actual release condition is correct escalation of ambiguous legal cases with reviewer attribution.

Question 280

A travel-insurance assistant can calculate a settlement but the transfer must be approved by a named financial officer. In a separate network-operations service, risk ranges from harmless diagnostics to disconnecting a critical site. Which TWO approval designs are appropriate to the respective consequences? Choose TWO.

  1. Require named authorization before moving money
  2. Adopt risk-based sampling for low-impact decisions
  3. Require stronger approval paths for exceptional high-impact actions
  4. Escalate binding legal interpretations to qualified reviewers
  5. Capture reviewer overrides as structured improvement signals

Explanation

Correct Answers

A:

Separate recommendation from execution and require an authenticated human approval bound to the exact amount, payee, and transaction; payment simulation tests prove approvals are bound to the exact side effect; its corresponding workstream should be validated by zero disbursements without an auditable transaction-specific approver.

C:

Apply risk-tiered approvals, including dual control or independent sign-off where policy demands, to exceptionally consequential mutations; approval testing must target the cases where one mistake affects many users; its corresponding workstream should be validated by critical-action dual-approval compliance and zero bypasses during staged tests.

Incorrect Answers

B:

Combine automated checks with representative human sampling and exception-based review scaled to risk; universal manual review may be disproportionate when actions are reversible and errors are easily detected; its natural acceptance measure is defect-detection rate and reviewer workload under a risk-tiered sampling policy, which does not satisfy the separate workstreams being assessed.

D:

Define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; a high model confidence estimate cannot replace delegated legal decision authority; its natural acceptance measure is correct escalation of ambiguous legal cases with reviewer attribution, which does not satisfy the separate workstreams being assessed.

E:

Record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; counting approval clicks alone does not describe why humans disagreed with the AI; its natural acceptance measure is recurrence rate of categorized, previously corrected error types, which does not satisfy the separate workstreams being assessed.

Claude Certified Architect - Professional (CCAR-P) Practice Tests