Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 11

Question 100

A report generator frequently reaches output limits before its mandatory findings section is complete. What is the most defensible next step?

  1. Allocate input and output token budgets per workflow step
  2. Remove obsolete tool output while preserving current evidence
  3. Persist operational decisions outside transient chat history
  4. Compact extended sessions without discarding commitments

Explanation

Correct Answer

A:

Reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure; budgeted completion tests directly reveal truncation at maximum expected load; evaluate the design with complete mandatory report sections under the worst supported input size.

Incorrect Answers

B:

Prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; pruning without version markers can erase the evidence needed to explain the incident's evolution; this alternative would measure current-state citation accuracy across log-update sequences; it does not establish complete mandatory report sections under the worst supported input size for the described use case.

C:

Store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger; long context alone cannot provide transactional durability or a trustworthy authorization trail; this alternative would measure correct recovery of approvals and pending actions after forced session reset; it does not establish complete mandatory report sections under the worst supported input size for the described use case.

D:

Store a structured, reviewable summary of decisions, open issues, and provenance; inject it at checkpoints rather than replaying every chat turn; blind summarization can lose critical exceptions unless durable commitments are explicitly represented; this alternative would measure retention of signed-off requirements after long simulated conversations; it does not establish complete mandatory report sections under the worst supported input size for the described use case.

Question 101

A document parser emits machine-readable records to a downstream validator. The release suite injects nulls, invalid dates, and unrecognized status values. Which result demonstrates that malformed outputs cannot pass the boundary?

  1. Holdout quality stability across successive prompt versions
  2. Zero schema violations accepted by the downstream consumer across generated cases
  3. Attack success rate and safe handling of intentionally unanswerable prompts
  4. Per-stratum pass rates with no uncovered critical scenario classes

Explanation

Correct Answer

B:

Executable contract tests deliver reproducible feedback at scale; measure zero schema violations accepted by the downstream consumer across generated cases at the release gate; apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules.

Incorrect Answers

A:

Untouched data estimates whether improvements transfer beyond the examples used during tuning; maintain untouched holdout cases and periodic refreshes representing real traffic, with clear version and leakage controls; this tests holdout quality stability across successive prompt versions; the actual release condition is zero schema violations accepted by the downstream consumer across generated cases.

C:

Red-team cases make trust-boundary performance observable; add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; this tests attack success rate and safe handling of intentionally unanswerable prompts; the actual release condition is zero schema violations accepted by the downstream consumer across generated cases.

D:

Stratified coverage prevents abundant easy cases from hiding unsupported workflows; build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; this tests per-stratum pass rates with no uncovered critical scenario classes; the actual release condition is zero schema violations accepted by the downstream consumer across generated cases.

Question 102

Engineers cannot tell when degraded service requires pausing feature releases. What is the most direct remediation?

  1. Establish a documented escalation path for service incidents
  2. Define SLOs and error budgets for agreed user outcomes
  3. Set observability and reporting expectations with stakeholders
  4. Clarify shared responsibility for the service boundary
  5. Route user feedback into a triaged, owned backlog

Explanation

Correct Answer

B:

Establish measurable user-facing service objectives, reporting windows, and error-budget policies tied to accountable actions; operational SLO tests connect observed service quality to response commitments; verify the repair using sLO compliance and prescribed actions during a simulated error-budget breach.

Incorrect Answers

A:

Generic 'contact support' guidance lacks accountable timing and action for critical failures; define incident severity, on-call ownership, manual fallback authority, and customer communication triggers; that control is assessed by time to appropriate escalation and fallback activation during drills; the incident instead requires sLO compliance and prescribed actions during a simulated error-budget breach.

C:

Platform telemetry is necessary but not equivalent to business service reporting; agree on outcome-aligned service reports, cadence, escalation thresholds, and audiences during solution definition; that control is assessed by delivery of agreed business and risk indicators on schedule with decision-ready interpretation; the incident instead requires sLO compliance and prescribed actions during a simulated error-budget breach.

D:

A vendor SLA typically does not assign responsibility for every custom integration and decision rule; define provider, platform, application, security, and business-owner responsibilities in an explicit operating model; that control is assessed by ownership coverage for failures across model, application, data, and tool layers; the incident instead requires sLO compliance and prescribed actions during a simulated error-budget breach.

E:

Unstructured rating totals cannot by themselves direct a safe corrective action; classify feedback by severity and root-cause type, assign owners, link trace IDs, and feed validated cases into regression tests; that control is assessed by time from actionable feedback to triage, owner assignment, and verified fix; the incident instead requires sLO compliance and prescribed actions during a simulated error-budget breach.

Question 103

A customer-support agent reads user input, calls account tools, and generates recommendations involving personal data. Which approach best meets the stated requirement?

  1. Allowlist generated fields and destinations before external calls
  2. Treat retrieved instructions as untrusted data
  3. Enforce least privilege where tool actions execute
  4. Apply controls at input, tool execution, and output boundaries

Explanation

Correct Answer

D:

Layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority; end-to-end adversarial cases assess the combined defenses and their failure isolation; evaluate the design with denial of unauthorized actions and harmful outputs across multi-stage red-team paths.

Incorrect Answers

A:

Validate destinations, argument types, sensitive fields, and authorization before dispatch; reject values outside the approved contract; formatting the request as JSON does not make arbitrary field values safe to transmit; this alternative would measure zero unapproved recipients or fields accepted by the outbound service; it does not establish denial of unauthorized actions and harmful outputs across multi-stage red-team paths for the described use case.

B:

Keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; assuming retrieved text is authoritative because it came from a connector opens a prompt-injection path; this alternative would measure attack success rate on adversarial web pages and zero unauthorized egress calls; it does not establish denial of unauthorized actions and harmful outputs across multi-stage red-team paths for the described use case.

C:

Enforce role and scope in the backend execution service using credentials that cannot perform the forbidden action; natural-language reminders do not remove capabilities exposed by the API or bearer token; this alternative would measure zero payment writes when tested with the analytics identity and forged instructions; it does not establish denial of unauthorized actions and harmful outputs across multi-stage red-team paths for the described use case.

Question 104

A procurement risk estimate depends on extracting clauses, identifying exceptions, and applying a transparent scoring policy. What action best fits the business and technical boundary?

  1. Chain prompts when one result is a prerequisite for another
  2. Use contrastive examples when two outputs look similar
  3. Break multi-step reasoning into checkable intermediate outputs
  4. Add few-shot examples for difficult boundary cases

Explanation

Correct Answer

C:

Require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; checkable milestones provide evidence of reasoning quality beyond rhetorical fluency; evaluate the design with intermediate artifact validity and final score agreement with a policy-engine oracle.

Incorrect Answers

A:

Use a staged prompt chain where a verified intermediate classification is required before the next generation step; chaining every trivial substep increases latency without improving reliability for simple independent tasks; this alternative would measure explanations generated only for records that passed the eligibility stage; it does not establish intermediate artifact validity and final score agreement with a policy-engine oracle for the described use case.

B:

Show a paired positive and negative example with the decision rule that separates the formats; then test unseen cases; adding examples without highlighting the differentiating condition can reinforce superficial formatting only; this alternative would measure correct type selection on unseen evidence-versus-action pairs; it does not establish intermediate artifact validity and final score agreement with a policy-engine oracle for the described use case.

D:

Provide a few diverse, accurately labeled counterexamples that illustrate the exact class boundary, then verify against held-out cases; examples are not a substitute for a clean class definition or representative test labels; this alternative would measure recall on previously confused boundary examples outside the prompt set; it does not establish intermediate artifact validity and final score agreement with a policy-engine oracle for the described use case.

Question 105

A retrieval pilot deliberately embeds forged administrator directives inside otherwise useful source paragraphs. Which result would demonstrate that those passages remain evidence, not instructions, without sacrificing legitimate answers?

  1. Prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use
  2. Secret-exposure rate under seeded credential-bearing diagnostic inputs
  3. Rate of user messages that fail to override defined task boundaries in regression tests
  4. Valid schema rate and zero unauthorized enum values reaching the consumer
  5. Out-of-scope refusal and handoff accuracy on a policy-boundary test set

Explanation

Correct Answer

A:

Adversarial retrieval tests exercise the trust boundary rather than only neutral documents; measure prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use at the release gate; clearly separate lower-trust source material from application instructions and enforce permissions at the tool layer as well.

Incorrect Answers

B:

Seeded canaries allow a measurable check of unwanted disclosure paths; redact sensitive fields upstream, prohibit secret reproduction in instructions, and screen outbound content before release; this tests secret-exposure rate under seeded credential-bearing diagnostic inputs; the actual release condition is prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use.

C:

Boundary-specific tests confirm that changing user content cannot rewrite the application contract; separate stable system-level policy from user content and tool results, retaining consistent privilege boundaries across requests; this tests rate of user messages that fail to override defined task boundaries in regression tests; the actual release condition is prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use.

D:

Parser-level rejection tests verify the actual interface boundary; request a documented structured format and validate schema, allowed values, and business semantics before downstream use; this tests valid schema rate and zero unauthorized enum values reaching the consumer; the actual release condition is prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use.

E:

Boundary tests demonstrate task scoping rather than only fluent responses on ordinary requests; define the assistant's permitted tasks, out-of-scope handling, and escalation behavior in stable application-level instructions, backed by independent access enforcement; this tests out-of-scope refusal and handoff accuracy on a policy-boundary test set; the actual release condition is prompt-injection resistance on hostile retrieved snippets without loss of legitimate source use.

Question 106

A service-desk assistant performs well on common password resets but rarely sees high-stakes security incidents in testing. What should the solution architect recommend?

  1. Include hostile prompt-injection and no-answer probes
  2. Use deterministic code checks for objective output contracts
  3. Stratify a golden dataset by real workload segments
  4. Use expert rubrics for nuanced qualitative outputs

Explanation

Correct Answer

C:

Build a curated, versioned golden set covering volume, business risk, language, format, and difficult edge cases; stratified coverage prevents abundant easy cases from hiding unsupported workflows; evaluate the design with per-stratum pass rates with no uncovered critical scenario classes.

Incorrect Answers

A:

Add adversarial retrieved text, invalid queries, ambiguous requests, and missing-evidence cases with expected safe behavior; testing normal prompts alone cannot estimate resilience to malicious or unsatisfiable requests; this alternative would measure attack success rate and safe handling of intentionally unanswerable prompts; it does not establish per-stratum pass rates with no uncovered critical scenario classes for the described use case.

B:

Apply fast programmatic validators for JSON shape, type constraints, allowed values, and deterministic business rules; subjective model grading is unnecessary and less reliable for an objectively checkable parser contract; this alternative would measure zero schema violations accepted by the downstream consumer across generated cases; it does not establish per-stratum pass rates with no uncovered critical scenario classes for the described use case.

D:

Use a criterion-based expert rubric, optionally supported by calibrated model grading, for semantic accuracy and appropriateness; pure string comparison is unsuitable when many substantively correct phrasings exist; this alternative would measure expert-rated factual fidelity and calibrated grader agreement on sampled explanations; it does not establish per-stratum pass rates with no uncovered critical scenario classes for the described use case.

Question 107

A finance automation team wants to compare a new posting policy with historical transactions without risking live corrections. Elsewhere, overall support resolution improves while customers using an assistive channel report worse outcomes. Which TWO evaluation strategies address these risks? Choose TWO.

  1. Release canaries with guardrail metrics and rollback conditions
  2. Require an explicit rollback trigger for harmful regressions
  3. Predefine success criteria and sample-size assumptions
  4. Inspect segment-level effects rather than only global lift
  5. Use shadow evaluation before enabling new side effects

Explanation

Correct Answers

D:

Compare treatment effects across predeclared critical segments and require safeguards where harm would be unacceptable; segment-aware decisions expose concentrated regressions before full rollout; its corresponding workstream should be validated by resolution and safety deltas by channel and accessibility segment.

E:

Replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; non-mutating evidence reduces risk before enabling operational authority; its corresponding workstream should be validated by shadow-policy agreement and zero write side effects across replayed billing cases.

Incorrect Answers

A:

Deploy to a bounded cohort while observing quality, errors, and tail latency, with predefined stop and rollback thresholds; a single offline score cannot show all production interactions or blast radius; its natural acceptance measure is canary threshold compliance and time to revert a seeded regression, which does not satisfy the separate workstreams being assessed.

B:

Establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; monitoring without an action threshold can document harm while allowing it to continue; its natural acceptance measure is time from triggered guardrail breach to verified rollback, which does not satisfy the separate workstreams being assessed.

C:

Define primary metrics, minimal meaningful effect, sample-size method, and stopping rules before examining results; repeatedly peeking and stopping at a favorable fluctuation inflates false-positive decisions; its natural acceptance measure is pre-registered experiment compliance and interval estimates at the planned sample size, which does not satisfy the separate workstreams being assessed.

Question 108

An integration architecture is being transferred to a new technical lead months after the initial decision. Which review result would demonstrate that earlier trade-offs and assumptions remain auditable?

  1. Stakeholder identification of boundaries and fallback responsibilities after the demo
  2. Completion and ownership of high-severity mitigations before the release gate
  3. Reconstructability of key design decisions after a simulated staff handoff
  4. Sponsor comprehension of expected outcomes, decision options, and risk ownership

Explanation

Correct Answer

C:

Decision records preserve rationale beyond a particular architect's memory; measure reconstructability of key design decisions after a simulated staff handoff at the release gate; use an ADR recording context, considered options, decision, trade-offs, implications, and conditions for revisiting it.

Incorrect Answers

A:

Understanding checks reveal whether decision-makers grasp operational limits; use representative happy-path and failure-path demonstrations with explicit fallback, uncertainty, and human ownership; this tests stakeholder identification of boundaries and fallback responsibilities after the demo; the actual release condition is reconstructability of key design decisions after a simulated staff handoff.

B:

A control register makes risk decisions reviewable rather than rhetorical; record probability and impact where appropriate, control owners, evidence, residual risk, and required approvals; this tests completion and ownership of high-severity mitigations before the release gate; the actual release condition is reconstructability of key design decisions after a simulated staff handoff.

D:

Decision-focused review checks whether the audience can act on the information; summarize business value, alternatives, cost envelope, measured evidence, and requested decisions in concise nontechnical terms; this tests sponsor comprehension of expected outcomes, decision options, and risk ownership; the actual release condition is reconstructability of key design decisions after a simulated staff handoff.

Question 109

A claims advice assistant must answer covered policy cases but defer when evidence is missing. Which approach best meets the stated requirement?

  1. Use task-correctness metrics tied to verified labels
  2. Test whether the tool layer denies unauthorized requests
  3. Measure rate and severity of safety policy violations
  4. Evaluate whether abstention and escalation are calibrated

Explanation

Correct Answer

D:

Measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; coverage-risk curves expose the tradeoff between reliability and unnecessary handoffs; evaluate the design with correct-answer rate at each supported coverage threshold including no-answer cases.

Incorrect Answers

A:

Measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; positive conversational tone does not establish that a legally or financially meaningful label is correct; this alternative would measure critical-class precision and recall on independently verified tax-code cases; it does not establish correct-answer rate at each supported coverage threshold including no-answer cases for the described use case.

B:

Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; model refusal scores can be high even when the underlying API is vulnerable; this alternative would measure unauthorized API requests rejected under role, tenant, and resource attack matrices; it does not establish correct-answer rate at each supported coverage threshold including no-answer cases for the described use case.

C:

Define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; general task correctness does not measure policy-conformance failures on dangerous requests; this alternative would measure severity-weighted policy violation incidence on a reviewed adversarial test set; it does not establish correct-answer rate at each supported coverage threshold including no-answer cases for the described use case.

Question 110

A retrieval connector is suspected of increasing end-user latency. Before scaling an instrumentation change, which evidence would identify the slow stage rather than attributing the entire delay to the model?

  1. Capture rate of seeded critical failures at a bounded telemetry cost
  2. Trace completeness and correct bottleneck attribution on instrumented fault injections
  3. Stage-specific p95 deltas identifying injected bottlenecks
  4. Time to detect and stop a simulated unauthorized bulk-write burst

Explanation

Correct Answer

C:

Budget attribution points remediation at the stage that truly regressed; measure stage-specific p95 deltas identifying injected bottlenecks at the release gate; instrument per-stage timing and compare each segment against its allocated latency budget before changing model configuration.

Incorrect Answers

A:

Risk-aware sampling preserves evidence for rare faults without uncontrolled storage growth; retain error and high-risk traces at a higher rate while sampling successful routine traffic with privacy controls; this tests capture rate of seeded critical failures at a bounded telemetry cost; the actual release condition is stage-specific p95 deltas identifying injected bottlenecks.

B:

Distributed spans expose ownership boundaries across the whole execution path; end-to-end traces with consistent request and span identifiers identify which service and step consumed time; this tests trace completeness and correct bottleneck attribution on instrumented fault injections; the actual release condition is stage-specific p95 deltas identifying injected bottlenecks.

D:

Action telemetry and response drills measure whether harmful changes can be contained promptly; monitor write volume, denials, unusual resource patterns, and action-specific thresholds with fast containment routing; this tests time to detect and stop a simulated unauthorized bulk-write burst; the actual release condition is stage-specific p95 deltas identifying injected bottlenecks.

Claude Certified Architect - Professional (CCAR-P) Practice Tests