Question 320
Support staff have no clear threshold or owner for switching to manual handling. Which corrective change best addresses the root cause?
- Set observability and reporting expectations with stakeholders
- Establish a documented escalation path for service incidents
- Route user feedback into a triaged, owned backlog
- Define SLOs and error budgets for agreed user outcomes
Explanation
Correct Answer
B:Define incident severity, on-call ownership, manual fallback authority, and customer communication triggers; incident response tests verify that operational ownership works under pressure; verify the repair using time to appropriate escalation and fallback activation during drills.
Incorrect Answers
A:Platform telemetry is necessary but not equivalent to business service reporting; agree on outcome-aligned service reports, cadence, escalation thresholds, and audiences during solution definition; that control is assessed by delivery of agreed business and risk indicators on schedule with decision-ready interpretation; the incident instead requires time to appropriate escalation and fallback activation during drills.
C:Unstructured rating totals cannot by themselves direct a safe corrective action; classify feedback by severity and root-cause type, assign owners, link trace IDs, and feed validated cases into regression tests; that control is assessed by time from actionable feedback to triage, owner assignment, and verified fix; the incident instead requires time to appropriate escalation and fallback activation during drills.
D:Generic uptime claims can conceal slow or unusable answer experiences; establish measurable user-facing service objectives, reporting windows, and error-budget policies tied to accountable actions; that control is assessed by sLO compliance and prescribed actions during a simulated error-budget breach; the incident instead requires time to appropriate escalation and fallback activation during drills.
Question 321
A procurement service enforces spending thresholds at its backend endpoint in addition to the assistant instructions. Security testers bypass Claude and call the API directly. Which result proves the authorization design is effective?
- Rejection of unauthorized tool calls even when sent outside the model workflow
- Zero critical network changes without a recorded action-specific approval
- Zero cross-user CRM record disclosures in permission-matrix tests
- Credential-expiry and scope-denial tests after simulated token compromise
- Denial of cross-tenant identifier enumeration attempts in direct tool tests
- Audit reconstruction of actor, authority, target, and result for sampled mutations
Explanation
Correct Answer
A:Direct API tests distinguish true backend policy from model cooperation; measure rejection of unauthorized tool calls even when sent outside the model workflow at the release gate; implement enforceable role, amount, and resource checks at the action service, not just in natural-language instructions.
Incorrect Answers
B:Approval-path tests ensure the precise mutation was authorized before execution; bind high-impact actions to a separately verified approval and narrowly scoped execution credential; this tests zero critical network changes without a recorded action-specific approval; the actual release condition is rejection of unauthorized tool calls even when sent outside the model workflow.
C:User-context tests verify enforcement of the source system's individual access rules; carry an appropriate user-bound authorization context to downstream services and enforce record-level access per request; this tests zero cross-user CRM record disclosures in permission-matrix tests; the actual release condition is rejection of unauthorized tool calls even when sent outside the model workflow.
D:Short validity and minimum scope bound the impact of a leaked credential; prefer tightly scoped, expiring credentials with secure rotation and audited issuance for the precise operation; this tests credential-expiry and scope-denial tests after simulated token compromise; the actual release condition is rejection of unauthorized tool calls even when sent outside the model workflow.
E:Object-level access checks close the horizontal privilege gap that prompt filtering misses; check the requested resource against the authenticated tenant and role at the data service rather than trusting user-provided identifiers; this tests denial of cross-tenant identifier enumeration attempts in direct tool tests; the actual release condition is rejection of unauthorized tool calls even when sent outside the model workflow.
F:End-to-end attribution makes unauthorized use investigable rather than anonymous; propagate authenticated actor, tenant, delegated scope, request ID, and action outcome into tamper-resistant audit records; this tests audit reconstruction of actor, authority, target, and result for sampled mutations; the actual release condition is rejection of unauthorized tool calls even when sent outside the model workflow.
Question 322
A new safety filter lowers observed policy alerts but starts blocking ordinary support requests. Which monitoring practice would reveal the trade-off between safety incidents and false-positive blocks?
- Detect distribution and quality drift in ongoing samples
- Track safety policy events across product updates
- Compare production metrics by model and prompt version
- Instrument end-to-end execution spans for causal debugging
- Alert on unexpected token and cost anomalies
Explanation
Correct Answer
B:Monitor both harmful-output violations and false-positive blocks by version and user segment; paired monitoring distinguishes beneficial safety controls from excessive refusals; verify the repair using safety precision/recall and legitimate-task completion before and after the change.
Incorrect Answers
A:A static validation set cannot reflect every change in real input distribution over time; periodically sample representative production outputs for labeling, segment analysis, and drift signals without retaining unnecessary personal data; that control is assessed by quality delta between recent audited production samples and the launch baseline; the incident instead requires safety precision/recall and legitimate-task completion before and after the change.
C:A global metric is inadequate when several behaviors are concurrently deployed; partition traces and scored outcomes by deployed configuration, using release identifiers to support rollback and diagnosis; that control is assessed by detection of version-specific quality regressions in a staged rollout; the incident instead requires safety precision/recall and legitimate-task completion before and after the change.
D:Model usage totals do not reveal which integration failed within a particular request; correlation and span IDs across components allow latency and failure attribution along the actual path; that control is assessed by percent of failures with a complete trace to the responsible stage; the incident instead requires safety precision/recall and legitimate-task completion before and after the change.
E:Request-volume alerts alone miss a sudden rise in tokens per individual workflow; monitor input/output tokens, cache reads, per-stage costs, and cost-per-task anomalies with accountable ownership; that control is assessed by detection lag for injected per-request token inflation; the incident instead requires safety precision/recall and legitimate-task completion before and after the change.
Question 323
The pilot reports impressive demonstrations yet cannot quantify savings against the existing claims process. Which intervention targets the defect rather than its symptoms?
- Measure the current manual task before choosing automation
- Define which decisions remain with people
- Pilot one bounded workflow with acceptance gates
- Translate vague accuracy demands into task-specific criteria
Explanation
Correct Answer
A:A baseline separates real business improvement from attractive demos; use completed cases, reviewer effort, errors, and exception handling as measurable comparators; comparing complete case outcomes includes downstream corrections rather than rewarding only faster draft generation; verify the repair using change in verified minutes and rework per adjudicated claim versus the documented baseline.
Incorrect Answers
B:Decision ownership is an authorization and accountability control; it does not fix missing input records or prompt-token waste; explicit decision-rights boundaries distinguish advisory generation from actions requiring a credentialed human or deterministic policy engine; that control is assessed by share of binding decisions showing the required named approver and rationale; the incident instead requires change in verified minutes and rework per adjudicated claim versus the documented baseline.
C:A pilot bounds deployment exposure; it does not substitute for an authorization model or document freshness; a narrow pilot validates one representative flow and pre-agreed quality, latency, and escalation thresholds before scaling to other workflows; that control is assessed by passing the acceptance gate on a representative pilot cohort including difficult exceptions; the incident instead requires change in verified minutes and rework per adjudicated claim versus the documented baseline.
D:A clear success rubric evaluates outcomes but cannot on its own implement retrieval access controls; convert vague requests into observable task outcomes, rubric-based quality dimensions, and failure thresholds that stakeholders can agree to test; that control is assessed by agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set; the incident instead requires change in verified minutes and rework per adjudicated claim versus the documented baseline.
Question 324
A policy QA assistant has stable launch metrics but receives novel questions and changing source documents. Which design decision directly addresses this constraint?
- Set an evidence gate before moving beyond discovery
- Include security and governance checks during design
- Use bounded pilots with exit criteria before expansion
- Run a recurring monitoring-to-improvement loop
- Transfer durable ownership and operations before handoff
- Plan deprecation and rollback from the start
Explanation
Correct Answer
D:Feed production samples and feedback into versioned evals, prioritize changes, and release improvements behind regression gates; continuous assessment links real usage to controlled change; evaluate the design with cycle time from detected drift to validated improvement without safety regression.
Incorrect Answers
A:Require validated problem scope, value hypothesis, data authority, and feasibility evidence before full design commitment; a procurement timeline does not eliminate uncertainty about whether the problem is solvable; this alternative would measure closure of high-risk feasibility questions before design approval; it does not establish cycle time from detected drift to validated improvement without safety regression for the described use case.
B:Incorporate threat modeling, authorization, data processing, and human oversight into architecture acceptance before implementation hardens; post-build security sign-off is costly and may not undo prior unauthorized processing; this alternative would measure resolution of high-severity architecture risks before implementation approval; it does not establish cycle time from detected drift to validated improvement without safety regression for the described use case.
C:Define pilot size, critical acceptance metrics, escalation paths, and scale-up criteria before broad deployment; more launch enthusiasm does not replace production-like testing under load and edge conditions; this alternative would measure pilot gate pass rate with defined performance and safety thresholds; it does not establish cycle time from detected drift to validated improvement without safety regression for the described use case.
E:Establish production owners for reliability, security, spend, data, prompts, and escalation, with verified access and runbooks; a completed repository is not proof of ongoing service operability; this alternative would measure named owner coverage and support-drill completion at handoff; it does not establish cycle time from detected drift to validated improvement without safety regression for the described use case.
F:Inventory consumers, define migration dates and fallback, preserve data obligations, and test the rollback or retirement path; removing an old endpoint alone does not manage hidden dependencies or operational continuity; this alternative would measure successful retirement with zero unplanned dependent-system failures; it does not establish cycle time from detected drift to validated improvement without safety regression for the described use case.
Question 325
After an outage, operators cannot tell which dependency failed because logs contain only the final error text. Which corrective change best addresses the root cause?
- Instrument end-to-end execution spans for causal debugging
- Track safety policy events across product updates
- Associate human corrections with exact release versions
- Detect distribution and quality drift in ongoing samples
- Alert on unexpected token and cost anomalies
Explanation
Correct Answer
A:Correlation and span IDs across components allow latency and failure attribution along the actual path; trace completeness measures incident diagnosability rather than raw log volume; verify the repair using percent of failures with a complete trace to the responsible stage.
Incorrect Answers
B:A falling count of harmful completions can coexist with an unusable overrestrictive assistant; monitor both harmful-output violations and false-positive blocks by version and user segment; that control is assessed by safety precision/recall and legitimate-task completion before and after the change; the incident instead requires percent of failures with a complete trace to the responsible stage.
C:Aggregate thumbs-down counts without version linkage cannot identify regressions or prove a fix; log consented feedback alongside prompt revision, model route, source version, and case context for reproducible analysis; that control is assessed by reproduction of sampled reviewer defects against their original configuration; the incident instead requires percent of failures with a complete trace to the responsible stage.
D:A static validation set cannot reflect every change in real input distribution over time; periodically sample representative production outputs for labeling, segment analysis, and drift signals without retaining unnecessary personal data; that control is assessed by quality delta between recent audited production samples and the launch baseline; the incident instead requires percent of failures with a complete trace to the responsible stage.
E:Request-volume alerts alone miss a sudden rise in tokens per individual workflow; monitor input/output tokens, cache reads, per-stage costs, and cost-per-task anomalies with accountable ownership; that control is assessed by detection lag for injected per-request token inflation; the incident instead requires percent of failures with a complete trace to the responsible stage.
Question 326
A warranty-routing prototype uses a concise category definition with no demonstrations. Reviewers compare it with a lengthy example-heavy prompt. Which outcome would justify keeping the simpler approach?
- Explanations generated only for records that passed the eligibility stage
- Intermediate artifact validity and final score agreement with a policy-engine oracle
- Label accuracy and token use versus a heavier example-based prompt
- Correct separation of document claims and application instructions in adversarial input tests
Explanation
Correct Answer
C:Controlled comparisons show whether added prompt complexity yields a genuine benefit; measure label accuracy and token use versus a heavier example-based prompt at the release gate; a concise task statement with precise labels is a defensible baseline when success is readily evaluated and examples add little value.
Incorrect Answers
A:A prerequisite gate catches unvalidated reasoning before downstream text is produced; use a staged prompt chain where a verified intermediate classification is required before the next generation step; this tests explanations generated only for records that passed the eligibility stage; the actual release condition is label accuracy and token use versus a heavier example-based prompt.
B:Checkable milestones provide evidence of reasoning quality beyond rhetorical fluency; require explicit intermediate evidence and structured rule checks that can be independently validated; avoid treating verbose reasoning text as proof; this tests intermediate artifact validity and final score agreement with a policy-engine oracle; the actual release condition is label accuracy and token use versus a heavier example-based prompt.
D:The acceptance set targets role confusion caused by mixed prompt sections; use clearly named sections or XML-style tags so Claude can distinguish task policy, input, and supporting documents; this tests correct separation of document claims and application instructions in adversarial input tests; the actual release condition is label accuracy and token use versus a heavier example-based prompt.
Question 327
A support system treats deterministic status explanations as if they required its most costly reasoning model. What is the most targeted architecture choice?
- Route straightforward requests to a validated economical model
- Queue non-urgent work for bounded batch processing
- Reuse expensive stable prefixes through prompt caching
- Parallelize independent retrieval calls on the critical path
Explanation
Correct Answer
A:Evaluate a smaller model on straightforward request strata and escalate only where measurable capability is needed; stratified monitoring confirms economic gains do not displace expensive errors onto users; evaluate the design with accepted resolution rate and cost per case across routed difficulty levels.
Incorrect Answers
B:Use suitable batch orchestration and queue control to trade immediate response time for predictable throughput and expense; batch pipelines are inappropriate where a person needs a subsecond answer for an active decision; this alternative would measure cost and successful throughput per nightly processing window; it does not establish accepted resolution rate and cost per case across routed difficulty levels for the described use case.
C:Keep a stable eligible prompt prefix and measure cache-read tokens to reduce repeated processing; selecting a smaller model alone may reduce price but does not address wasted recomputation of invariant content; this alternative would measure cache-read rate, p95 latency, and cost per accepted answer under repeated queries; it does not establish accepted resolution rate and cost per case across routed difficulty levels for the described use case.
D:Parallel retrieval with per-call budgets reduces wall time while preserving authorization and complete aggregation; parallelism adds little if operations depend on each other or contend for the same rate-limited service; this alternative would measure p95 response time with all three source checks present under load; it does not establish accepted resolution rate and cost per case across routed difficulty levels for the described use case.
Question 328
A travel-support workflow requests multiple independent sources before writing one response. Load tests compare sequential and concurrent fetching. Which metric demonstrates a benefit without omitting essential evidence?
- End-to-end p95 improvement with all required sources still present
- Accuracy by difficulty segment and unit cost under p95 constraints
- Grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths
- Deadline compliance and correct indication of omitted enrichment
- Repeated-query latency with zero stale or cross-scope cache responses after updates
Explanation
Correct Answer
A:Concurrency wins only if the final synthesis retains complete inputs; measure end-to-end p95 improvement with all required sources still present at the release gate; execute independent lookups concurrently with bounded timeouts and aggregate the results after all necessary dependencies complete.
Incorrect Answers
B:Joint segmented reporting reveals whether expensive processing is spent where it helps; use an evaluated routing policy that sends straightforward lookups to a light path and escalates uncertain cases to more capable reasoning; this tests accuracy by difficulty segment and unit cost under p95 constraints; the actual release condition is end-to-end p95 improvement with all required sources still present.
C:A quality-latency frontier shows whether an extra ranking pass is worth its overhead; retrieve a bounded candidate set and rerank against the actual question to concentrate useful evidence without feeding all candidates to the model; this tests grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths; the actual release condition is end-to-end p95 improvement with all required sources still present.
D:Timed fault injection confirms graceful degradation without inventing evidence; keep the critical verified signal independent and return a clearly limited result when optional enrichment exceeds its budget; this tests deadline compliance and correct indication of omitted enrichment; the actual release condition is end-to-end p95 improvement with all required sources still present.
E:Freshness and authorization checks are essential alongside speed gains; use a cache keyed by query, permissions where relevant, and content version, with invalidation tied to source updates; this tests repeated-query latency with zero stale or cross-scope cache responses after updates; the actual release condition is end-to-end p95 improvement with all required sources still present.
Question 329
An enterprise search system migrates from one embedding configuration to another. During transition tests, which result proves query vectors and indexed records remain compatible enough for relevant material to be found?
- Retrieval of both governing clause and exception for labeled contract questions
- Publication-to-search freshness lag and absence of withdrawn clauses in results
- Unauthorized chunk exposure rate under staff permission matrices, targeting zero
- Fraction of material claims linked to resolvable source locations
- Retrieval recall before and after a complete embedding-index migration
Explanation
Correct Answer
E:Version-aligned search tests detect vector-compatibility regressions; measure retrieval recall before and after a complete embedding-index migration at the release gate; record embedding model and preprocessing versions, re-embed consistently, and use controlled index migration.
Incorrect Answers
A:Structure-aware test cases measure preservation of legally relevant context; chunk by headings and semantic sections with deliberate overlap where necessary while maintaining parent-document references; this tests retrieval of both governing clause and exception for labeled contract questions; the actual release condition is retrieval recall before and after a complete embedding-index migration.
B:Update/delete tests prove the index tracks the authoritative corpus; build a monitored ingestion path for adds, updates, deletes, and effective dates, with freshness checks and removal of superseded chunks; this tests publication-to-search freshness lag and absence of withdrawn clauses in results; the actual release condition is retrieval recall before and after a complete embedding-index migration.
C:The security test must inspect retrieved inputs, not just final visible answers; apply document permissions to retrieval candidates before material enters model context or logs; this tests unauthorized chunk exposure rate under staff permission matrices, targeting zero; the actual release condition is retrieval recall before and after a complete embedding-index migration.
D:Citation resolution testing verifies that evidence can actually be audited; store source IDs, locations, dates, and stable links with chunks and propagate them to answer citation validation; this tests fraction of material claims linked to resolvable source locations; the actual release condition is retrieval recall before and after a complete embedding-index migration.
Question 330
A spare-parts assistant must locate a precise equipment code and its corresponding revision suffix. Which option addresses the immediate design need?
- Use semantic vector search for paraphrased knowledge questions
- Query structured data through authorized SQL or typed APIs
- Use lexical search for exact identifiers and literals
- Rerank candidate passages for question relevance
Explanation
Correct Answer
C:Keyword or lexical matching is effective for codes, exact phrases, and terms whose characters carry the meaning; identifier-focused tests reveal errors that a semantic relevance score may ignore; evaluate the design with exact-ID hit rate at the top retrieval ranks.
Incorrect Answers
A:Vector retrieval can match conceptual similarity between user wording and document text, followed by evidence verification; semantic search is not guaranteed to respect exact version strings, numerals, or access boundaries without additional controls; this alternative would measure recall on human-authored paraphrases absent from manual headings; it does not establish exact-ID hit rate at the top retrieval ranks for the described use case.
B:Use a governed structured query interface with parameter checks and business semantics for exact aggregates; unstructured document retrieval cannot ensure completeness or mathematical correctness for database totals; this alternative would measure exact aggregate agreement with a trusted query on authorized records; it does not establish exact-ID hit rate at the top retrieval ranks for the described use case.
D:Apply a query-aware reranker after broad retrieval, respecting freshness and permissions, to prioritize evidence for synthesis; a bigger first-stage candidate set without selection can dilute context and increase cost; this alternative would measure relevant-procedure rank and end-to-end answer support rate after reranking; it does not establish exact-ID hit rate at the top retrieval ranks for the described use case.