Question 300
An insurance operations pilot now handles both ordinary and disputed claims. Finance wants to know whether the proposed assistant improves actual adjudication rather than merely generating drafts quickly. What measured comparison would justify the business case?
- Passing the acceptance gate on a representative pilot cohort including difficult exceptions
- Agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set
- Accepted cases per reviewer-hour with exception share tracked separately
- Percentage of required fields with validated owners, quality checks, and approved processing purposes
- Change in verified minutes and rework per adjudicated claim versus the documented baseline
Explanation
Correct Answer
E:Comparing complete case outcomes includes downstream corrections rather than rewarding only faster draft generation; measure change in verified minutes and rework per adjudicated claim versus the documented baseline at the release gate; a baseline separates real business improvement from attractive demos; use completed cases, reviewer effort, errors, and exception handling as measurable comparators.
Incorrect Answers
A:A bounded deployment can prove business fitness before a large enterprise-wide dependency is created; a narrow pilot validates one representative flow and pre-agreed quality, latency, and escalation thresholds before scaling to other workflows; this tests passing the acceptance gate on a representative pilot cohort including difficult exceptions; the actual release condition is change in verified minutes and rework per adjudicated claim versus the documented baseline.
B:Consistent scoring makes a go-live decision defensible rather than impressionistic; convert vague requests into observable task outcomes, rubric-based quality dimensions, and failure thresholds that stakeholders can agree to test; this tests agreement between domain reviewers on a rubric and pass rate against a labeled acceptance set; the actual release condition is change in verified minutes and rework per adjudicated claim versus the documented baseline.
C:Counting both throughput and escalations prevents a fast happy path from hiding a costly exception queue; select a high-frequency, well-described task with measurable work reduction and explicit escalation for ambiguous records; this tests accepted cases per reviewer-hour with exception share tracked separately; the actual release condition is change in verified minutes and rework per adjudicated claim versus the documented baseline.
D:Measured coverage of eligible sources exposes blockers that a high demo score conceals; data-readiness and usage-rights mapping identifies legitimate inputs, owners, quality gaps, and prohibited transfers before committing to an implementation; this tests percentage of required fields with validated owners, quality checks, and approved processing purposes; the actual release condition is change in verified minutes and rework per adjudicated claim versus the documented baseline.
Question 301
A developer uses Claude to modify authorization middleware in a multi-tenant application. Which intervention is most appropriate for this use case?
- Package reusable engineering instructions as governed Skills
- Use test-first acceptance cases for generated changes
- Review AI-generated diffs for logic and security regressions
- Break large refactors into independently reviewable changes
Explanation
Correct Answer
C:Require focused code review of the actual diff, threat-relevant tests, and approval before merging sensitive changes; diff-focused review finds changes that ordinary output fluency cannot reveal; evaluate the design with detection of seeded authorization regressions during peer review and CI.
Incorrect Answers
A:Maintain task-scoped reusable Skills or playbooks with versioning, ownership, and documented prerequisites; shared instructions improve consistency but do not replace enforced approval or automated tests; this alternative would measure checklist completeness across independent sessions using the reviewed Skill version; it does not establish detection of seeded authorization regressions during peer review and CI for the described use case.
B:Define executable tests and expected behavior before relying on AI-generated implementation, then review failures and iterate; merely reading generated code for plausibility does not replace tests of business invariants; this alternative would measure passing unit and property tests including specified rounding boundaries; it does not establish detection of seeded authorization regressions during peer review and CI for the described use case.
D:Use small scoped changes, compatibility checkpoints, and separate pull requests tied to specific tests; one massive patch can reduce initial coordination but increases regression and rollback risk; this alternative would measure review time and defect localization per incremental change set; it does not establish detection of seeded authorization regressions during peer review and CI for the described use case.
Question 302
A claims assistant will change processes for adjusters, legal reviewers, security, and data stewards. Before acceptance, the sponsor needs evidence that all affected roles have contributed their requirements. Which measure best demonstrates coverage?
- Measured percentage of required sources with approved owners, legal uses, and integration paths
- Signed acceptance criteria mapped to demonstrable tests and named owners
- Resolution rate of prioritized unknowns through measured prototypes
- Coverage of approved requirements traced to affected stakeholder groups
- Verified readiness of high-impact external dependencies against test quotas and outage cases
Explanation
Correct Answer
D:Traceable requirements reveal whether important operational perspectives were omitted; measure coverage of approved requirements traced to affected stakeholder groups at the release gate; use role-based discovery to capture goals, current workflows, authority boundaries, and failure impacts from relevant stakeholders.
Incorrect Answers
A:Data ownership completeness measures feasibility before engineering spends effort; create a source inventory with accountable owners, sensitivity, update rate, and approval dependencies before committing to the design; this tests percentage of required sources with approved owners, legal uses, and integration paths; the actual release condition is coverage of approved requirements traced to affected stakeholder groups.
B:An explicit agreement limits scope drift and creates a common release standard; document in-scope outcomes, exclusions, metrics, responsibilities, and go-live gates for accountable approval; this tests signed acceptance criteria mapped to demonstrable tests and named owners; the actual release condition is coverage of approved requirements traced to affected stakeholder groups.
C:Evidence-based discovery reduces commitment to unverified technical assumptions; identify high-risk assumptions and design small feasibility spikes with explicit success and stop criteria; this tests resolution rate of prioritized unknowns through measured prototypes; the actual release condition is coverage of approved requirements traced to affected stakeholder groups.
E:Dependency validation closes gaps between prototype success and operational feasibility; maintain a dependency register covering third-party SLAs, quotas, outage paths, and contractual conditions with mitigation owners; this tests verified readiness of high-impact external dependencies against test quotas and outage cases; the actual release condition is coverage of approved requirements traced to affected stakeholder groups.
Question 303
The first overnight incident remains unresolved because operators cannot identify the service owner or safe restart path. What is the most direct remediation?
- Maintain an end-to-end data-flow and trust-boundary diagram
- Version prompts, model routes, tools, and evaluation artifacts together
- Publish an incident runbook with containment and rollback steps
- Prepare operator onboarding and support handoff materials
Explanation
Correct Answer
D:Provide system overview, access setup, troubleshooting flowcharts, monitoring thresholds, and named support responsibilities; independent operator readiness is stronger evidence than architect-led demonstrations; verify the repair using successful completion of standard support drills by staff who did not build the system.
Incorrect Answers
A:A feature list does not reveal crossing of sensitive trust boundaries; a current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; that control is assessed by completeness of sensitive-data flow paths identified during security review; the incident instead requires successful completion of standard support drills by staff who did not build the system.
B:Recording only the model name omits other behavior-changing dependencies; maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; that control is assessed by reproducibility of a sampled historical answer using its full manifest; the incident instead requires successful completion of standard support drills by staff who did not build the system.
C:A high-level diagram alone is not an executable incident response procedure; maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; that control is assessed by successful containment time in a simulated stale-index incident; the incident instead requires successful completion of standard support drills by staff who did not build the system.
Question 304
A public-facing assistant faces deliberate boundary-testing prompts alongside ordinary conversations. A safety review wants to weight serious violations more heavily than harmless phrasing issues. Which release metric fits that requirement?
- Fully loaded cost per correctly completed document
- Severity-weighted policy violation incidence on a reviewed adversarial test set
- Unauthorized API requests rejected under role, tenant, and resource attack matrices
- Critical-class precision and recall on independently verified tax-code cases
- Correct-answer rate at each supported coverage threshold including no-answer cases
Explanation
Correct Answer
B:Risk-weighted scoring detects rare critical violations that a single average hides; measure severity-weighted policy violation incidence on a reviewed adversarial test set at the release gate; define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic.
Incorrect Answers
A:This metric includes rework rather than rewarding low-quality cheap drafts; relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; this tests fully loaded cost per correctly completed document; the actual release condition is severity-weighted policy violation incidence on a reviewed adversarial test set.
C:Direct penetration-style checks assess the enforceable security boundary; treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; this tests unauthorized API requests rejected under role, tenant, and resource attack matrices; the actual release condition is severity-weighted policy violation incidence on a reviewed adversarial test set.
D:Label-based metrics catch mistakes that surface-level satisfaction may conceal; measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; this tests critical-class precision and recall on independently verified tax-code cases; the actual release condition is severity-weighted policy violation incidence on a reviewed adversarial test set.
E:Coverage-risk curves expose the tradeoff between reliability and unnecessary handoffs; measure selective risk: error rate on answered items alongside abstention precision, coverage, and escalation cost; this tests correct-answer rate at each supported coverage threshold including no-answer cases; the actual release condition is severity-weighted policy violation incidence on a reviewed adversarial test set.
Question 305
A warehouse assistant passes demos but has never faced peak concurrency or malformed inputs. What is the most defensible next step?
- Include security and governance checks during design
- Use bounded pilots with exit criteria before expansion
- Transfer durable ownership and operations before handoff
- Run a recurring monitoring-to-improvement loop
Explanation
Correct Answer
B:Define pilot size, critical acceptance metrics, escalation paths, and scale-up criteria before broad deployment; a controlled pilot turns rollout into a measurable decision; evaluate the design with pilot gate pass rate with defined performance and safety thresholds.
Incorrect Answers
A:Incorporate threat modeling, authorization, data processing, and human oversight into architecture acceptance before implementation hardens; post-build security sign-off is costly and may not undo prior unauthorized processing; this alternative would measure resolution of high-severity architecture risks before implementation approval; it does not establish pilot gate pass rate with defined performance and safety thresholds for the described use case.
C:Establish production owners for reliability, security, spend, data, prompts, and escalation, with verified access and runbooks; a completed repository is not proof of ongoing service operability; this alternative would measure named owner coverage and support-drill completion at handoff; it does not establish pilot gate pass rate with defined performance and safety thresholds for the described use case.
D:Feed production samples and feedback into versioned evals, prioritize changes, and release improvements behind regression gates; a one-time launch test cannot guarantee reliability under changing input distributions; this alternative would measure cycle time from detected drift to validated improvement without safety regression; it does not establish pilot gate pass rate with defined performance and safety thresholds for the described use case.
Question 306
A company rolls it out to everyone and discovers a widespread delay without a quick rollback plan. Which intervention targets the defect rather than its symptoms?
- Use shadow evaluation before enabling new side effects
- Predefine success criteria and sample-size assumptions
- Release canaries with guardrail metrics and rollback conditions
- Require an explicit rollback trigger for harmful regressions
- Randomize consistently at the user or account level
Explanation
Correct Answer
C:Deploy to a bounded cohort while observing quality, errors, and tail latency, with predefined stop and rollback thresholds; controlled exposure and rollback readiness limit the damage of an unexpected release; verify the repair using canary threshold compliance and time to revert a seeded regression.
Incorrect Answers
A:A normal A/B test is inappropriate if experimental behavior can cause unreviewed irreversible changes; replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; that control is assessed by shadow-policy agreement and zero write side effects across replayed billing cases; the incident instead requires canary threshold compliance and time to revert a seeded regression.
B:Repeatedly peeking and stopping at a favorable fluctuation inflates false-positive decisions; define primary metrics, minimal meaningful effect, sample-size method, and stopping rules before examining results; that control is assessed by pre-registered experiment compliance and interval estimates at the planned sample size; the incident instead requires canary threshold compliance and time to revert a seeded regression.
D:Monitoring without an action threshold can document harm while allowing it to continue; establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; that control is assessed by time from triggered guardrail breach to verified rollback; the incident instead requires canary threshold compliance and time to revert a seeded regression.
E:Request-by-request randomization can be invalid when outcomes depend on ongoing user interactions; assign a stable randomized unit aligned to the behavior being measured and maintain consistent treatment during the experiment; that control is assessed by balanced treatment cohorts and clean user-level exposure histories; the incident instead requires canary threshold compliance and time to revert a seeded regression.
Question 307
A legacy document archive is transferred to a night-time processing lane. The sponsor cares about reliable completed classifications and unit cost rather than interactive response speed. Which evidence should decide acceptance?
- Completed valid classifications per batch window and cost per accepted record
- Summary fidelity and p95 completion time without any unapproved tool invocation
- Resolution of unseen diagnostic branches within a specified tool-call and time budget
- Correct citations to currently effective policy versions across update tests
Explanation
Correct Answer
A:Throughput and unit economics measure the benefit that a user-facing latency metric would miss; measure completed valid classifications per batch window and cost per accepted record at the release gate; batch-oriented orchestration handles throughput, retries, and cost planning while preserving consistent model instructions for a noninteractive workload.
Incorrect Answers
B:A bounded call should meet the content rubric without introducing an unnecessary action surface; for a fixed transformation, a prompt enriched with only needed context is simpler, cheaper, and easier to evaluate than an agentic loop; this tests summary fidelity and p95 completion time without any unapproved tool invocation; the actual release condition is completed valid classifications per batch window and cost per accepted record.
C:The agent must demonstrate flexible discovery without escaping its operational guardrails; bounded agentic planning chooses follow-up tools from observed evidence while enforcing time, budget, and tool permissions; this tests resolution of unseen diagnostic branches within a specified tool-call and time budget; the actual release condition is completed valid classifications per batch window and cost per accepted record.
D:Version-aware citation checks show whether answers rely on authoritative current evidence; retrieval from a versioned, access-filtered source grounds answers in current documents and can expose the authoritative citation; this tests correct citations to currently effective policy versions across update tests; the actual release condition is completed valid classifications per batch window and cost per accepted record.
Question 308
A technician's mobile repair assistant needs to combine photographs of equipment plates with typed notes. A separate factory deploys similar AI against noisy, multilingual, and very long maintenance reports absent from its clean development samples. Which TWO model-selection checks fit these needs? Choose TWO.
- Benchmark a smaller model for routine classification
- Use stronger reasoning capacity for genuinely complex decisions
- Evaluate model choice against the real input distribution
- Select a model path compatible with required modalities
- Use confidence-aware routing with a bounded fallback
Explanation
Correct Answers
C:Representative workload evaluation must include input quality, document lengths, languages, and failure consequences before setting model policy; stratified scores expose failures that an overall aggregate conceals; its corresponding workstream should be validated by per-segment acceptance rate across language, length, and noisy-input strata.
D:Confirm that the chosen Claude interface supports the required input modality and test image quality, extraction, and reasoning together; a multimodal acceptance set measures the capability absent from text-only trials; its corresponding workstream should be validated by correct equipment identification and recommendation accuracy on labeled photographs.
Incorrect Answers
A:Compare an efficient Claude model against labeled classification cases and select the lowest-cost model that meets the defined accuracy floor; a lightweight route should not be assumed suitable for complex multi-step decisions without a representative evaluation; its natural acceptance measure is macro-F1, critical-class recall, and cost per correctly triaged message on held-out traffic, which does not satisfy the separate workstreams being assessed.
B:Escalate tasks with demonstrated reasoning difficulty to a capable model after measuring gains on cases involving cross-document dependencies; high reasoning capacity is wasteful if the task only extracts a known field or matches a stable label; its natural acceptance measure is correct resolution of cross-exhibit conflicts at the required latency and review cost, which does not satisfy the separate workstreams being assessed.
E:Combine an evaluated uncertainty trigger with escalation to a stronger model or reviewer, using explicit limits and post-route quality measurement; blindly using one model tier on all cases neither controls the hardest error modes nor optimizes cost; its natural acceptance measure is high-risk escalation recall and total cost under the required error ceiling, which does not satisfy the separate workstreams being assessed.
Question 309
A reporting assistant is configured to reserve space for mandatory conclusions on unusually large inputs. Which result would demonstrate that the token-budget design works under worst-case documents?
- Correct recovery of approvals and pending actions after forced session reset
- Current-state citation accuracy across log-update sequences
- Retention of signed-off requirements after long simulated conversations
- Complete mandatory report sections under the worst supported input size
Explanation
Correct Answer
D:Budgeted completion tests directly reveal truncation at maximum expected load; measure complete mandatory report sections under the worst supported input size at the release gate; reserve output capacity and bound each stage's retrieved context and response length according to the deliverable structure.
Incorrect Answers
A:Durable-state testing proves behavior survives context eviction; store approvals and workflow state in an authoritative durable system; treat chat history as context, not a transaction ledger; this tests correct recovery of approvals and pending actions after forced session reset; the actual release condition is complete mandatory report sections under the worst supported input size.
B:Freshness-aware tests expose whether stale tool observations override newer facts; prune or summarize superseded tool data, preserve timestamps and current authoritative observations, and fetch details only when needed; this tests current-state citation accuracy across log-update sequences; the actual release condition is complete mandatory report sections under the worst supported input size.
C:A long-horizon test checks memory of obligations rather than raw conversation recall; store a structured, reviewable summary of decisions, open issues, and provenance; inject it at checkpoints rather than replaying every chat turn; this tests retention of signed-off requirements after long simulated conversations; the actual release condition is complete mandatory report sections under the worst supported input size.
Question 310
An answer regression cannot be reproduced because its tool schema and prompt revision are unknown. Which corrective change best addresses the root cause?
- Prepare operator onboarding and support handoff materials
- Keep an end-to-end data-flow and trust-boundary diagram
- Version prompts, model routes, tools, and evaluation artifacts together
- Publish an incident runbook with containment and rollback steps
Explanation
Correct Answer
C:Maintain a release manifest linking model configuration, prompt, tool contracts, data versions, and evaluation results; versioned provenance enables defensible debugging and rollback; verify the repair using reproducibility of a sampled historical answer using its full manifest.
Incorrect Answers
A:Source code alone rarely communicates the operational decisions and permission steps operators need; provide system overview, access setup, troubleshooting flowcharts, monitoring thresholds, and named support responsibilities; that control is assessed by successful completion of standard support drills by staff who did not build the system; the incident instead requires reproducibility of a sampled historical answer using its full manifest.
B:A feature list does not reveal crossing of sensitive trust boundaries; a current diagram should show data classes, flows, processing boundaries, identities, trust zones, and side effects; that control is assessed by completeness of sensitive-data flow paths identified during security review; the incident instead requires reproducibility of a sampled historical answer using its full manifest.
D:A high-level diagram alone is not an executable incident response procedure; maintain runbooks for detection, containment, safe fallback, rollback, and communication with named incident roles; that control is assessed by successful containment time in a simulated stale-index incident; the incident instead requires reproducibility of a sampled historical answer using its full manifest.