Question 380
A research platform executes collaborator-supplied scripts in a constrained runtime. Penetration tests attempt to reach restricted files and network services. Which outcome would establish that the execution sandbox is effective?
- Measured attack success rate on adversarial web pages and zero unauthorized egress calls
- Forbidden file and network access attempts blocked in execution tests
- Denial of unauthorized actions and harmful outputs across multi-stage red-team paths
- Zero payment writes when tested with the analytics identity and forged instructions
Explanation
Correct Answer
B:Sandbox escapes and denial logs are more relevant than the assistant's stated intentions; measure forbidden file and network access attempts blocked in execution tests at the release gate; isolate execution with restricted filesystem, network, credentials, resources, and observable policy denials.
Incorrect Answers
A:Simulated hostile documents test the trust boundary directly; keep source contents in a lower-trust evidence role and prevent embedded commands from crossing into tool authority; this tests attack success rate on adversarial web pages and zero unauthorized egress calls; the actual release condition is forbidden file and network access attempts blocked in execution tests.
C:End-to-end adversarial cases assess the combined defenses and their failure isolation; layer input handling, model instructions, backend authorization, schema checks, and output review so one failed defense does not grant action authority; this tests denial of unauthorized actions and harmful outputs across multi-stage red-team paths; the actual release condition is forbidden file and network access attempts blocked in execution tests.
D:Direct permission tests verify security independent of model obedience; enforce role and scope in the backend execution service using credentials that cannot perform the forbidden action; this tests zero payment writes when tested with the analytics identity and forged instructions; the actual release condition is forbidden file and network access attempts blocked in execution tests.
Question 381
A consumer-facing assistant handles abusive and manipulative prompts in addition to normal support issues. Which strategy should be prioritized for this requirement?
- Test whether the tool layer denies unauthorized requests
- Measure rate and severity of safety policy violations
- Use tail-latency indicators for interactive promises
- Use task-correctness metrics tied to verified labels
- Track cost per verified successful task
Explanation
Correct Answer
B:Define safety incident categories, risk-weighted violation rates, and appropriate exposure denominators for adversarial and real traffic; risk-weighted scoring detects rare critical violations that a single average hides; evaluate the design with severity-weighted policy violation incidence on a reviewed adversarial test set.
Incorrect Answers
A:Treat backend authorization denial rate and cross-scope access tests as primary security metrics, not merely refusal wording; model refusal scores can be high even when the underlying API is vulnerable; this alternative would measure unauthorized API requests rejected under role, tenant, and resource attack matrices; it does not establish severity-weighted policy violation incidence on a reviewed adversarial test set for the described use case.
C:Report p95 or p99 end-to-end latency over representative traffic rather than relying only on averages or raw model runtime; mean latency underweights rare long waits and omits retrieval and external-tool delays; this alternative would measure p95 time from user submission to fully validated answer under load; it does not establish severity-weighted policy violation incidence on a reviewed adversarial test set for the described use case.
D:Measure the business decision against a labeled reference, with metrics weighted for costly errors and explicit acceptance thresholds; positive conversational tone does not establish that a legally or financially meaningful label is correct; this alternative would measure critical-class precision and recall on independently verified tax-code cases; it does not establish severity-weighted policy violation incidence on a reviewed adversarial test set for the described use case.
E:Relate full inference and review costs to accepted outcomes that meet the task rubric, not to requests attempted; cost per token can improve while cost per correct business outcome deteriorates; this alternative would measure fully loaded cost per correctly completed document; it does not establish severity-weighted policy violation incidence on a reviewed adversarial test set for the described use case.
Question 382
A multinational organization wants to deploy employee profiling across several jurisdictions. The launch board requires proof that privacy analysis occurred before live processing. Which governance artifact or measure should it review?
- Documented authorization-boundary mapping for every service handling regulated data
- Deletion-request completion across all inventoried repositories within policy deadlines
- Verified geographic processing paths for all sensitive request components
- Approved contracting, access controls, and PHI data-flow verification for the selected service path
- Completion of required privacy assessment and recorded resolution of high risks before go-live
Explanation
Correct Answer
E:Compliance readiness must be demonstrated upstream of irreversible data processing; measure completion of required privacy assessment and recorded resolution of high risks before go-live at the release gate; involve privacy counsel to assess applicable lawful bases, transparency, rights, and whether a data-protection impact assessment is required before release.
Incorrect Answers
A:Boundary-specific checks prevent assuming certification transfers between unrelated services; verify the specific service, region, impact level, boundary, and agency authorization requirements with security officials before selection; this tests documented authorization-boundary mapping for every service handling regulated data; the actual release condition is completion of required privacy assessment and recorded resolution of high risks before go-live.
B:End-to-end lifecycle tests verify more than the source system's delete operation; map data copies, implement lifecycle deletion where required, and retain audit evidence of the authorized request and its completion; this tests deletion-request completion across all inventoried repositories within policy deadlines; the actual release condition is completion of required privacy assessment and recorded resolution of high risks before go-live.
C:A full data-flow audit finds secondary exports that primary-storage checks miss; trace storage, inference, retrieval, logs, backups, and subprocessors against contractual and legal location requirements; this tests verified geographic processing paths for all sensitive request components; the actual release condition is completion of required privacy assessment and recorded resolution of high risks before go-live.
D:Compliance verification must match the actual product configuration and agreement; confirm the specific service and contract support relevant HIPAA obligations, including any required business associate agreement and safeguards; this tests approved contracting, access controls, and PHI data-flow verification for the selected service path; the actual release condition is completion of required privacy assessment and recorded resolution of high risks before go-live.
Question 383
Staff treat a generated interpretation as an authoritative legal decision for a disputed case. What design correction resolves the reported failure?
- Escalate binding legal interpretations to qualified reviewers
- Give reviewers only necessary sensitive information
- Collect reviewer overrides as structured improvement signals
- Require stronger approval paths for exceptional high-impact actions
Explanation
Correct Answer
A:Define escalation thresholds and route consequential interpretations to qualified legal or compliance owners with source context; a consequence-aware test checks whether unsupported authoritative decisions are avoided; verify the repair using correct escalation of ambiguous legal cases with reviewer attribution.
Incorrect Answers
B:Human oversight is not permission to expand access to unrelated personal information; minimize, redact, and scope information in review interfaces according to purpose and authorized reviewer role; that control is assessed by unauthorized field exposure and successful review accuracy after data minimization; the incident instead requires correct escalation of ambiguous legal cases with reviewer attribution.
C:Counting approval clicks alone does not describe why humans disagreed with the AI; record correction reasons, provenance, and case labels under retention rules to support evaluation and process redesign; that control is assessed by recurrence rate of categorized, previously corrected error types; the incident instead requires correct escalation of ambiguous legal cases with reviewer attribution.
D:A uniform low-friction approval does not reflect differing blast radius and operational risk; apply risk-tiered approvals, including dual control or independent sign-off where policy demands, to exceptionally consequential mutations; that control is assessed by critical-action dual-approval compliance and zero bypasses during staged tests; the incident instead requires correct escalation of ambiguous legal cases with reviewer attribution.
Question 384
A proposed agent rewrite changes how billing corrections would be submitted to production tools. What action best fits the business and technical boundary?
- Predefine success criteria and sample-size assumptions
- Use shadow evaluation before enabling new side effects
- Require an explicit rollback trigger for harmful regressions
- Randomize consistently at the user or account level
Explanation
Correct Answer
B:Replay or mirror representative inputs through a non-mutating shadow path, compare decisions, then gate any live actions separately; non-mutating evidence reduces risk before enabling operational authority; evaluate the design with shadow-policy agreement and zero write side effects across replayed billing cases.
Incorrect Answers
A:Define primary metrics, minimal meaningful effect, sample-size method, and stopping rules before examining results; repeatedly peeking and stopping at a favorable fluctuation inflates false-positive decisions; this alternative would measure pre-registered experiment compliance and interval estimates at the planned sample size; it does not establish shadow-policy agreement and zero write side effects across replayed billing cases for the described use case.
C:Establish automated and human-owned rollback triggers tied to critical error and unauthorized-action indicators; monitoring without an action threshold can document harm while allowing it to continue; this alternative would measure time from triggered guardrail breach to verified rollback; it does not establish shadow-policy agreement and zero write side effects across replayed billing cases for the described use case.
D:Assign a stable randomized unit aligned to the behavior being measured and maintain consistent treatment during the experiment; request-by-request randomization can be invalid when outcomes depend on ongoing user interactions; this alternative would measure balanced treatment cohorts and clean user-level exposure histories; it does not establish shadow-policy agreement and zero write side effects across replayed billing cases for the described use case.
Question 385
A high-volume inference platform samples telemetry to control observability costs. Rare fault scenarios are seeded during a load run. Which test result would demonstrate the sampling policy still captures critical evidence?
- Error diagnosability with seeded sensitive fields absent from collected logs
- Capture rate of seeded critical failures at a bounded telemetry cost
- Stage-specific p95 deltas identifying injected bottlenecks
- Time to detect and stop a simulated unauthorized bulk-write burst
Explanation
Correct Answer
B:Risk-aware sampling preserves evidence for rare faults without uncontrolled storage growth; measure capture rate of seeded critical failures at a bounded telemetry cost at the release gate; retain error and high-risk traces at a higher rate while sampling successful routine traffic with privacy controls.
Incorrect Answers
A:Redaction and observability must be validated together, not traded off blindly; record tool type, status, duration, correlation IDs, and carefully redacted diagnostic fields under controlled retention; this tests error diagnosability with seeded sensitive fields absent from collected logs; the actual release condition is capture rate of seeded critical failures at a bounded telemetry cost.
C:Budget attribution points remediation at the stage that truly regressed; instrument per-stage timing and compare each segment against its allocated latency budget before changing model configuration; this tests stage-specific p95 deltas identifying injected bottlenecks; the actual release condition is capture rate of seeded critical failures at a bounded telemetry cost.
D:Action telemetry and response drills measure whether harmful changes can be contained promptly; monitor write volume, denials, unusual resource patterns, and action-specific thresholds with fast containment routing; this tests time to detect and stop a simulated unauthorized bulk-write burst; the actual release condition is capture rate of seeded critical failures at a bounded telemetry cost.
Question 386
After a connector release, normal transaction volume remains flat but prompts become much larger. Which detection measure would validate a cost-anomaly alert before a billing surprise?
- Safety precision/recall and legitimate-task completion before and after the change
- Detection lag for injected per-request token inflation
- Quality delta between recent audited production samples and the launch baseline
- Detection of version-specific quality regressions in a staged rollout
Explanation
Correct Answer
B:Unit-level telemetry catches runaway expense before invoices provide delayed evidence; measure detection lag for injected per-request token inflation at the release gate; monitor input/output tokens, cache reads, per-stage costs, and cost-per-task anomalies with accountable ownership.
Incorrect Answers
A:Paired monitoring distinguishes beneficial safety controls from excessive refusals; monitor both harmful-output violations and false-positive blocks by version and user segment; this tests safety precision/recall and legitimate-task completion before and after the change; the actual release condition is detection lag for injected per-request token inflation.
C:Live sampling reveals degradation that unchanged historical tests cannot see; periodically sample representative production outputs for labeling, segment analysis, and drift signals without retaining unnecessary personal data; this tests quality delta between recent audited production samples and the launch baseline; the actual release condition is detection lag for injected per-request token inflation.
D:Release segmentation isolates changes before broad impact; partition traces and scored outcomes by deployed configuration, using release identifiers to support rollback and diagnosis; this tests detection of version-specific quality regressions in a staged rollout; the actual release condition is detection lag for injected per-request token inflation.
Question 387
A portfolio assistant requests a report assembled from several external systems and the job can take minutes. Which option addresses the immediate design need?
- Return a clearly marked partial answer rather than miss the deadline
- Move slow operations into asynchronous jobs with progress states
- Parallelize independent read-only tool calls
- Rerank a short candidate set for ambiguous retrieval queries
Explanation
Correct Answer
B:Acknowledge an asynchronous job, persist its state, and let the client poll or receive authorized completion notifications; durable job identity decouples user interaction time from background work duration; evaluate the design with job completion reliability and duplicate-submission rate during slow-system simulations.
Incorrect Answers
C:Execute independent lookups concurrently with bounded timeouts and aggregate the results after all necessary dependencies complete; parallel execution offers no benefit when one call requires another's output or modifies shared state; this alternative would measure end-to-end p95 improvement with all required sources still present; it does not establish job completion reliability and duplicate-submission rate during slow-system simulations for the described use case.
D:Retrieve a bounded candidate set and rerank against the actual question to concentrate useful evidence without feeding all candidates to the model; taking every retrieved hit into context may raise token cost and distraction without raising answer quality; this alternative would measure grounded-answer accuracy and retrieval-stage p95 latency at several rerank depths; it does not establish job completion reliability and duplicate-submission rate during slow-system simulations for the described use case.
A:Keep the critical verified signal independent and return a clearly limited result when optional enrichment exceeds its budget; silently presenting an unverified enriched conclusion would violate accuracy requirements despite meeting a timer; this alternative would measure deadline compliance and correct indication of omitted enrichment; it does not establish job completion reliability and duplicate-submission rate during slow-system simulations for the described use case.
Question 388
Executives cannot connect technical investment to financial impact, service quality, or residual risk. What design correction resolves the reported failure?
- Maintain a risk register with controls and accountable owners
- Present executive decisions through outcomes, cost, and risk
- Keep a versioned decision and change history
- Demonstrate known limitations alongside successful workflows
Explanation
Correct Answer
B:Summarize business value, alternatives, cost envelope, measured evidence, and requested decisions in concise nontechnical terms; decision-focused review checks whether the audience can act on the information; verify the repair using sponsor comprehension of expected outcomes, decision options, and risk ownership.
Incorrect Answers
A:A high-level statement that AI has risks does not create an actionable governance plan; record probability and impact where appropriate, control owners, evidence, residual risk, and required approvals; that control is assessed by completion and ownership of high-severity mitigations before the release gate; the incident instead requires sponsor comprehension of expected outcomes, decision options, and risk ownership.
C:Chat history and scattered slides rarely provide an authoritative current decision state; maintain a version-controlled decision log linked to implementation and review milestones, including superseded guidance; that control is assessed by correct retrieval of active decisions and their supersession chain during review; the incident instead requires sponsor comprehension of expected outcomes, decision options, and risk ownership.
D:A demo focused solely on best cases can mask residual risks and create unrealistic expectations; use representative happy-path and failure-path demonstrations with explicit fallback, uncertainty, and human ownership; that control is assessed by stakeholder identification of boundaries and fallback responsibilities after the demo; the incident instead requires sponsor comprehension of expected outcomes, decision options, and risk ownership.
Question 389
A Claude agent has several similarly named lookup tools and picks the wrong tenant-filtered search. What action best fits the business and technical boundary?
- Write unambiguous tool descriptions and input schemas
- Remove duplicate and deprecated tools from the catalog
- Audit tool manifests as versioned production configuration
- Offer each agent only the tools required for its role
Explanation
Correct Answer
A:Define task-specific descriptions, typed arguments, required fields, and clear side-effect semantics so tool selection is meaningful; confusion tests exercise the ambiguity that caused misrouting; evaluate the design with correct tool-selection rate for ambiguous intents and invalid-argument rejection.
Incorrect Answers
B:Consolidate overlapping operations behind a maintained interface and retire obsolete tool definitions after migration testing; more tool variants do not add genuine capability when they expose the same action with conflicting contracts; this alternative would measure single correct connector selection and consistent ticket states across repeated requests; it does not establish correct tool-selection rate for ambiguous intents and invalid-argument rejection for the described use case.
C:Maintain owner-approved manifests with risk classification, contract checks, permission reviews, and change-control gates; ad hoc tool registration lacks the inventory needed to catch capability expansion before deployment; this alternative would measure changes rejected when requested tool scopes exceed the approved task manifest; it does not establish correct tool-selection rate for ambiguous intents and invalid-argument rejection for the described use case.
D:Give the agent a least-capability tool set mapped to the exact tasks it may perform, reducing accidental and adversarial action paths; monitoring a broad tool catalog does not remove the authority to invoke dangerous operations; this alternative would measure unauthorized tool availability and invocation count in hostile transcript tests; it does not establish correct tool-selection rate for ambiguous intents and invalid-argument rejection for the described use case.
Question 390
Human sign-off remains present on paper, but audits show reviewers never inspect contradictory source details. How should the design be corrected?
- Validate tool-result provenance and semantics
- Counter automation bias with visible uncertainty and review
- Constrain outbound channels to prevent data exfiltration
- Threat-model prompt injection across lower-trust inputs
Explanation
Correct Answer
B:Design review interfaces that expose citations, uncertainty, and targeted challenge checks rather than only an approval button; the review test measures meaningful human oversight, not click-through compliance; verify the repair using reviewer error interception on seeded incorrect recommendations.
Incorrect Answers
A:A successful HTTP status does not mean returned data has decision authority; preserve source identity and trust level, validate schemas, and separate observed facts from authoritative approval signals; that control is assessed by zero external-result fields promoted to internal approvals in test cases; the incident instead requires reviewer error interception on seeded incorrect recommendations.
C:Redaction of the final chat answer does not prevent a tool from transmitting data elsewhere; restrict egress destinations, data classes, and tool arguments at the network and application layers, with audit trails; that control is assessed by blocked unauthorized destinations and absence of confidential fields in egress logs; the incident instead requires reviewer error interception on seeded incorrect recommendations.
D:Stronger wording in the system prompt is not a complete defense against an authorized dangerous tool; model external content as untrusted input, test role spoofing and instruction attempts, and enforce downstream permissions regardless of the prompt; that control is assessed by unauthorized tool-action success rate over injected emails and documents; the incident instead requires reviewer error interception on seeded incorrect recommendations.