ISTQB CT-AI: Testing Systems That Learn

A conventional application normally implements behavior deliberately specified by its developers. A machine-learning system can also change its behavior when training data, model versions or the operating environment change. This makes testing more complex than checking whether a button produces the expected screen. ISTQB Certified Tester AI Testing (CT-AI) addresses the distinctive problems of testing AI-based systems: uncertain outputs, data quality, probabilistic performance, hidden bias and the challenge of recognizing failures when no simple expected answer exists.

As of October 2026, ISTQB identifies CT-AI syllabus version 2.0 as the newer qualification material. Version 1.0 exams, including retakes, remain scheduled only until April 21, 2027 for English and October 21, 2027 for other languages; candidates must choose the syllabus matching their examination provider and booking. Candidates must check which syllabus their chosen examination provider is actually offering; mixing objectives from two versions can misdirect preparation. ISTQB requires Foundation Level certification as a prerequisite for the specialist route. The exam's purpose is not to transform a tester into a research scientist, but to make testing decisions more rigorous when software incorporates learning models and AI services.

Understand why model behavior demands different evidence

An AI component may classify an image, rank a transaction or recommend an action based on learned relationships rather than a written decision table. That means a correct result on one input says less about future behavior than it might for a deterministic algorithm. Distribution changes, unusual combinations of features and training artifacts can cause unexpected errors. The tester needs to understand the application context: which decisions are automated, who relies on them, what harms result from mistakes and whether the system is permitted to abstain or escalate to a human.

Consider a model that flags potentially fraudulent purchases. Accuracy alone can hide a severe weakness if fraud is rare: a classifier that approves everything might still appear impressive on a highly imbalanced test set. Precision, recall, false positives, false negatives and business cost must be interpreted together. A bank may tolerate some extra investigation to avoid severe loss, while an emergency medical service may make a different judgment about delay and missed detection. Good test objectives specify the error types, operational thresholds and the populations on which performance matters.

Examine the data pipeline before testing predictions

Training, validation and test datasets play different roles. Training data shapes the learned model; validation data guides development choices; a held-out test set should provide a less biased assessment of performance. Leakage occurs when information that would not be known at prediction time enters the development process, making results look stronger than production reality. Sampling bias appears when the collected examples do not represent intended users. Data labeling errors, duplicated records and mismatched transformations can change outcomes before anyone reaches the model itself.

A retail demand model might appear reliable because training and evaluation data both contain patterns from the same holiday period. Test its predictions in other seasons and regions, including new stores and products with limited history. Check data completeness, formats, allowed values and the timing of recorded features. Preserve dataset versions so failed tests can be reproduced. When sensitive personal information is involved, apply privacy controls to collection, storage and test environments. CT-AI preparation asks candidates to treat data as a source of defects with traceable origins, not as an unquestioned input supplied by someone else.

Choose metrics that correspond to the real decision

Different model tasks demand different measurements. Classification may require a confusion matrix and class-specific precision or recall. Regression may need absolute and squared-error measures, with careful attention to outliers. Ranking systems may need measures that capture whether relevant results appear early. A global aggregate can hide failure among small groups, so metrics should be broken down by relevant segments while respecting sample sizes and privacy constraints. Establish acceptable ranges before examining the final held-out results, not afterward to justify a favorable score.

Suppose a speech-recognition feature works well in quiet offices but fails in noisy warehouses. The relevant test report should separate conditions rather than average away the problem. Build test datasets around expected accents, devices, noise levels and speech styles, then look for unacceptable disparities. An apparently small aggregate error can still break a safety-critical command path. Operational monitoring needs metrics too, because offline performance degrades when inputs change. A tester should explain how to recognize drift and when to trigger retraining, rollback or human review—not simply present a model score.

Test where the expected answer is uncertain

For many AI outputs, a perfect oracle is unavailable: reviewers disagree on the correct label or a recommendation has more than one defensible answer. Metamorphic testing examines relationships that should hold across controlled input changes. If a photo is rotated within expected operating limits, an object classifier might be expected to preserve its category. If an irrelevant field in a loan application changes, the decision should not change solely because of that field. Property-based and statistical testing can provide evidence even without a single golden output.

Metamorphic relationships must be chosen carefully. Rotating a handwritten character may change its meaning; changing an address field might legitimately affect geographic service availability. Define the business property first and challenge its assumptions. Combine test techniques: curated edge cases, boundary examples, adversarial or noisy inputs, comparison with a trusted baseline, expert review and repeat runs where nondeterminism matters. Document expected variation and acceptance criteria. A useful CT-AI test case explains why a transformation should be invariant or change in a predictable way, rather than just generating more samples without an oracle strategy.

Investigate bias, fairness and explainability

Fairness is not a property that can be established with one generic checklist. Relevant groups, protected attributes, proxy features and acceptable treatment depend on the application, jurisdiction and affected people. Compare error rates across meaningful cohorts with enough data to support conclusions, and examine whether historical decisions encoded in training data reproduce previous discrimination. Some fairness objectives can conflict, requiring governance decisions rather than an engineer silently optimizing a single number. Transparency and explainability also have limits: a plausible explanation is not proof that it faithfully represents the model's decision.

Imagine an automated candidate-screening tool. Historical hiring outcomes may favor applicants from certain educational backgrounds because of earlier selection practices. The tester should question whether those labels provide a legitimate ground truth. Evaluate sensitivity to irrelevant demographic proxies, document disparities, and escalate findings to accountable policy and legal owners. Where a model produces explanations, test their stability and usefulness for actual appeals or human review. AI governance and security controls are adjacent concerns, but CT-AI is specifically concerned with evidence for the system's behavior and its effect on users.

Validate AI components inside a larger product

Even a strong model can fail when integrated into an unreliable product. Test interface contracts, missing feature values, latency, timeouts, authentication, caching, model-selection logic and fallback behavior. Observe how confidence estimates are handled: a classification probability is not automatically calibrated, and a low-confidence result may need review rather than a forced answer. Model versioning, feature extraction and deployment pipelines should support reproducibility. Changes to prompts or surrounding components can alter behavior even when the underlying model weights are unchanged.

A traffic-management system might use an AI prediction to adjust signal timings. If the prediction service times out, the wider application needs safe default behavior. Test stale recommendations, inconsistent data versions, delayed telemetry and rollback to a previously accepted model. Measure error handling under realistic load, not just accuracy on an offline benchmark. Security testing also matters: inputs might be manipulated to change predictions or expose sensitive training information. The tester must understand the boundary between the model, its inputs and the downstream software that turns a prediction into action.

Keep AI evaluation alive after deployment

AI quality is not a one-time certification of a model artifact. Data drift occurs when input patterns change; concept drift occurs when the relationship between features and outcomes changes. Monitor prediction distributions, downstream outcomes, feedback delays and error patterns where ground truth becomes available. Set thresholds for investigation and document who can approve updates. A model should have an identifiable owner, version history, rollback mechanism and plan for collecting meaningful production evidence without breaching privacy or security requirements.

For example, a fraud model trained before a new payment method launches may start rejecting legitimate customers. Comparing outcomes by payment method can reveal the issue faster than an overall dashboard. Yet a sudden increase in alerts might indicate genuine attacker behavior, not merely drift, so the investigation needs fraud operations and business context. Use shadow deployments or canary releases where appropriate, with clear stop conditions. ISTQB candidates should be comfortable articulating why a monitoring signal matters, how to distinguish model degradation from environment failure and what evidence supports intervention.

Build a practical CT-AI study program

Begin by reviewing the syllabus version applicable to your booking and comparing its learning objectives with the foundational knowledge assessed by ISTQB Foundation Level. Work through a small classification exercise with a confusion matrix, class imbalance and segmented performance. Create a synthetic dataset with deliberate leakage and explain why the first attractive score is misleading. Design metamorphic tests for a text classifier or recommendation service, stating the invariant properties and the exceptions. These exercises develop judgment about testing strategy, not just familiarity with AI terminology.

Then design an end-to-end assurance plan for a system used by real people: what its decision affects, how labels are obtained, which error types matter, how deployment is monitored and how incidents are escalated. Compare that focus with AI-assisted testing, which is a distinct ISTQB specialist subject concerned with the tester's use of generative tools. CT-AI asks how to test an AI-enabled product; CT-GenAI asks how testers can use AI responsibly in their work. Knowing that difference prevents a surprisingly common category error in exam preparation.