Microsoft AI-103: CI/CD and GenAIOps for Azure AI

AI-103 explicitly expects candidates to integrate Foundry projects with continuous integration and continuous deployment pipelines and to operationalize generative AI systems through evaluation, observability, security, and controlled change. That makes deployment a first-class exam topic. The development process sits within the broader Microsoft certifications, but AI-103 is concerned with how an AI application moves safely from an experiment to a repeatable production system.

The key difference from a conventional application is that AI behavior can change even when ordinary application code does not. Prompts, model deployments, retrieval indexes, evaluation data, agent definitions, tool schemas, safety settings, and model versions can all alter output. A useful CI/CD design therefore versions more than source code and tests more than whether the application starts.

Define what belongs under version control

Start by identifying every artifact that can change behavior. Application code is obvious. Prompt templates, system instructions, tool schemas, agent configuration, model parameters, retrieval configuration, index mappings, safety thresholds, and infrastructure definitions can be just as important. If these items live only in a portal or in a developer notebook, the team cannot reliably reproduce a release.

Not every operational value should be committed as plain text. Secrets, credentials, and environment-specific endpoints belong in secure configuration stores or deployment variables. The repository should contain the definition of the configuration, while the deployment process supplies sensitive values through managed mechanisms.

Versioning is useful only when a deployed behavior can be mapped back to a specific set of artifacts. Release metadata should make it possible to answer which prompt, model deployment, index version, and tool contract produced a problematic result.

Use CI for validation, not just packaging

Traditional CI checks syntax, unit tests, and build integrity. AI CI should add behavior-focused tests. The existing Azure DevOps tooling discussion is useful background for pipeline concepts, but an AI pipeline needs additional gates such as prompt tests, schema validation, retrieval checks, safety evaluations, and representative end-to-end cases.

Deterministic parts should still have deterministic tests. Tool functions can be unit tested. JSON schemas can be validated. Access-control policies can be inspected. Search ingestion can be checked for missing fields. These tests isolate ordinary engineering failures before the model is involved.

Then add evaluation for nondeterministic behavior. Instead of expecting an exact sentence, measure whether the response is grounded, relevant, safe, and complete enough for the task. The acceptance rule should be defined before the release so the team does not rationalize a regression after seeing the result.

Promote changes through environments deliberately

Development, test, staging, and production environments may use different quotas, data sets, identities, or model deployments. The pipeline should make those differences explicit. A prompt that works against a tiny development index may fail in production because retrieval behavior changes with volume. A tool may work with an administrator identity in development and fail correctly under least privilege in production.

Infrastructure as code helps make environment differences reviewable. Foundry projects and supporting Azure resources should be created or configured through repeatable deployment where practical. Even when a service still requires some portal configuration, document the remaining manual step so it does not become invisible operational knowledge.

Promotion should preserve evidence. Store evaluation results, package or artifact versions, and deployment metadata so that an approved staging build is the same build promoted to production rather than a fresh reconstruction.

Make evaluation a release gate

AI evaluation belongs inside the release process because quality is part of functionality. A new model or prompt can improve average fluency while making grounding worse. A retrieval change can increase recall but introduce irrelevant evidence. A safety update can reduce harmful output while blocking legitimate requests. Release decisions need a balanced set of metrics.

Evaluation sets should reflect the real distribution of tasks, including difficult and adversarial cases. Separate offline benchmark tests from production monitoring. Offline evaluation protects the release boundary; production evaluation detects behavior that the test set did not anticipate.

Avoid one global score that hides trade-offs. Track dimensions such as task success, relevance, grounding, safety, latency, and cost separately. The acceptable threshold may differ by use case and by risk.

Deploy with rollback and blast radius in mind

AI releases can fail subtly, so deployment strategy matters. A staged rollout, canary group, or limited traffic slice can reveal problems before the entire user base is affected. The team should be able to roll back the application and its behavioral configuration together. Rolling code back while leaving a changed prompt or index in place may not restore the previous behavior.

Model upgrades deserve similar care. A newer model is not automatically a drop-in replacement. Tokenization, tool-calling behavior, latency, response style, and safety behavior may change. Run the same evaluation set against both versions and compare operational metrics before switching production traffic.

Retrieval changes also need rollback thinking. If an indexing pipeline corrupts metadata or chunk boundaries, restoring application code alone is insufficient. Keep index versions or a repeatable rebuild path that can return the knowledge layer to a known state.

Build observability into the deployment

The current AI-103 objectives call for tracing, token analytics, safety signals, latency breakdowns, model performance, grounding quality, ingestion quality, and index health. These are not optional dashboard decorations. They are how the team distinguishes a model issue from a retrieval issue, a tool issue, or an infrastructure issue.

Distributed tracing is particularly useful in agentic workflows because one user request may involve several model calls, retrieval operations, tool invocations, and approval steps. A trace should connect those operations without exposing unnecessary sensitive content. Operational metrics should show both technical health and behavior quality.

Define alerts around symptoms that require action. A latency increase, tool-error spike, grounding-quality drop, or sudden safety-event change may each need a different owner and runbook.

Treat security configuration as deployable policy

Managed identity, private networking, keyless credentials, role assignments, tool permissions, and approval policies are part of the release. If developers manually grant broad access to make a deployment work, the pipeline is not production ready. Security should be represented in repeatable configuration and verified in the target environment.

Secrets must not leak into prompts, logs, test fixtures, or evaluation exports. The deployment pipeline should use protected variables or identity-based access, and log scrubbing should be tested just like application behavior. A trace is useful only if it does not become a new path for credential exposure.

For agents, tool permissions deserve a separate review. A small change to a tool schema can expand what the model is able to do. Treat those changes with the same seriousness as changing an API permission or database role.

Think in terms of a GenAIOps lifecycle

A mature operating loop is continuous: plan, build, evaluate, deploy, observe, learn, and revise. Production examples feed the evaluation set, evaluation results shape the next release, and traces help identify whether failures come from prompts, models, retrieval, tools, or user inputs. CI/CD is the mechanism that keeps that loop controlled.

Exam scenarios often reveal the right answer through missing lifecycle pieces. If a team cannot reproduce a prompt change, version control is missing. If a new model is pushed directly to production, evaluation and staged rollout are missing. If users report bad answers but the team cannot explain the chain of calls, tracing is missing. If a rollback cannot restore the old index, artifact management is incomplete.

When studying, draw the release path for one agentic application and label every artifact that can change behavior. Then add the test, approval, deployment, monitoring, and rollback step for each artifact. That exercise turns CI/CD from a generic DevOps topic into the AI-specific operating model the exam expects.

Separate application releases from knowledge and model releases

AI systems often have several release cadences at once. Application code may ship weekly, retrieval content may refresh hourly, prompts may change after evaluation, and model deployments may be upgraded on a separate schedule. Treating all of those changes as one undifferentiated release creates poor rollback options and makes incident analysis harder.

A stronger design gives each change type an identity and dependency. An application release can declare which prompt set, model deployment, and retrieval schema it expects. A knowledge refresh can record the source snapshot and indexing configuration. A model migration can be evaluated against the currently approved prompt and retrieval stack. This makes comparisons meaningful because only one dimension changes at a time when possible.

For exam questions, this idea helps identify the real missing control. If content is stale, rebuilding application code is irrelevant. If a model change causes tool-call regressions, re-indexing documents will not help. Operational maturity means knowing which artifact owns the behavior and which release path can restore it.

Test retrieval and tool contracts alongside model behavior

Agentic applications depend on components outside the model, so the release pipeline should validate those interfaces as well. A tool schema that changes a parameter name can break a previously reliable agent. A search index that drops metadata can reduce grounding quality without changing a single line of application code. Contract tests should verify tool inputs and outputs, retrieval filters, expected fields, and representative authorization behavior before promotion.

This layered testing also improves rollback decisions. If the model evaluation is stable but tool failures spike, reverting the model is unlikely to help. If retrieval relevance drops after an indexing change, redeploying the application package is the wrong response. GenAIOps becomes practical when each dependency has a measurable contract and a release owner.