AI-200: Securing and Observing AI Services

A retrieval service seems healthy because its home page responds, yet users say that some answers are empty and others arrive after thirty seconds. The logs reveal almost nothing: no clear request identifiers, no explanation of which dependency slowed down, and occasional authentication errors that no one can reproduce. This is both an observability problem and a security problem. AI-200’s backend-development syllabus explicitly includes Azure Key Vault, Azure App Configuration, OpenTelemetry and Kusto Query Language (KQL) because reliable AI systems must protect their dependencies and make failures diagnosable.

These subjects are easy to treat as miscellaneous tools at the end of a study plan. In practice, they connect every other AI-200 topic: containers need configuration, database connections need credentials, message handlers need correlation and operations teams need evidence of where time and failures accumulate. AI-200 differs from some other Microsoft certifications by focusing on the components surrounding a model: secret retrieval, configuration state, distributed request traces and log analysis. Those are the controls that let an engineer explain why an AI application misbehaved and restore it without guesswork. Secure configuration and meaningful telemetry must support every request, from admission to final result.

Separate secrets from ordinary application configuration

A database endpoint, a feature flag and a credential are all configuration inputs, but they have different sensitivity and lifecycle requirements. A feature flag may be safely visible to authorized operators; a signing key or API token must be protected. Azure App Configuration can organize application settings and supported feature-management patterns. Azure Key Vault is designed for managed protection of secrets and other supported sensitive material. The two can work together without being treated as interchangeable stores.

Suppose a production service needs to change the address of a vector index. That endpoint is an environment setting. Rotating a key used to authenticate to the index requires a separate process that protects the secret, updates authorized consumers and avoids accidental exposure. Baking either value into a container image creates unnecessary coupling between deployment and runtime environment. Baking a secret into source control additionally creates an incident that cannot be solved merely by deleting one Git commit.

The application should retrieve only the settings it needs. A document-processing worker does not require the administrator credentials for every other service in the subscription. Isolation reduces the impact of one compromised workload and makes permission reviews more understandable.

Key Vault is useful only when access is actually controlled

Key Vault can store supported secrets and enables applications to retrieve them through authorized mechanisms. Access must be configured using an appropriate identity and the selected permission model. A common design gives a compatible Azure workload its own managed identity, then grants that identity the narrow Key Vault access it needs. This avoids distributing reusable long-lived credentials through deployment scripts. The authorization boundary must still be checked: possessing a valid workload identity is not equivalent to having permission to read every stored secret.

Rotation requires operational planning. A service may cache a secret for performance, and the old credential may remain in memory after the value in Key Vault changes. A safe rotation procedure accounts for overlapping validity where supported, refresh behavior, fallback or rollback conditions, and the effect of failed retrieval. If a vault is temporarily unreachable, the team should know which operations will continue, which will fail and how that appears in telemetry. A blanket “retry forever” rule can make an outage longer and obscure the original cause.

The principle of least-privilege role assignment applies to secret access just as it does to ordinary Azure resources. Treat secret-read, secret-write and resource-administration duties as separate wherever practical. Logging should record relevant access outcomes without dumping secret values into diagnostic events.

Configuration changes need version awareness and release discipline

Feature settings and endpoint configuration can change more often than code. That agility is useful, but it creates another source of release risk. Imagine that one replica uses a new embedding model while another replica still queries an index produced by the old model. Requests may appear randomly wrong because configuration is inconsistent, even though both instances run the same container image. Versioning model identifiers, index names, schema expectations and related configuration helps expose such mismatches.

Use environment-scoped configuration so development experiments do not silently affect production. Establish an owner and review process for settings that change data formats, logging volume, security posture or request limits. Feature flags should have a retirement plan; an old flag that remains forever can create contradictory execution paths that are hard to test. Changes affecting the retrieval model or prompt construction need quality evaluation in addition to deployment smoke tests.

A coherent deployment record links image version, configuration revision, model or embedding version and relevant database schema state. If an incident begins immediately after a setting change, operators should be able to reconstruct that combination. The ability to identify “what changed” is often more valuable than adding another dashboard after the outage has started.

OpenTelemetry gives a vocabulary for request journeys

OpenTelemetry provides SDKs and semantic conventions for collecting observability signals such as traces, metrics and logs. A distributed trace can show a request entering an API, making a database query, calling a model service and waiting for a queue or downstream handler. Spans describe individual timed operations within that trace. Correlation allows engineers to ask which specific operation consumed the latency budget, rather than observing that the whole request took thirty seconds.

Good instrumentation uses stable and meaningful operation names. A span called database.query or embedding.request is more actionable than dozens of unstructured “started” and “finished” log lines. Attributes can describe safe dimensions such as dependency type, deployment revision or outcome category. Avoid putting raw prompts, personal information or credentials into tracing attributes by default. High-cardinality values can also raise telemetry cost and impair aggregation, so choose carefully what becomes a metric label rather than a trace detail.

Instrumentation should distinguish an upstream timeout from a business rejection. An unauthorized document may be correctly excluded; that is not automatically the same failure as a lost database connection. Operations teams need separate counters and tracing context for expected policy outcomes, dependency errors and user-facing defects.

KQL turns logs into diagnostic evidence

Kusto Query Language is used with relevant Azure log and data-analysis experiences to filter, join, summarize and investigate event records. Its exact table schema depends on where telemetry is stored and how it is collected, so a copied query is not universally portable. The useful skill is to ask a precise operational question and translate it into filtering and grouping logic. For example: did latency increase after a deployment, and was the increase concentrated in vector queries or model calls?

Start by selecting an appropriate time window and correlating request identifiers or traces. Summarize failures by dependency and deployment version, examine latency distributions rather than only averages, and compare behavior before and after a change. High error counts during an expected maintenance window need different interpretation from a sudden sustained rise in authorization failures. A service with low average latency may still have severe tail delays affecting a small but important set of requests.

The AI-300 qualification explores GenAIOps and broader quality and model operations. AI-200’s use of KQL and OpenTelemetry has a narrower application-development emphasis: can the engineer diagnose how the Azure backend behaves and which component is responsible for an observed failure? That requires disciplined instrumentation more than a large collection of unrelated metrics.

Investigate with a hypothesis, not a restart button

Suppose customers see intermittent empty responses. First identify whether the application returned an intentional “no relevant sources” result, whether retrieval produced candidates and filtering removed all of them, or whether a dependency call failed. Trace the specific request using its correlation identifier. Compare model input preparation with database response timing, examine identity failures, and check the deployment and configuration versions. A generic HTTP 200 response from the front end cannot tell you whether a retrieval pipeline actually fulfilled its business purpose.

Another incident shows p95 latency jumping after a release. The model service is unchanged, but database spans now dominate traces. The likely investigative branch concerns new query patterns, index configuration, connection pooling or higher result volume. Increasing web replicas before inspecting those spans could increase database pressure. If instead Key Vault calls suddenly fail, examine authorization changes, network reachability and secret rotation rather than tuning database throughput.

Meaningful incident analysis also guards privacy. Teams may want to log every prompt and retrieved document because it seems convenient for debugging. That practice can expose sensitive source material to operators or retention systems that were never designed to hold it. Favor redacted identifiers, controlled sampling and access restrictions where possible, with deliberate approval for exceptional diagnostic capture.

Make cost and security part of the service-level objective

Telemetry has a price. Excessive span volume, repeated large log payloads and unbounded retention can turn observability into a significant cost center. The right answer is not to disable logging wholesale but to decide which signals are needed for incident response, operational performance and audit requirements. Aggregate metrics can show service health cheaply; detailed traces can provide high-value context through appropriate sampling strategies. Production incidents sometimes require temporarily richer collection, with an expiration plan for that extra visibility.

Security obligations also extend beyond secrets. Application identities need lifecycle ownership; network paths to data services should reflect the intended trust boundaries; dependencies must handle certificate or token failures in predictable ways. Azure Policy and resource access controls provide governance layers, while application validation protects the content being processed. The identity-focused material of SC-300 goes deeper than AI-200, yet the backend developer must be able to use secure identity patterns correctly in the application workflow.

A useful service-level objective might specify successful authorized document retrieval within a latency threshold under representative load, with clear handling for missing source material and temporary dependency failure. That measures a user-facing outcome rather than merely whether pods are running.

Run a failure drill with evidence you can trust

Deploy a small retrieval API with separate development and production configuration, identity-backed access to sensitive settings, and trace instrumentation for database and model operations. Record the deployment version and correlation identifiers. Then introduce three controlled faults: reject access to one secret, slow a database query and publish an incompatible configuration setting. For each fault, confirm that the incident appears as a distinct symptom, that no secret is exposed in logs and that the tracing evidence leads to the right dependency.

Finish by rolling back the unsafe setting and documenting what the next operator needs to know. If the trace cannot distinguish configuration, identity and performance failures, the observability design is incomplete. If rotating a credential requires rebuilding the application image, the configuration design is fragile. The purpose of AI-200’s security and monitoring objectives is to ensure an engineer can operate AI backend services safely under change, not simply recognize the names of four Microsoft products.