Data ingestion looks simple until a real system has to survive retries, schema changes, late records, duplicate events, source outages, and different latency requirements. DP-700 tests whether you can choose and operate loading patterns in Microsoft Fabric rather than merely name the available tools. The strongest design starts with the behavior of the source and the contract the downstream analytics solution needs.<\/p>\n
Fabric gives data engineers several ways to bring data into OneLake and related analytical stores: pipelines, Dataflows Gen2, notebooks, shortcuts, mirroring, Eventstream, Spark, SQL, and KQL all appear in the broader exam landscape. The difficult part is deciding which path matches the workload without creating unnecessary copies, hidden state, or brittle recovery logic.<\/p>\n
A full load replaces or rebuilds a target from the source data set. It is easy to reason about and useful for small reference data, one-time backfills, or sources that do not expose reliable change information. Its weakness is scale: repeatedly scanning and moving an entire data set wastes time and compute, and a long-running refresh increases the window in which the target is stale.<\/p>\n
Incremental loading narrows the work to new or changed data. That can be based on timestamps, increasing keys, change tracking, change data capture, source partitions, event offsets, or another durable watermark. The important DP-700 idea is that \u201cincremental\u201d is not a feature switch. You must know what identifies change, where progress is recorded, and what happens when the process is restarted.<\/p>\n
The broader data engineering<\/a> role depends on exactly this kind of operational reasoning: a pipeline is valuable only when the data it produces can be trusted after normal failures.<\/p>\n Sources differ in more than file format. A relational application may expose stable tables but restrict heavy extraction during business hours. An API can paginate, throttle, or change payload shape. Files might arrive once per day but be resent with corrections. Streaming systems produce ordered or partially ordered events that require durable offsets and late-event handling.<\/p>\n A useful landing strategy preserves enough source fidelity to recover from downstream mistakes. That might mean retaining raw files, capturing ingestion timestamps, preserving source keys, and separating arrival metadata from business fields. When teams overwrite the only copy of incoming data during transformation, they lose the ability to replay a failed or corrected processing step cleanly.<\/p>\n Landing does not require copying everything forever. Retention should follow business, compliance, and recovery needs. The architectural goal is to make the ingestion boundary explicit so later transformations can be rebuilt without repeatedly hitting operational systems.<\/p>\n Fabric pipelines are strong when orchestration and data movement are central. They can coordinate copy activity, dependencies, parameters, schedules, and downstream processing. Dataflows Gen2 are useful when low-code Power Query transformations are maintainable for the team and workload. Notebooks are better when ingestion or preparation needs code-centric logic, libraries, complex parsing, or Spark-scale processing.<\/p>\n DP-700 scenarios often become easier when you separate the movement question from the transformation question. A pipeline can orchestrate a notebook without forcing every transformation into pipeline activities. A Dataflow can prepare a manageable source while a notebook handles a large engineering workload elsewhere. Tool boundaries should make support simpler, not prove that one service can do everything.<\/p>\nDesign the landing layer around source behavior<\/h2>\n
Choose pipelines, Dataflows, and notebooks for the work they fit<\/h2>\n