Generative AI for Enterprise Data Engineering

About this book

Most books about generative AI for data engineering begin with the technology: models, prompts, agents, vector stores and tools. This book begins with the business decision that has to be right.

A service planner needs to know whether a spare part can genuinely be promised by Friday. A finance process has to be modernised without silently changing the business rules hidden inside a legacy ETL job. A warehouse incident may have a technically correct recovery action that an AI agent still should not be allowed to execute. Only after the decision, consequence and business meaning are clear does the architecture begin.

Every capability, approached twice

First, the greenfield architecture. What should remain deterministic? Where can GenAI help? What evidence may the model use? Which semantic rules and data contracts must hold? What authority can an agent receive?

Then, production pressure. A plausible schema mapping is wrong. Generated SQL runs but changes what a metric means. Events arrive twice or out of order. Lineage is stale. An agent finds the right repair but tries to widen its approved scope. The question changes from “How should we build this?” to “Does it still hold when something difficult happens?”

This greenfield-plus-failure narrative runs through the continuing Orion Global Industries case across the full lifecycle: document and multimodal ingestion, schema reasoning, APIs and streaming, governed SQL, transformation engineering, legacy modernisation, semantic layers and agentic data contracts, knowledge graphs, data quality, GenAI evaluation, cost engineering, DataOps, root-cause analysis, supervised remediation, agent runtimes, multi-agent systems, identity, sensitive-data protection, prompt-injection defence, model and vendor risk, and enterprise adoption.

The final part brings everything together through two industry business cases (financial-services data modernisation and healthcare document-to-data processing) and an enterprise data-platform transformation capstone.

An optional companion pack adds practitioner exercises, evidence files, mappings, contracts, test cases, traces, decision scenarios and worked solutions.