dbt is the right call when more than one person will read or edit the SQL, when tests matter, and when the models need to be a shared thing the team can rely on. Hand-rolled SQL is fine when one person owns the whole stack, the scope is small, and the schedule is simple. The failure mode is not picking the wrong tool. It is picking the right tool for the team you have and outgrowing it silently.

What "write my own SQL pipelines" actually means

Hand-rolled SQL pipelines are usually SQL files run on a schedule. The schedule is cron, or Airflow, or a Python script triggered by a job runner. Sometimes there is a light framework wrapper around it. Sometimes it is a folder of .sql files and a bash script.

dbt is a specific tool that compiles SQL against a warehouse and provides testing, documentation, lineage, and dependency management. It is opinionated about the shape of the work. Models are SQL files, references between them use a ref() function, tests are declared in YAML, deployments run through a CLI.

Both build tables in a warehouse. The differences are entirely in what you get around the SQL.

What dbt gives you, in practitioner terms

Cutting through the marketing, here is the working list.

A dependency graph, inferred automatically from the ref() function. If model B reads from model A, dbt runs A first. If you rename a column in A, the lineage tells you every model downstream that touches it.

Tests, defined in YAML alongside the models. not_null, unique, relationships, accepted_values, and any custom test you write. They run in the same command that runs the models, and they fail loudly.

Documentation, generated from the same YAML. A model gets a description, columns get descriptions, and dbt builds a browsable site out of it. This is the part most teams underuse.

Jinja templating, which lets you write macros that reduce SQL duplication. Useful once. Overused often.

Environment separation. Dev, staging, prod schemas, with the same code running against each. Materializations that handle the annoying incremental logic for you, so you do not have to write another insert-into-where-not-exists by hand.

None of this is magic. Every one of these is something you would eventually build yourself if you had the time. dbt is the thing you would have built.

What hand-rolled pipelines give you

Full control and zero framework overhead. If you know exactly what you want the SQL to do and the surrounding orchestration to look like, hand-rolled is faster to ship on day one.

You skip the learning curve. You skip the opinions. You skip the version compatibility questions. You write the SQL, you write the runner, you write the tests if you write them at all, and every piece does exactly what you told it to.

The tradeoff is that everything dbt gives you for free, you build. Dependency tracking, testing framework, documentation, environment separation, incremental logic. Each one is a script. Each script is fine until it is not, and the not-fine moment always arrives later than you expect.

When hand-rolled is the right call

There are real cases where dbt is the wrong tool. The most common ones.

One person, one warehouse, a small stack. You are the only one reading the code, the schedule is a daily cron, and the models are simple enough that you can hold the graph in your head. dbt would add process without adding safety. Skip it.

Non-warehouse pipelines. dbt is a warehouse tool. If your data lives in Kafka, Spark, or somewhere the SQL is not being run against a warehouse table, dbt is the wrong shape. Use the tool your compute layer expects.

Custom orchestration. If your pipelines have complex triggers, streaming windows, or dependencies on external events, an orchestrator like Dagster or Prefect will give you more than dbt does. dbt runs SQL on a schedule. It does not do arbitrary Python.

Prototypes. If the schema is still moving weekly and you are not sure the pipeline will exist in three months, setting up dbt is overhead you will pay before you know if it was worth paying.

When dbt is the right call

Most of the time, at most stacks, once the situation is no longer one of the four above.

Two or more people will read or edit the models. The moment the SQL becomes shared, the framework matters more than the syntax. ref() prevents the accidental rewrite where two people implement the same join two ways, and the testing surface prevents one person from breaking another's downstream model silently.

The data feeds anything that requires trust. A dashboard, an AI, a report that goes to a board. Trust needs tests, and dbt gives you tests without a project.

You want the stack to survive turnover. dbt is standard enough that a new hire has probably used it. A homegrown pipeline is not, and the person who wrote it is the documentation.

The stack is going to grow. dbt gets stronger as the stack grows. More tests, more docs, more lineage, more reasons the framework earns its keep. Homegrown gets weaker, because every new model is a new opportunity for the informal patterns to drift.

What happens when hand-rolled outgrows the person who built it

This is the failure mode nobody plans for and almost every one of these projects hits.

The person who built the stack is competent. The stack works. They understand every piece of it because they wrote every piece of it. Testing is in their head. Deployment is a runbook they never wrote down. Debugging is muscle memory nobody transferred.

Then something happens. They take a vacation. They get pulled onto another project. They leave. And the next person on the team cannot ship a change to the pipeline without a walkthrough that never happens, because the person who could give the walkthrough is not there.

Meanwhile the stack is now doing something the business depends on. A dashboard reads from it. Finance reads from it. Someone stood up a chatbot on top of it last quarter. Every one of those downstream consumers assumes the pipeline is a stable input, and every one of them will notice quickly if it stops being one.

The team's time starts going into keeping the stack running instead of moving it forward. Changes get slower. Bugs get older. New questions from the business get answered from the version of the data that predates whatever went wrong two months ago.

The migration to dbt is not intrinsically hard. What is hard is that you cannot both keep the stack running and rewrite it with the same person. This is where a scoped outside engagement usually shows up. Someone whose job is the migration, not the day-to-day operation, doing it against a stack that looks like yours because they have done it before against stacks that looked like the last one.

A decision framework

If the question is really "what should we do," here is the table I use when a stack audit lands on this exact fork in the road.

The two rows that catch most teams by surprise are the third and the fifth. The third is the moment where staying on hand-rolled is a decision, not a default, and most teams miss it because the pipeline is still working. The fifth is the moment where the AI or dashboard project that just launched changes the tolerance for wrong data, and the hand-rolled stack was never built for that tolerance.

The straight answer

Adopt dbt if you are past the prototype phase, more than one person will touch the code, or anything downstream matters enough to need tests. Stay hand-rolled if you are one person on a small stack that is not going to grow and the schedule is simple. If you are on hand-rolled and the situation changed a while ago without anyone noticing, the migration is the fix, and it is a scoped project rather than a rewrite that has to happen alongside daily operations.

The tool matters less than whether the person who understands it is still around.

If your homegrown pipeline has outgrown the person who built it, and the migration to dbt has to happen without pausing the stack, that is the scoped shape of the warehouse and dbt build engagement. See the build engagement →

A three-day question is not three days of work. It is fifteen minutes of work spread across three days, distributed across four people, each of whom has to reconstruct the same context from scratch because there is no shared source they can all point at. The question feels simple to you because it is. The answer takes three days because the plumbing behind it does not exist.

What actually happens between the email and the answer

Here is the sequence I have watched more times than I can count on the agency side.

The account lead reads the email. They do not have the number. They ping the paid social lead, the search lead, the programmatic lead, the influencer lead, one at a time or in parallel depending on how the account is run. Each of those people has a different tool. Each of those tools shows something slightly different from the others. Each of them pulls their piece into an Excel workbook, their workbook, formatted the way they format things, and sends it back.

Someone, usually the account lead or an analyst, has to combine the workbooks by hand. Date ranges do not line up. Campaign names do not line up. One person included agency fees in spend, another did not. Reconciling the mismatches takes an hour. Writing up the answer takes twenty minutes. Reviewing it before it goes out takes another cycle.

By the time the response arrives in your inbox, four people spent thirty to sixty minutes each. The wall clock says three days because the work sat in a queue between every step.

Why the simple questions are the expensive ones

Large projects have process. A campaign launch, a quarterly deck, a media plan, all of these go through a workflow. There are templates, there are review steps, there are people whose job includes them.

Simple questions skip the process. They arrive as email, get treated as one-offs, and every one of them runs the fire drill from scratch. Nobody counts them because counting them would require someone to add up how many hours the team spent answering ad-hoc client questions this month, and nobody at the agency wants to know the answer.

On my last agency team we ran that math for one account. It was around eighty hours a month. The retainer had no line item for ad-hoc analysis. All of it lived inside "account management" and "reporting," both of which were technically for something else. The client did not know it was happening. The agency did not want to.

What no shared source looks like in practice

A CPG brand I worked on had a monthly review meeting where the client would ask, casually, midway through, so what was our total spend on the launch push in July. The question was not on the agenda. Two channel leads on the agency side would open two dashboards and quietly disagree. One number would include the influencer spend booked through the talent agency. The other would not. The account lead would pick whichever one sounded closer to what the client seemed to be remembering, and the meeting would move on.

Nobody was lying. Nobody had done a bad job. There was no shared definition of the launch push, no shared definition of spend, and no way to reconcile them in the moment. The answer that made it into the meeting notes was the answer whoever had the loudest number in the room happened to have.

The client was on the receiving end of that same pattern every time they asked a number question, and they did not know it. They assumed the answer they got was the answer.

That is where three days comes from. Not from anyone being slow. From every question having to be reconstructed, because there is no layer where the definitions live.

Why the usual fix does not work

The instinct when someone points out the three-day problem is that the account needs a dashboard. Sometimes the account lead has already built one. Sometimes the client has asked for one. Sometimes there are three of them, each showing slightly different numbers.

A dashboard shortens the loop for questions that fit inside it. That is real value. But dashboards do not shorten the loop for new questions, and dashboards built on top of the same drifted definitions that the channel owners are already using just move the reconciliation problem into a visualization. Now the client is looking at a wrong number faster than before.

The other common fix is more analysts. Adding people to the fire drill makes the drill run faster. It does not stop the drill from happening. The unit economics of ad-hoc analysis do not improve with headcount. They improve with the drill going away.

The drill goes away when there is one place to get the number and one definition of what the number is.

What actually shortens the loop

The fix is not glamorous and it does not ship in a week. But every step of it collapses a different piece of the three-day cycle, and the compound effect is that account leads answer client questions the same morning instead of three days later.

One place the numbers live. Not the dashboard. The mart the dashboard reads from. If the channel owners are still going to their own tools to answer questions, you have not consolidated. The account lead should be able to point at one source and say, that is where every number comes from.

One person owns each definition. Not the team, not whoever built the last dashboard. The person whose job includes deciding whether influencer fees count as spend, and updating the definition when that decision changes. If the answer to who defines spend is a role, not a name, you have your first gap.

Delete duplicate sources before you add new ones. Every extra copy of the same metric is a future disagreement. The old dashboard the channel owner still uses out of habit, the export that gets emailed monthly, the tab in the account lead's tracker. All of them are places the answer can diverge from the source. If they are not the source, they need to go.

The account lead has to be able to self-serve. If answering the question still requires channel owners to open Excel and pull numbers, the loop is not shortened. The account lead needs a view that already contains the answer for the top eighty percent of questions. Which questions those are is knowable, because they are the same questions every month.

Log what is getting asked. Not a survey. Actually log the ad-hoc requests for one month. What was asked, how long the answer took, whether the same question has come back. The list is shorter than the agency thinks it is, and most of it can be pre-answered in the shared view once it is written down.

The three-day question stops being a three-day question the day the numbers stop having three versions. Everything else, the dashboard, the review meeting, the reporting cadence, works the same. The only difference is that nobody has to reconstruct the answer from scratch every time.

If the account team is spending eighty hours a month reconstructing answers, the warehouse and dbt build engagement is what makes that reconstruction unnecessary. One source, one definition, same-morning answers. See the build engagement →

Retrieval is a distribution mechanism. It moves whatever is in the underlying corpus to the model, faster and more precisely. If the corpus disagrees with itself, retrieval distributes the disagreement. The fixes that get most of the attention, better embeddings, smarter models, longer context, all improve the reasoning over what was retrieved. None of them repair a source that says two different things.

What retrieval actually does

Retrieval takes a question, converts it to an embedding, and finds the chunks or tables whose embeddings sit closest to it. The top few get handed to the model as context. The model reasons over that context and returns an answer.

It has no notion of truth. It has a notion of proximity. If two chunks about the same topic contradict each other, retrieval will return both, or one, depending on how the question is phrased. The model then synthesizes an answer from whichever it got, with the same confidence either way.

For a data-facing AI the corpus is your warehouse. The chunks are marts, tables, semantic definitions, whatever the retrieval layer is pointed at. This is the semantic layer problem at a different altitude. If revenue is defined three ways across three marts, retrieval will return one of them for any given query. The model will build a paragraph on top of it. Nobody will flag the ambiguity because nobody sees the other two definitions.

Why teams debug the retrieval instead of the corpus

Retrieval is the visible layer. The corpus feels done. It was populated months or years ago, and nobody wants to open it back up. That is the same reason bad hygiene persists in warehouses generally: the transformation layer is where the problems settled in, and reopening it is a project.

There is also an incentive gap. Chunk size and embedding models are knobs. Knobs feel like progress. Deleting a mart, or rewriting a definition, is a conversation with the person who owns it, which is a project. The knob is faster to turn, so the knob gets turned first.

The last piece is that the AI ecosystem is loud about retrieval improvements and quiet about corpus hygiene. Every week there is a new reranker, a new embedding model, a new retrieval technique. Almost none of the content is about the boring, unshippable work of making the source consistent. So teams keep swapping components in a layer that was not broken.

What a dirty corpus looks like in practice

An engagement earlier this year. The client had a warehouse chatbot pointed at their marketing marts. Numbers were wrong often enough that the marketing team stopped using it. When we looked, three marts each had a version of a spend metric. One rolled up at platform grain. One at campaign grain with a filter that excluded internal test spend. One had a WHERE clause added six months ago to patch a specific report and never removed. All three were reachable by the retrieval layer. Depending on the exact phrasing of the question, it returned a different one.

The team had been tuning the reranker for a month. The fix was deleting two of the three marts and rewiring the retrieval layer to point at one mart, one definition, one number. The reranker did not need to be smarter. The corpus needed to stop contradicting itself.

None of the retrieval tuning would have fixed this. A perfect reranker returning one of three wrong numbers is still returning a wrong number. Two of the three had to go.

Why a bigger model does not fix it

The obvious next move when retrieval gives inconsistent answers is to reach for a smarter component. A larger context window, so the model sees more chunks. A stronger model, so it reasons better. A reranker, so the top chunks are more relevant.

All three help with reasoning over what was retrieved. None of them repair the underlying inconsistency. A smarter model given three contradictory chunks does not tell you the corpus is broken. It picks one, or averages, or hedges, and returns a confident paragraph. The failure is not intelligence. It is that the model is doing exactly what it was asked to do on inputs that disagree.

The trap is that each upgrade produces a small improvement, which reads as progress, which delays the corpus conversation. The corpus conversation is the one that actually fixes it, and it does not ship in a weekend.

What to do this week

The check is smaller than the tuning work you were about to do. It takes a few hours.

Pull the top twenty queries the assistant has received. You should have logs. If you do not, that is the first thing to fix. For each query, note whether a knowledgeable human would give a different answer depending on which document, mart, or table they reached first.

Grep the corpus for the concept in each query. How many places is "revenue" defined? How many marts have a version of "active user"? If the answer is more than one and the definitions do not match, retrieval was never the problem.

Delete or archive anything outdated, draft, or superseded. Old marts that got replaced but never removed. Draft docs that got scraped. Test tables. Every one of them is a source retrieval will happily reach for. If nobody has referenced it in six months, it should not be in the corpus.

Point retrieval at one canonical version per concept. For docs, one source of record with everything else linking to it. For data, the semantic layer or a small set of documented marts. If you cannot name the canonical version, retrieval cannot find it either.

Log what retrieval actually returned for the wrong answers. Not just the query and the response. The chunks or tables that got pulled. When the top three come from sources that disagree with each other, that is the bug. It is easier to see when you can look at it.

Only then start tuning. A cleaner corpus makes every retrieval improvement land harder. A dirtier corpus absorbs them. Doing the cleanup first is the difference between a reranker that helps and a reranker that gives you faster wrong answers.

Retrieval was fine. The corpus was the problem. That is the version of this story that shows up in almost every failed rollout, and it is the one nobody debugs first.

If your AI is answering questions from a warehouse that quietly says three things at once, a stack audit finds the disagreements before you spend another sprint tuning retrieval. See the stack audit →

Between the transformation layer that produces your tables and the reporting layer that displays them, there is supposed to be a place where the definitions live. One place. Active user gets defined once, every downstream tool inherits it, and nobody argues about which number is right because there is only one number to have. In most stacks that place does not exist, and no one has been given the job of making it exist.

What the semantic layer actually is

The semantic layer sits between dbt and everything that reads from it. Its job is to hold the metric definitions and the dimensions they can be sliced by, and to hand out a canonical query when anything downstream, a dashboard, a notebook, an AI assistant, asks for a number.

Concretely: revenue is defined once as sum of order_total where status is confirmed and refunded_at is null. Active user is defined once. So is churn, MRR, gross margin, whatever your business runs on. The definition includes which dimensions are valid to group by and which joins are allowed. Downstream tools do not write their own SQL against the raw tables. They query the definition.

LookML was an early version of this idea inside Looker. dbt has semantic models and metrics now. Cube, MetricFlow, and a few others sell it as a product. What they have in common is that the metric is expressed once, in code, in a place a person can read and edit, and every consumer of that metric goes through the same door.

That is what the layer is. What it is not: a data catalog, another dashboard, or a documentation site. Those describe metrics. A semantic layer computes them.

Why nobody owns it

The layer falls into a gap between three teams and rarely lands on any of them.

Data engineering owns ingestion and raw storage. Their answer is that shaping business logic is not their job. Analytics engineering owns the transformation layer. Their answer is that the metric lives in the BI tool, downstream of the models they build. The BI or reporting team owns the dashboards. Their answer is that the metric is upstream, in the SQL that populates the visualization.

Between the three, the metric definition ends up copy-pasted into individual dashboard queries. Same field, five slightly different implementations, no one point of truth. When someone asks who owns the definition of revenue, everyone points at someone else.

This is not a hiring problem. Adding a headcount to any of the three teams does not fix it, because the layer is not naturally in any of their scopes. Someone has to be given the job explicitly, with authority to say no to a dashboard that computes its own version of a shared metric. That role does not exist at most companies.

What "no semantic layer" looks like in practice

On the agency side I ran into this weekly. Three account leads on the same brand, three versions of "media spend" for the same month. One included agency fees, one did not. One was in reported currency, one was normalized to USD. All three were labeled "media spend" in their respective dashboards, and all three were correct against whatever query produced them. What was missing was a definition of media spend that any of them could point to and say, that is the number.

The fire drill fix is to reconcile the three dashboards, pick a winner, and align the others to it. That works for one meeting. It does not survive the next metric someone adds, because the process that produced the drift is still the process the team uses. New dashboard, new SQL, new definition of whatever field it needs, slightly different from the last one.

The real fix is the layer. Move the definition of media spend into a semantic model, point all three dashboards at it, and the next dashboard that gets built inherits the same definition. The drift stops at the point of creation instead of getting rediscovered in a meeting six months later.

Why AI makes this worse

A dashboard shows one number at a time. When two dashboards disagree, a person notices, chases the discrepancy, escalates. The disagreement is visible and slow, and the human loop eventually catches it.

An AI answering questions across the warehouse does not work like that. It picks a table, runs a query, returns an answer in a paragraph. If two tables have revenue defined differently, the model does not flag the ambiguity. It picks the one it found first, and the answer arrives as one clean sentence that reads like it was pulled from a single trusted source.

I covered a version of this in the hygiene post: AI systems absorb inconsistency instead of surfacing it. The semantic layer version is the same failure at a different altitude. The dashboard problem was three visible numbers in three tabs. The AI problem is one confident paragraph with no way for the reader to know that three definitions were in play.

The teams that put an AI in front of a warehouse without a semantic layer typically get a good demo, a rough first month, and a slow decline in trust as reconciliation problems accumulate. Not a crash. A drift.

What to do this week

You do not have to buy a product to start. The first version can be a documented mart and a rule about who is allowed to change it.

List the top ten metrics your business actually asks about. Revenue, active users, churn, whatever it is. If you cannot get to ten, list five. This is the surface area that matters.

For each one, name the owner. Not the team. The person. If the answer is "the dashboard" or "whoever built it," that is your first gap.

Pick one metric and grep for every place it is computed. Every dashboard query, every notebook, every dbt model that produces a version of it. Read them side by side. The differences are where the disagreement lives.

Write the canonical version once. In a dbt semantic model, a metrics YAML, or a plain documented mart with a fixed definition. Include which dimensions are valid to slice by. This is the definition of record from now on.

Point one downstream tool at it. Not all of them at once. One dashboard, migrated to the canonical version, with the old query deleted. Prove the pattern works before you scale it.

Publish the rule. When someone spins up a new dashboard, the answer to "where does revenue come from" is a link to the semantic model, not "ask the person who built the last one." Without the rule, the layer erodes as fast as you build it.

You do not need to buy a semantic layer product. You need someone whose job is to own the definitions. The product is the easy part.

If your metrics live inside dashboard queries instead of one shared place, the warehouse and dbt build engagement is what makes that layer exist. One definition, one owner, one number. See the build engagement →

That is not an AI strategy. It is an intern with no context and unlimited confidence, and it produces the same three problems every time: hallucinated answers, silent shortcuts, and eventually, wrong data written back into the stack. The fix is not to stop using Claude. The fix is to know your stack well enough to decide exactly where it belongs.

What "using Claude for data" usually means

The phrase covers at least four different activities, and teams rarely separate them.

Asking questions of an export. Someone drops a spreadsheet into a chat and asks for insights. The model has no idea what the columns mean, which rows are test accounts, or that the revenue field was redefined in Q2. It answers anyway. I covered why in the chatbot post: the model is correct about whatever it was given, and what it was given is a file with no lineage.

Writing SQL from a description. An analyst describes what they want and the model produces a query. This works surprisingly well when the model can see the schema and the definitions. It works badly when it cannot, because the model will guess a join key that looks plausible, pick the first table with a matching column name, and return a number that reconciles with nothing.

Cleaning or transforming data. The model is asked to dedupe, fill nulls, standardize labels, or reshape a dataset. This is where shortcuts happen. Dropping rows with nulls is a shortcut. Coalescing two date formats by assuming one of them is always US-style is a shortcut. Each one is a reasonable move for a script and a wrong move for your business, and the output looks clean.

Writing back to the stack. Someone takes the cleaned output, or the generated transformation, and loads it. Now the shortcut is in the warehouse, downstream models read from it, and the next person to ask a question is querying a table whose logic nobody wrote down. This is how a chat session becomes a data quality incident.

The first activity produces wrong answers. The last one produces wrong data. The difference matters, because wrong answers get corrected in a meeting and wrong data gets discovered in a quarterly review.

Know the stack before you introduce the model

You cannot decide where Claude belongs until you can draw the stack it is being introduced to. Most teams cannot draw it. They can name the tools, but not who owns each layer, what flows between them, or which tables are actually trusted.

Here is the version I build in the first week of an audit. It is a table, not a diagram, because a table forces you to fill in the blanks.

The two columns that matter are "Trusted?" and "Claude's role." If you cannot honestly fill in the first, you have no business filling in the second. A model pointed at a layer nobody trusts does not become a source of truth. It becomes a faster way to distribute the distrust.

Alongside the table, you need the data side of the same inventory. Which fields drive the questions people actually ask. Where each field is produced. Whether the definition in the YAML matches the SQL. Whether there are tests. The hygiene post walks through that check. It takes an afternoon and most teams have never done it.

Introduce Claude to specific parts, with a specific job

Once the map exists, the placement decision gets easy, because the good jobs share a shape. They are narrow, the model can be given the full context it needs, and there is a human review step between the model's output and anything that persists.

Documentation in the transformation layer. Point the model at a dbt model and its upstream sources and ask it to draft the YAML description and column-level docs. An engineer reviews and commits. This is the highest-value, lowest-risk job in the stack, and it directly fixes the documentation gap that makes every other problem worse.

Tests. Given a model and its description, the model can propose accepted_values, not_null, unique, and relationship tests, and it is good at spotting the columns that should have one and do not. Same review gate: proposed tests go in a PR.

Corrections as pull requests. An analyst describes a mapping fix in plain English. The model translates it into a change to the seed file or the CASE statement and opens a PR. The engineer reviews a diff instead of a ticket. I built this for a regional bank whose marketing team was spending half a day a week reconciling partner naming by hand. Correction time went from two and a half days to thirty minutes, and the reason it worked is that the model never touched the warehouse. It touched a branch.

SQL against a defined layer. Natural language querying is fine when the model is reading the semantic layer or a set of documented marts, with the definitions in its context, and the query is logged. It is not fine against raw tables, and it is not fine when "the definitions" are in someone's head.

Explaining what a model does. Hand it a 400-line model that the original author no longer works here to explain. Ask what it does and where the logic is fragile. This is read-only, cheap, and useful the day a number gets challenged in a meeting.

The pattern across all five: the model reads freely, proposes in a branch, and never writes to production without a person approving the diff. That one rule eliminates the "making wrong data" problem entirely. The others get smaller as the documentation and tests the model helped write start to exist.

What to stop doing

Stop pasting exports into a chat and calling the output insight. Stop letting generated transformation code land anywhere without a review. Stop pointing a chatbot at a layer you would not trust a new analyst to query unsupervised.

None of that is anti-Claude. I use it daily and it is the reason a one-person shop can take on a warehouse rebuild. But it is a tool that does exactly what the context tells it to, and if the context is a CSV with no lineage, that is the quality of the work you will get back.

Map the stack. Fix what you find. Then give the model a specific job in a specific layer, with a person between it and production. That is what "using Claude for data" should mean.

If drawing that stack table feels like more work than you have time for, that map is what week one of a stack audit produces. See the stack audit →

What data hygiene actually means

Skip the consulting version. The practical one has three parts.

Accurate means the values in your tables reflect reality. If a campaign ran from January to March, the date range in the table says January to March. If a customer is tagged "Enterprise," they meet whatever criteria your company uses for Enterprise, and those criteria are the current ones.

Consistent means the same thing is named the same way everywhere. "Facebook Newsfeed" is not "fb_newsfeed" in one table and "Facebook Feed" in another, and the logic that turns a raw value into a business label runs the same way in every model that touches it.

Documented means a new engineer can look at a transformation and tell what it does, why it does it that way, and when it last changed, from the code and its description rather than by asking someone who may have left.

Most stacks are accurate in some places, inconsistent in others, and documented nowhere, and that combination compounds. Each problem makes the other two harder to find.

Where it breaks down: ingestion

Hygiene fails first at the point where data enters the stack. Raw sources arrive with whatever naming, formats, and assumptions the upstream system used. Sometimes that is a third-party API with a clean schema. Sometimes it is a form someone built in 2018 that nobody has touched since, or a CRM where reps type freetext into fields that were supposed to be dropdowns.

The ingestion layer is supposed to normalize this: load the raw data, land it in staging, start the cleaning. In practice the cleaning stops early. The data is technically loaded but still carries the source's inconsistencies, and those flow downstream into every model that reads from staging.

The usual suspects are date fields arriving as strings in three formats, nulls that mean three different things depending on which system produced them, product codes with trailing spaces that miss the lookup table, and customer IDs formatted one way in the CRM export and another way in billing. Each one is small on its own and none of them throws an error.

Where it breaks down: transformation

The transformation layer is where hygiene problems settle in and stay. This is the dbt models, the CASE statements that turn raw campaign codes into business labels, the joins that assemble the tables the business actually reads. It is also where logic accumulates, drifts, and goes undocumented, because it is where the most people have touched the most code over the longest time.

Three patterns show up in nearly every stack I have worked in.

Stale mappings. A CASE statement was written eighteen months ago to classify placement types. Three new placement codes have shipped since. Nobody updated the statement, so every new code falls into the ELSE branch and gets labeled "Other." I have watched "Other" quietly become the third-largest placement type in a client's reporting before anyone asked why. The pipeline ran fine the whole time.

Duplicate logic. The same classification exists in two models, implemented slightly differently. One was written by the engineer who built the original pipeline. The other was written six months later by someone who did not know the first one existed. Both are in production and they sometimes disagree, which means two dashboards can show two answers to the same question and both are "correct."

Undocumented exceptions. Someone added a WHERE clause to exclude a specific account after an executive flagged it as a test account. Three years later the clause is still there. Nobody knows whether it is still a test account, and nobody knows why that report runs consistently low.

None of this shows up as a failure. The pipeline runs, the tables populate, and the data looks fine until someone asks a question precise enough to expose it.

Where it breaks down: documentation

Documentation is the most neglected of the three because most teams treat it as something you write after building. So it either does not exist, or it describes what the model did when it was first written rather than what it does today.

Institutional knowledge ends up in Slack threads and in the heads of specific engineers. When one of them leaves, or is simply out of the room during the meeting where a number gets challenged, the gap becomes a business problem in real time.

It also makes the other two problems harder to fix. If you do not know what a transformation is supposed to do, you cannot tell whether it is doing it. If you cannot read the intent behind a CASE statement, you cannot update it without risking a regression, so it does not get updated, and the stale mapping problem gets worse.

Why it matters more when AI is in the loop

Traditional BI has a specific failure mode. A dashboard shows a wrong number, a user notices, a ticket gets filed, engineering investigates, and eventually the logic gets fixed. The loop is slow, but it exists, and it exists because a wrong number on a chart looks like a wrong number.

AI systems fail differently. They do not show you a wrong number. They answer a question in a paragraph using whatever data they can reach, with the same confidence regardless of what that data looks like. If the mapping says "Facebook Newsfeed" in some places and "fb_newsfeed" in others, the model picks one and keeps going. If your ELSE branch is catching 15% of placements, the model's answer about placement performance will be wrong in a way that does not look wrong. It will look like an answer.

With a dashboard, trust erodes slowly. Users start double-checking, add caveats in meetings, and eventually stop relying on it. With an AI system, someone asks which campaigns drove the most conversions, gets a clean answer that is off because three codes fell into "Other," and acts on it. They might notice when the number fails to reconcile with something else. They might never notice.

Bad hygiene has always been a tax on reporting. With AI in the loop, it is a faster-acting one, and it is harder to trace back to the source.

What fixing it actually looks like

The standard advice is to audit everything, document everything, and clean up the transformation layer. That advice is correct, and almost nobody follows it because it competes with shipping features. The practical version is smaller and you can start it this week.

Start with the models the AI actually reads. Not the whole DAG. Pull the list of tables your chatbot or semantic layer is pointed at and work only on those.

Find where the ELSE branches are catching real values. One query per CASE statement: group by the raw code where the output label is "Other" and count. Anything with meaningful volume is an active accuracy problem. Fix those first.

Add a test so it cannot happen silently again. In dbt, an accepted_values test on the raw code column, or a simple custom test that fails when "Other" exceeds a threshold, turns the next stale mapping into a red build instead of a wrong answer six months later.

Document in the same PR. Not a documentation sprint. One sentence in the model's YAML saying what the CASE statement classifies and what the ELSE branch is supposed to catch, written while the code is open in front of you.

Take consistency problems upstream. Conflicting values across tables usually mean a conversation with whoever owns the source systems. That conversation is harder than a PR, but it is worth having before an AI is answering questions from two versions of the same field.

The goal is not a perfect stack. That does not exist. The goal is a stack where the problems are known, the active ones are fixed, and the logic is documented well enough that someone can maintain it without relying on institutional memory.

That is what hygiene means in practice. Not pristine, but maintained.

If hygiene is the gap between your data and the AI you want to run on top of it, an audit is the fastest way to see how big that gap is. See the stack audit →

What the transformation layer actually does

The transformation layer sits between raw sources and the clean tables your teams use. Engineers write SQL models that rename fields, apply business logic, map codes to labels, and join tables into something a human can reason about. In most modern stacks that means dbt, and it is where data quality either gets built in or gets skipped.

It is not a query tool and it does not answer questions. It builds the foundation everything else sits on, and when the foundation has errors, every dashboard and every report downstream inherits them. Most business users have no idea this layer exists. They just know the numbers are sometimes wrong and that fixing them takes a long time.

Two use cases that get conflated

Natural language tooling does not replace the transformation layer. It changes who can get answers from it and who can flag problems with it. Those are two different jobs and vendors tend to blur them.

The first is querying. Someone asks "what were our top campaigns last quarter" in plain English and gets a number back. This is conversational access to data that already exists and is already correct. It is the easier of the two problems and it is the one every demo shows.

The second is correction. An analyst notices that a field is mapped wrong, a label is stale, or a calculation does not match what finance is reporting. Today that means filing a ticket, explaining the logic to an engineer who was not in the room when the logic was written, waiting for a PR, and hoping the fix ships before quarter close. This is the harder problem and the one almost nobody is solving well.

Who this is actually for

If you are a data engineer writing SQL every day, natural language querying is not for you. You already have direct access and the tooling around dbt is already your workflow. Nobody should be trying to sell you a chat box.

The people who benefit are the ones locked out of the process. I spent years on the agency side running pipelines for CPG and financial services clients, and the same person showed up at every one of them: the analyst who knew the campaign type field had been wrong for six months, could describe exactly what the fix was, and had no path to make it happen except pinging me. I had a backlog. Their fix waited. The dashboard stayed wrong.

That analyst is not asking to replace the transformation layer. They are asking to participate in it without becoming an engineer.

The real bottleneck

Data teams are not slow because engineers are slow. They are slow because every request, from "what did we spend in January" to "the region mapping is out of date," flows through the same channel and competes for the same person's attention.

Natural language does not remove the engineer from that loop. It changes where they enter it. Instead of triaging every question and every correction, they review the ones that need review. The mechanics look something like this: a business user flags that "Paid Social" should include the new TikTok source, the tool drafts the change to the mapping seed or the model's case statement, and the engineer sees a proposed diff instead of a Slack message that starts with "hey, quick question." The engineer still approves it. They just spend five minutes instead of an afternoon.

That is the actual value proposition. Not "no more SQL." SQL for the people who need to write it, and plain language for everyone else.

What this requires from your stack

None of this works if the transformation layer is a mess. Natural language on top of inconsistent field names, stale mappings, and undocumented logic does not produce better answers. It produces confident wrong answers faster, and now they arrive in a friendly chat window that makes them look more trustworthy than the spreadsheet did.

Before you put a natural language tool in front of anyone, three things need to be true. Your models reflect current business logic, not the logic from the last reorg. Your field names mean what they say. Your transformation rules are documented somewhere other than the head of the engineer who wrote them two years ago, which in dbt means the YAML descriptions are filled in and someone has read them recently.

That is the pre-work most teams skip, and it is the reason most natural language tools underdeliver in production. The tool gets blamed for a foundation problem.

You still need SQL. You still need the transformation layer. What you do not need is every question and every correction waiting on the same engineer every time.

Rebuilding the transformation layer so non-engineers can participate without becoming engineers is what the warehouse and dbt build engagement is for. See the build engagement →

What hallucination actually is

"Hallucination" is a bad word for this. It implies the model is inventing facts from nothing, and in a company data context that is rarely what happens.

A language model does not check facts. It predicts. Given the inputs it has, it generates the most likely output and delivers it with the same confidence every time. If one table labels a category "Electronics / Consumer" and another labels it "Consumer Electronics," the model picks one and answers. It does not flag the discrepancy or hedge. It has no way of knowing there is a discrepancy to flag.

So the reasoning is usually fine. The problem is what the reasoning is working from.

Where the bad data lives

Almost no chatbot is reading raw files. It reads the transformed layer, the cleaned and modeled tables your data team built on top of the raw sources. That layer is where the trouble is, because that layer is where every undocumented decision from the last five years went to live.

I have inherited a few of these. At one agency, we had a client whose channel mapping table was last touched during a rebrand. Two years later, half the campaign names in the raw feed no longer matched anything in the mapping, so they fell into a bucket called "Other." The dashboard had quietly been reporting "Other" as the third-largest channel for months. Nobody caught it because nobody looked at "Other." The AI, when we pointed one at that warehouse, reported it too, in a nice paragraph, with total confidence.

That is the typical shape of the problem. Transformation logic two engineers interpreted differently and neither documented. A revenue field recalculated every quarter with a definition that drifted between implementation and today. A join that silently drops rows. None of it looks like an AI problem, so nobody debugs it as one.

When a VP asks for last quarter's top campaigns and gets a number that is off by 30%, the instinct is to blame the prompt. Rewrite the system message, swap the model, add more context. None of that repairs a mapping table.

Why teams debug the wrong thing

Prompt engineering is real, and a well-structured prompt produces better output. But it has a ceiling. If the underlying field is wrong, the model will answer accurately about wrong data every single time, and a better prompt gets you a more articulate version of the same wrong answer.

This is uncomfortable to say to a team that just spent a quarter on the AI layer: the model is not confused. It is correct about something incorrect. That is a different problem from a bad prompt, and it needs a different fix.

The teams I have seen get reliable results are not the ones with the cleverest prompts. They treated the data layer as the product before the AI project existed. Documented definitions, consistent naming, tests on the models that feed anything customer-facing. It is not the work that gets demoed at a board meeting, but it is the work that determines whether the demo holds up in week three.

What to check before you file another ticket

Here is a check you can run in an afternoon.

Pull the ten fields the chatbot touches most. Look at the query logs or the semantic layer config. It will be a shorter list than you expect.

Trace each one to the model that produces it. In dbt, that means finding the model and the upstream sources. If you cannot find it in under ten minutes, that is finding number one.

Ask someone to define the field without opening the code. If the definition they give does not match what the SQL does, you have found where the AI is getting its wrong answer. Write the real definition down in the model's YAML before you touch anything else.

Check when the mapping tables were last updated. Compare that date to the last rebrand, platform migration, or reorg. If the mapping predates the change, assume it is wrong until proven otherwise.

Look at whatever bucket is called "Other," "Unknown," or "Uncategorized." Its size tells you how much of your data the model is confidently reporting from a category that means nothing.

If any of those come back bad, that is where the problem lives. Switching models will not help. A retrieval layer on top of dirty data just gives you faster access to wrong answers.

Fix the data first. The chatbot was fine.

If your team is chasing prompt fixes when the numbers are wrong upstream, a stack audit points at the real problem in one week. See the stack audit →