What is data hygiene, and why does it matter for AI

Data hygiene means your data is accurate, consistent, and documented. Most data stacks manage one of the three. The definition is simple, and the distance between that definition and what is actually running in production is where most AI projects quietly fail.

What data hygiene actually means

Skip the consulting version. The practical one has three parts.

Accurate means the values in your tables reflect reality. If a campaign ran from January to March, the date range in the table says January to March. If a customer is tagged "Enterprise," they meet whatever criteria your company uses for Enterprise, and those criteria are the current ones.

Consistent means the same thing is named the same way everywhere. "Facebook Newsfeed" is not "fb_newsfeed" in one table and "Facebook Feed" in another, and the logic that turns a raw value into a business label runs the same way in every model that touches it.

Documented means a new engineer can look at a transformation and tell what it does, why it does it that way, and when it last changed, from the code and its description rather than by asking someone who may have left.

Most stacks are accurate in some places, inconsistent in others, and documented nowhere, and that combination compounds. Each problem makes the other two harder to find.

Where it breaks down: ingestion

Hygiene fails first at the point where data enters the stack. Raw sources arrive with whatever naming, formats, and assumptions the upstream system used. Sometimes that is a third-party API with a clean schema. Sometimes it is a form someone built in 2018 that nobody has touched since, or a CRM where reps type freetext into fields that were supposed to be dropdowns.

The ingestion layer is supposed to normalize this: load the raw data, land it in staging, start the cleaning. In practice the cleaning stops early. The data is technically loaded but still carries the source's inconsistencies, and those flow downstream into every model that reads from staging.

The usual suspects are date fields arriving as strings in three formats, nulls that mean three different things depending on which system produced them, product codes with trailing spaces that miss the lookup table, and customer IDs formatted one way in the CRM export and another way in billing. Each one is small on its own and none of them throws an error.

Where it breaks down: transformation

The transformation layer is where hygiene problems settle in and stay. This is the dbt models, the CASE statements that turn raw campaign codes into business labels, the joins that assemble the tables the business actually reads. It is also where logic accumulates, drifts, and goes undocumented, because it is where the most people have touched the most code over the longest time.

Three patterns show up in nearly every stack I have worked in.

Stale mappings. A CASE statement was written eighteen months ago to classify placement types. Three new placement codes have shipped since. Nobody updated the statement, so every new code falls into the ELSE branch and gets labeled "Other." I have watched "Other" quietly become the third-largest placement type in a client's reporting before anyone asked why. The pipeline ran fine the whole time.

Duplicate logic. The same classification exists in two models, implemented slightly differently. One was written by the engineer who built the original pipeline. The other was written six months later by someone who did not know the first one existed. Both are in production and they sometimes disagree, which means two dashboards can show two answers to the same question and both are "correct."

Undocumented exceptions. Someone added a WHERE clause to exclude a specific account after an executive flagged it as a test account. Three years later the clause is still there. Nobody knows whether it is still a test account, and nobody knows why that report runs consistently low.

None of this shows up as a failure. The pipeline runs, the tables populate, and the data looks fine until someone asks a question precise enough to expose it.

Where it breaks down: documentation

Documentation is the most neglected of the three because most teams treat it as something you write after building. So it either does not exist, or it describes what the model did when it was first written rather than what it does today.

Institutional knowledge ends up in Slack threads and in the heads of specific engineers. When one of them leaves, or is simply out of the room during the meeting where a number gets challenged, the gap becomes a business problem in real time.

It also makes the other two problems harder to fix. If you do not know what a transformation is supposed to do, you cannot tell whether it is doing it. If you cannot read the intent behind a CASE statement, you cannot update it without risking a regression, so it does not get updated, and the stale mapping problem gets worse.

Why it matters more when AI is in the loop

Traditional BI has a specific failure mode. A dashboard shows a wrong number, a user notices, a ticket gets filed, engineering investigates, and eventually the logic gets fixed. The loop is slow, but it exists, and it exists because a wrong number on a chart looks like a wrong number.

AI systems fail differently. They do not show you a wrong number. They answer a question in a paragraph using whatever data they can reach, with the same confidence regardless of what that data looks like. If the mapping says "Facebook Newsfeed" in some places and "fb_newsfeed" in others, the model picks one and keeps going. If your ELSE branch is catching 15% of placements, the model's answer about placement performance will be wrong in a way that does not look wrong. It will look like an answer.

With a dashboard, trust erodes slowly. Users start double-checking, add caveats in meetings, and eventually stop relying on it. With an AI system, someone asks which campaigns drove the most conversions, gets a clean answer that is off because three codes fell into "Other," and acts on it. They might notice when the number fails to reconcile with something else. They might never notice.

Bad hygiene has always been a tax on reporting. With AI in the loop, it is a faster-acting one, and it is harder to trace back to the source.

What fixing it actually looks like

The standard advice is to audit everything, document everything, and clean up the transformation layer. That advice is correct, and almost nobody follows it because it competes with shipping features. The practical version is smaller and you can start it this week.

Start with the models the AI actually reads. Not the whole DAG. Pull the list of tables your chatbot or semantic layer is pointed at and work only on those.

Find where the ELSE branches are catching real values. One query per CASE statement: group by the raw code where the output label is "Other" and count. Anything with meaningful volume is an active accuracy problem. Fix those first.

Add a test so it cannot happen silently again. In dbt, an accepted_values test on the raw code column, or a simple custom test that fails when "Other" exceeds a threshold, turns the next stale mapping into a red build instead of a wrong answer six months later.

Document in the same PR. Not a documentation sprint. One sentence in the model's YAML saying what the CASE statement classifies and what the ELSE branch is supposed to catch, written while the code is open in front of you.

Take consistency problems upstream. Conflicting values across tables usually mean a conversation with whoever owns the source systems. That conversation is harder than a PR, but it is worth having before an AI is answering questions from two versions of the same field.

The goal is not a perfect stack. That does not exist. The goal is a stack where the problems are known, the active ones are fixed, and the logic is documented well enough that someone can maintain it without relying on institutional memory.

That is what hygiene means in practice. Not pristine, but maintained.