Should I use dbt or write my own SQL pipelines?

You are the technical founder, the CTO, or the analytics lead who has to decide whether to use dbt or write your own SQL pipelines. Both work. That is the whole problem. Neither breaks on day one, and by the time you find out which one has broken quietly for six months, you have already committed to it.

dbt is the right call when more than one person will read or edit the SQL, when tests matter, and when the models need to be a shared thing the team can rely on. Hand-rolled SQL is fine when one person owns the whole stack, the scope is small, and the schedule is simple. The failure mode is not picking the wrong tool. It is picking the right tool for the team you have and outgrowing it silently.

What "write my own SQL pipelines" actually means

Hand-rolled SQL pipelines are usually SQL files run on a schedule. The schedule is cron, or Airflow, or a Python script triggered by a job runner. Sometimes there is a light framework wrapper around it. Sometimes it is a folder of .sql files and a bash script.

dbt is a specific tool that compiles SQL against a warehouse and provides testing, documentation, lineage, and dependency management. It is opinionated about the shape of the work. Models are SQL files, references between them use a ref() function, tests are declared in YAML, deployments run through a CLI.

Both build tables in a warehouse. The differences are entirely in what you get around the SQL.

What dbt gives you, in practitioner terms

Cutting through the marketing, here is the working list.

A dependency graph, inferred automatically from the ref() function. If model B reads from model A, dbt runs A first. If you rename a column in A, the lineage tells you every model downstream that touches it.

Tests, defined in YAML alongside the models. not_null, unique, relationships, accepted_values, and any custom test you write. They run in the same command that runs the models, and they fail loudly.

Documentation, generated from the same YAML. A model gets a description, columns get descriptions, and dbt builds a browsable site out of it. This is the part most teams underuse.

Jinja templating, which lets you write macros that reduce SQL duplication. Useful once. Overused often.

Environment separation. Dev, staging, prod schemas, with the same code running against each. Materializations that handle the annoying incremental logic for you, so you do not have to write another insert-into-where-not-exists by hand.

None of this is magic. Every one of these is something you would eventually build yourself if you had the time. dbt is the thing you would have built.

What hand-rolled pipelines give you

Full control and zero framework overhead. If you know exactly what you want the SQL to do and the surrounding orchestration to look like, hand-rolled is faster to ship on day one.

You skip the learning curve. You skip the opinions. You skip the version compatibility questions. You write the SQL, you write the runner, you write the tests if you write them at all, and every piece does exactly what you told it to.

The tradeoff is that everything dbt gives you for free, you build. Dependency tracking, testing framework, documentation, environment separation, incremental logic. Each one is a script. Each script is fine until it is not, and the not-fine moment always arrives later than you expect.

When hand-rolled is the right call

There are real cases where dbt is the wrong tool. The most common ones.

One person, one warehouse, a small stack. You are the only one reading the code, the schedule is a daily cron, and the models are simple enough that you can hold the graph in your head. dbt would add process without adding safety. Skip it.

Non-warehouse pipelines. dbt is a warehouse tool. If your data lives in Kafka, Spark, or somewhere the SQL is not being run against a warehouse table, dbt is the wrong shape. Use the tool your compute layer expects.

Custom orchestration. If your pipelines have complex triggers, streaming windows, or dependencies on external events, an orchestrator like Dagster or Prefect will give you more than dbt does. dbt runs SQL on a schedule. It does not do arbitrary Python.

Prototypes. If the schema is still moving weekly and you are not sure the pipeline will exist in three months, setting up dbt is overhead you will pay before you know if it was worth paying.

When dbt is the right call

Most of the time, at most stacks, once the situation is no longer one of the four above.

Two or more people will read or edit the models. The moment the SQL becomes shared, the framework matters more than the syntax. ref() prevents the accidental rewrite where two people implement the same join two ways, and the testing surface prevents one person from breaking another's downstream model silently.

The data feeds anything that requires trust. A dashboard, an AI, a report that goes to a board. Trust needs tests, and dbt gives you tests without a project.

You want the stack to survive turnover. dbt is standard enough that a new hire has probably used it. A homegrown pipeline is not, and the person who wrote it is the documentation.

The stack is going to grow. dbt gets stronger as the stack grows. More tests, more docs, more lineage, more reasons the framework earns its keep. Homegrown gets weaker, because every new model is a new opportunity for the informal patterns to drift.

What happens when hand-rolled outgrows the person who built it

This is the failure mode nobody plans for and almost every one of these projects hits.

The person who built the stack is competent. The stack works. They understand every piece of it because they wrote every piece of it. Testing is in their head. Deployment is a runbook they never wrote down. Debugging is muscle memory nobody transferred.

Then something happens. They take a vacation. They get pulled onto another project. They leave. And the next person on the team cannot ship a change to the pipeline without a walkthrough that never happens, because the person who could give the walkthrough is not there.

Meanwhile the stack is now doing something the business depends on. A dashboard reads from it. Finance reads from it. Someone stood up a chatbot on top of it last quarter. Every one of those downstream consumers assumes the pipeline is a stable input, and every one of them will notice quickly if it stops being one.

The team's time starts going into keeping the stack running instead of moving it forward. Changes get slower. Bugs get older. New questions from the business get answered from the version of the data that predates whatever went wrong two months ago.

The migration to dbt is not intrinsically hard. What is hard is that you cannot both keep the stack running and rewrite it with the same person. This is where a scoped outside engagement usually shows up. Someone whose job is the migration, not the day-to-day operation, doing it against a stack that looks like yours because they have done it before against stacks that looked like the last one.

A decision framework

If the question is really "what should we do," here is the table I use when a stack audit lands on this exact fork in the road.

The two rows that catch most teams by surprise are the third and the fifth. The third is the moment where staying on hand-rolled is a decision, not a default, and most teams miss it because the pipeline is still working. The fifth is the moment where the AI or dashboard project that just launched changes the tolerance for wrong data, and the hand-rolled stack was never built for that tolerance.

The straight answer

Adopt dbt if you are past the prototype phase, more than one person will touch the code, or anything downstream matters enough to need tests. Stay hand-rolled if you are one person on a small stack that is not going to grow and the schedule is simple. If you are on hand-rolled and the situation changed a while ago without anyone noticing, the migration is the fix, and it is a scoped project rather than a rewrite that has to happen alongside daily operations.

The tool matters less than whether the person who understands it is still around.