Joel Labes

As part of releasing dbt v2 to general availability, we've renamed the two distributions to reflect their positions in the ecosystem. The dbt Fusion engine is now just called dbt, and dbt Core v2 is now called dbt OSS. We expect that most people adopting dbt v2 will adopt the primary dbt distribution, which is free to use and locally installable.

The name changes do not change our commitment to open source: there is and always will be a completely open source implementation of the dbt standard. All of the code we released to Apache 2.0 from ELv2 or proprietary licenses in June stays the same, and moving to a shared foundation (instead of having dbt Core remain on the legacy Python engine) makes the open source version of dbt stronger.

The change to two freely usable distributions with a shared foundation is the biggest point of confusion we've seen during the v2 beta. This post will help explain the differences between dbt and dbt OSS, and provide guidance on which distribution you should choose.

After 18 months of engineering, reverse-engineering, de-engineering, and re-engineering, the dbt Fusion engine has graduated into General Availability under its new name: dbt. (Read more about that here). This complete rewrite of dbt solves a number of longstanding problems in dbt v1's implementation, and lays the foundations for another decade of progress. Before:

• dbt didn't understand the SQL it was generating, so it couldn't warn about invalid code without executing the code and consuming compute resources.
• At large scale, dbt could take as long as 20 minutes to parse a project before any execution could begin.
• dbt's artifacts made interoperability difficult, with enormous JSON files being the only way to communicate from one product to another.
• dbt docs was unable to work for many teams due to scaling and performance issues.

The new dbt v2:

• Includes rich SQL comprehension and the ability to catch code mistakes before they hit the warehouse, saving developers time and money and providing verifiable loops for agents.
• Is built for speed throughout, by moving to an optimized Rust binary and using modern technologies like ADBC. We've seen projects that used to take 20 minutes to parse now complete in less than one, and more normal-sized projects will also see a big speedup in daily usage (such as 2x or more faster compilation).
• Introduces the dbt Information Schema which describes an entire project as standard Parquet files. This can reduce file sizes by over 10x while providing richer data, and enables speedy querying through dbt show --info, external tools like DuckDB, and the brand new checks feature.
• Includes a brand new version of dbt docs, which is much more scalable.

When building a dbt project, you start with the simplest strategy and only add complexity when you need more speed or reduced cost. At a model-by-model level, this is the view → table → incremental materialization pattern. At a project-wide level, optimizations used to require a lot more complexity - rebuilding only the relevant parts of a project meant you spent a lot of mental energy chaining together selectors like state:modified, source_status:fresher, or config.materialized:view and ensuring fresh JSON artifacts were distributed across your organization.

dbt State provides more speed, reduced cost, and less complexity all in one. It works wherever you do: dbt Core or Fusion, locally or on any orchestrator you choose. Since it's backed by a new paid service (with a 30 day free trial), you don't need to adopt the whole dbt platform to get these savings. dbt State supersedes state-aware orchestration, which required a dbt platform subscription and moving to Fusion.

The most technically interesting part of this is the query normalization process1 which means running the new dbt lint command doesn't instantly sign you up for a full rebuild. But before getting into the depths of syntax trees, let's recap what dbt State actually does for an end user.

The basic dbt State workflow​

When dbt runs, it will check with the dbt State server whether a model:

• already exists in the target schema and can be reused
• already exists in another schema and can be reused, or
• needs to be built from scratch.

Today, we published the first alpha release of dbt Core version 2.0, raising the baseline for what dbt users can expect from their transformation tool. As always, dbt Core's code is completely open source under the Apache 2.0 license. What makes this release significant is that dbt Core 2.0 is now built on the same foundations as the dbt Fusion engine – we've open-sourced a lot of Fusion code for the first time.

The launch of dbt Core 2.0 comes with key feature developments:

• Significant parse time improvements, especially on the largest dbt projects
• A tightly-defined language spec which makes it impossible to accidentally configure a desciptin instead of a description, and gives anyone integrating with the dbt ecosystem (including dbt's own development team!) a stable interface to build against.
• New Parquet artifacts as a high-performance alternative to large JSON files that can be directly queried through DuckDB or your agent of choice
• A completely revamped local documentation experience, powered by those new artifacts and capable of scaling to projects of arbitrary size.
• A more streamlined way to build new adapters, powered by ADBC and the Arrow ecosystem
• A simplified installation process that removes the need to fight with Python's virtual environments

Community-driven creation and curation of best practices is perhaps the driving factor behind dbt and analytics engineering’s rise - transferrable workflows and processes enable everyone to create and disseminate organizational knowledge. In the early days, dbt Labs’ Fishtown Analytics’ dbt_style_guide.md contained foundational guidelines for anyone adopting the dbt viewpoint for the first time.

Today we released a collection of dbt agent skills so that AI agents (like Claude Code, OpenAI's Codex, Cursor, Factory or Kilo Code) can follow the same dbt best practices you would expect of any collaborator in your codebase. This matters because by extending their baseline capabilities, skills can transform generalist coding agents into highly capable data agents.

dbt agent skills allow you to transform generalist coding agents into highly capable data agents

These skills encapsulate a broad swathe of hard-won knowledge from the dbt Community and the dbt Labs Developer Experience team. Collectively, they represent dozens of hours of focused work by dbt experts, backed by years of using dbt.

With access to skills, agents like Claude take a systematic approach to tasks

Today, we announced the dbt Fusion engine.

Fusion isn't just one thing — it's a set of interconnected components working together to power the next generation of analytics engineering.

This post maps out each piece of the Fusion architecture, explains how they fit together, and clarifies what's available to you whether you're compiling from source, using our pre-built binaries, or developing within a dbt Fusion powered product experience.

From the Rust engine to the VS Code extension, through to new Arrow-based adapters and Apache-licensed foundational technologies, we'll break down exactly what each component does, how each component is licensed (for why, see Tristan's accompanying post), and how you can start using it and get involved today.

Today, we announced that the dbt Fusion engine is available in beta.

• If Fusion works with your project today, great! You're in for a treat 😄
• If it's your first day using dbt, welcome! You should start on Fusion — you're in for a treat too.

Today is Launch Day — the first day of a new era: the Age of Fusion. We expect many teams with existing projects will encounter at least one issue that will prevent them from adopting the dbt Fusion engine in production environments. That's ok!

We're moving quickly to unblock more teams, and we are committing that by the time Fusion reaches General Availability:

• We will support Snowflake, Databricks, BigQuery, Redshift — and likely also Athena, Postgres, Spark, and Trino — with the new Fusion Adapter pattern.
• We will have coverage for (basically) all dbt Core functionality. Some things are impractical to replicate outside of Python, or so seldom-used that we'll be more reactive than proactive. On the other hand, many existing dbt Core behaviours will be improved by the unique capabilities of the dbt Fusion engine, such as speed and SQL comprehension. You'll see us talk about this in relevant GitHub issues, many of which we've linked below.
• The source-available dbt-fusion repository will contain more total functionality than what is available in dbt Core today. (Read more about this here.)
• The developer experience will be even speedier and more intuitive.

These statements aren't true yet — but you can see where we're headed. That's what betas are for, that's the journey we're going on together, and that's why we want to have you all involved.

Remember how dbt felt when you had a small project? You pressed enter and stuff just happened immediately? We're bringing that back.

Benchmarking tip: always try to get data that's good enough that you don't need to do statistics on it

After a series of deep dives into the guts of SQL comprehension, let's talk about speed a little bit. Specifically, I want to talk about one of the most annoying slowdowns as your project grows: project parsing.

When you're waiting a few seconds or a few minutes for things to start happening after you invoke dbt, it's because parsing isn't finished yet. But Lukas' SDF demo at last month's webinar didn't have a big wait, so why not?

Ever since dbt Labs acquired SDF Labs last week, I've been head-down diving into their technology and making sense of it all. The main thing I knew going in was "SDF understands SQL". It's a nice pithy quote, but the specifics are fascinating.

For the next era of Analytics Engineering to be as transformative as the last, dbt needs to move beyond being a string preprocessor and into fully comprehending SQL. For the first time, SDF provides the technology necessary to make this possible. Today we're going to dig into what SQL comprehension actually means, since it's so critical to what comes next.

Cloud Data Platforms make new things possible; dbt helps you put them into production​

The original paradigm shift that enabled dbt to exist and be useful was databases going to the cloud.

All of a sudden it was possible for more people to do better data work as huge blockers became huge opportunities:

• We could now dynamically scale compute on-demand, without upgrading to a larger on-prem database.
• We could now store and query enormous datasets like clickstream data, without pre-aggregating and transforming it.

Today, the next wave of innovation is happening in AI and LLMs, and it's coming to the cloud data platforms dbt practitioners are already using every day. For one example, Snowflake have just released their Cortex functions to access LLM-powered tools tuned for running common tasks against your existing datasets. In doing so, there are a new set of opportunities available to us:

Environmental segmentation has always been an important part of the analytics engineering workflow:

• When developing new models you can process a smaller subset of your data by using target.name or an environment variable.
• By building your production-grade models into a different schema and database, you can experiment in peace without being worried that your changes will accidentally impact downstream users.
• Using dedicated credentials for production runs, instead of an analytics engineer's individual dev credentials, ensures that things don't break when that long-tenured employee finally hangs up their IDE.

Historically, dbt Cloud required a separate environment for Development, but was otherwise unopinionated in how you configured your account. This mostly just worked – as long as you didn't have anything more complex than a CI job mixed in with a couple of production jobs – because important constructs like deferral in CI and documentation were only ever tied to a single job.

But as companies' dbt deployments have grown more complex, it doesn't make sense to assume that a single job is enough anymore. We need to exchange a job-oriented strategy for a more mature and scalable environment-centric view of the world. To support this, a recent change in dbt Cloud enables project administrators to mark one of their environments as the Production environment, just as has long been possible for the Development environment.

Explicitly separating your Production workloads lets dbt Cloud be smarter with the metadata it creates, and is particularly important for two new features: dbt Explorer and the revised CI workflows.

Once your data warehouse is built out, the vast majority of your data will have come from other SaaS tools, internal databases, or customer data platforms (CDPs). But there’s another unsung hero of the analytics engineering toolkit: the humble spreadsheet.

Spreadsheets are the Swiss army knife of data processing. They can add extra context to otherwise inscrutable application identifiers, be the only source of truth for bespoke processes from other divisions of the business, or act as the translation layer between two otherwise incompatible tools.

Because of spreadsheets’ importance as the glue between many business processes, there are different tools to load them into your data warehouse and each one has its own pros and cons, depending on your specific use case.

As we get closer to dbt v1.0 shipping in December, it's a perfect time to get your installation up to scratch. dbt 1.0 represents the culmination of over five years of development and refinement to the analytics engineering experience - smoothing off sharp edges, speeding up workflows and enabling whole new classes of work.

Even with all the new shinies on offer, upgrading can be daunting – you rely on dbt to power your analytics workflow and can’t afford to change things just to discover that your daily run doesn’t work anymore. I’ve been there. This is the checklist I wish I had when I owned my last company’s dbt project.