RFC: Improve Plan Algorithm With Column Level Lineage

Summary

SQLMesh OSS's plan algorithm is not as efficient as Tobiko Cloud. There are numerous efficiency improvements that can be introduced into the plan logic by introducing column level lineage.

Plan Algorithm Comparison

Parent users has id, email, username. Child A selects id, email. Child B selects only id. Grandchild C selects id from A.

        users
 (id, email, username)
       /         \
      /           \
     A             B
(id, email)       (id)
     |
     C
    (id)

Change (to users)	OSS today	Tobiko Cloud (inferred)
Add `phone`	A, B, C skip	Same
Change `email` expression	A, B, C rebuild	A rebuilds; B, C skip
Drop unused `username`	A, B, C rebuild	A, B, C skip (`users` still rebuilds)
Change `WHERE` statement	A, B, C rebuild	A, B, C rebuild

Current Pseudo-Code

plan (DAG order)
  │
  ├─ new model ──────────────────────── BREAKING
  │
  ├─ directly modified
  │     ├─ kind change / SELECT * ───── BREAKING
  │     └─ compare old vs new query
  │           ├─ only added columns ─── NON_BREAKING
  │           └─ anything else ──────── BREAKING
  │
  └─ indirectly modified
        ├─ parent NON_BREAKING ──────── SKIP
        └─ parent BREAKING ──────────── REBUILD

Future Pseudo-Code

plan (DAG order)
  │
  ├─ new model ──────────────────────── BREAKING
  │
  ├─ directly modified
  │     ├─ kind change / SELECT * /
  │     │  can't render / Python ────── BREAKING   (same as today)
  │     └─ column diff (old vs new)
  │           ├─ only added columns ─── NON_BREAKING
  │           ├─ whole row set changed ─ BREAKING   (WHERE / JOIN / …)
  │           └─ some columns dirty ─── rebuild self
  │                                      (descendants decided below)
  │
  └─ indirectly modified
        ask lineage: do I depend on a dirty column?
          ├─ no ─────────────────────── SKIP
          └─ yes ────────────────────── REBUILD
              (and mark my outputs dirty
               so grandchildren can ask the same)

Logic Simplified

Directly Modified - Model's own query/data changed. Fingerprint data hash changed. It will rebuild itself (unless forward-only, ignore)
Indirectly Modified - Model's SQL did not change. A parent did. Current logic copies parent's "category". Future logic will use column-level to determine "category"

Category	This model	Meaning
`BREAKING` / `NON_BREAKING`	Direct	Always rebuild self. Difference is only what descendants do today.
`INDIRECT_BREAKING`	Indirect	Rebuild this model
`INDIRECT_NON_BREAKING`	Indirect	Skip — reuse the old physical table

For children descendants, we can improve the logic to determine categories INDIRECT_BREAKING or INDIRECT_NON_BREAKING by looking at the column level changes. We would traverse the table lineage and for each model that has children, we would determine for each column:

This column	Means	If the child reads it
Clean	Same as before	Does not force a rebuild. `INDIRECT_NON_BREAKING`
Dirty	Not the same (rewritten, from a dirty parent column, or row set changed)	Forces `INDIRECT_BREAKING`
Added	New column	Does not force a rebuild. `INDIRECT_NON_BREAKING`
Removed	Column is gone	Forces `INDIRECT_BREAKING`

Then:

Any dirty/removed column the child reads → model is INDIRECT_BREAKING → keep labeling that model’s outputs for its children.
Child reads only clean/added → model is INDIRECT_NON_BREAKING → all of its outputs are clean (this parent won’t force descendants to rebuild).
Can’t prove → INDIRECT_BREAKING (same as today).

Goal: more models correctly land on INDIRECT_NON_BREAKING and are not rebuilt.

High Level Pieces

We need three major pieces here:

Column diff on the changed model
Did this change only some columns, or the whole row set (WHERE / JOIN / etc.)? List clean, dirty, added, removed.
Ask who reads those columns
Using lineage (and filters/joins, not just SELECT): which downstream models/columns actually depend on the dirty columns?
Rebuild only that cone
The changed model still rebuilds. Descendants that lineage says are untouched skip. That’s B skipping when email changed, and C skipping when it only uses A’s id and A’s rows didn’t move.

Right now OSS does some of piece 1. It answers "did we only add columns to the model?" -- It does not list columns. Current logic treats "additive" as non-breaking. Anything else that isn't just a pure add ends with a "not sure" conclusion -> rebuild everyone.

Only comparison happens in sqlmesh/core/definition.py in is_breaking_change function right now. Current logic determines breaking or non_breaking. Then that affects descendants.

Starting Pieces

Already there	Role today	What it becomes
`sqlmesh/core/model/definition.py` — `is_breaking_change` / `_is_only_projection_additions`	Additive or not (False / None)	Piece 1: column diff: `clean`, `dirty`, `added`, `removed`
`sqlmesh/core/snapshot/categorizer.py` — `categorize_change`	Maps that to `NON_BREAKING` / `BREAKING` on the direct model	Same job, richer input
`sqlmesh/core/plan/builder.py` — `_categorize_snapshot`	Direct path calls categorizer; indirect path copies the "category" of the parent	Piece 3: indirect path asks lineage instead of copying "category" from parent
`sqlmesh/core/lineage.py`	UI/LSP column graph	Piece 2: “does this descendant depend on a `dirty` column?”

Initial Steps

To be determined. We have to be very careful about this because we're modifying core logic. This also introduces a much bigger layer of checks. How do we keep this performant?

References

Tobiko Cloud claims:

"Tobiko Cloud comes with even more change categorization capabilities than open-source SQLMesh, such as advanced column-level impact analysis." -- https://sqlmesh.readthedocs.io/en/stable/cloud/features/costs_savings/#where-to-find-cost-and-savings-information
"Unaffected Downstream: SQLMesh understands SQL, so we skip re-execution if a downstream model is not affected by an upstream change." -- https://sqlmesh.readthedocs.io/en/stable/cloud/features/costs_savings/#where-to-find-cost-and-savings-information
"While this study focuses on benchmarking SQLMesh against dbt Core, we also evaluated Tobiko Cloud, the hosted version of SQLMesh, to examine its handling of breaking changes. SQLMesh uses column-level lineage to differentiate between breaking and non-breaking changes — and Tobiko Cloud extends this capability with more surgical change categorization, refining how downstream updates are processed when a column is modified. This further reduces time and compute spent on backfills, making Tobiko Cloud even more performant than its counterpart — by a factor of seven." -- https://medium.com/dbsql-sme-engineering/databricks-benchmark-study-shows-sqlmesh-outperforms-dbt-core-by-orders-of-magnitude-on-speed-and-0f3b6f281888

I'd like to pick this up, but starting with design rather than code, given your note that the initial steps are TBD and that this touches the part of the plan logic where being wrong is expensive.

What I'd like to put in front of you first, before any implementation:

What sqlmesh/core/lineage.py can and cannot answer today for piece 2. It's built for the UI and LSP, so the question is whether "does this descendant read a dirty column" is answerable from it as-is, or whether it needs a different traversal.
Whether is_breaking_change / _is_only_projection_additions can be widened to return a real column diff (clean / dirty / added / removed) without disturbing the existing call sites, or whether that wants to sit beside them instead.
Where the cost lands. Column-level resolution per model on every plan is the obvious risk you flagged, so I'd want to know what it does to plan time on a wide project before anyone commits to it.
A staged sequence, so the riskiest change — the indirect path in builder.py no longer copying the parent's category — lands last and behind a flag, with the column diff landing first as a pure addition that changes no behaviour.

The table in your RFC is clear enough that the semantics aren't really in question; the risk is all in piece 2 and performance. Happy to be reassigned if you'd rather scope this yourself before anyone starts.

@tripleaceme

Hey @tripleaceme -- this is not in a state to be implemented just yet, more work needs done in figuring out how to safely implement the changes to the plan command. The RFC stands for "Request for Comment" -- basically means this isn't ready yet, but feedback/discussion is welcome to help get it in a "Ready" state and collaborate on an approach.

Given the scope of the issue, the other TSC members / maintainers will also need to be involved before any work to the codebase begins. We'll need to be very intentional with this.

If you want to investigate some of the sections of code presented and report findings here, I'd appreciate it!

@tripleaceme

Hey @tripleaceme -- I appreciate the initiative, but 3700 words of analysis isn't the right fit for an RFC thread. These are meant for conversation, not design docs. Please edit this down to a high-level summary: is this feasible? What are the top 2-3 blockers? If you want to share the detailed breakdown, hold it until we've had initial discussions. We need scannable contributions here so people can actually engage.

Discovered the sqlmesh.core.lineage.column_dependencies function that given a column, will show which upstream models provide the column (or columns needed for the calculation). So if SUM(quanity * price) AS revenue is provided as the column, it will go upstream and find the models that provide quantity and price. Only intakes one column though.

More functionality can obviously be added to the sqlmesh.core.lineage module. Needs better testing because it was developed for the IDE Extension/Web UI specifically.

Navigation Menu

Repository navigation

RFC: Improve Plan Algorithm With Column Level Lineage #6062

Summary

Plan Algorithm Comparison

Current Pseudo-Code

Future Pseudo-Code

Logic Simplified

High Level Pieces

Starting Pieces

Initial Steps

References

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC: Improve Plan Algorithm With Column Level Lineage #6062

Description

Summary

Plan Algorithm Comparison

Current Pseudo-Code

Future Pseudo-Code

Logic Simplified

High Level Pieces

Starting Pieces

Initial Steps

References

Activity

tripleaceme commented on Sep 17, 2026

cmgoffena13 commented on Sep 18, 2026

tripleaceme commented on Sep 18, 2026

cmgoffena13 commented on Sep 20, 2026

cmgoffena13 commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions