Release DataFusion `53.1.0` (minor) (Apr 2026)

[alamb]

Is your feature request related to a problem or challenge?

• Part of [EPIC] DataFusion Releases #19783
• Follow on to Release DataFusion 53.0.0 (Feb 2026 / Mar 2026) #19692

I suspect we'll end up needing to backport a few bugs on the branch-53 release, so I am proactively opening a ticket to propose / track such a release

Steps:

• Update version and changelog:
• Voting Thread: https://lists.apache.org/thread/krg9ghwhh58o83014ncjs4zw73mvdq0w
• Publish to crates.io

Potential Content:

NOTE: we should also make sure that we backport / release anything that also went into the branch-52 line:

• Release DataFusion 52.5.0 (minor) Release (Apr 2026) #21078

Requested Bug Fixes. Format: Issue / Requester / Backport PR

• Debug assertions currently failing on main #20831 / @alamb / Update to arrow/parquet 58.1.0 #21044

Requested improvements

• Super fast extended tests and improved planning speed linux #21084 / @alamb @blaginin: [branch-53] Planning speed improve (port of #21084) #21137
• chore: Optimize schema rewriter usages #21158 @comphead / [branch-53] chore: Optimize schema rewriter usages (#21158) #21183
• fix: do not recompute hash join exec properties if not required #20900 / @alamb @askalt / [branch-53] fix: do not recompute hash join exec properties if not required (#20900) #20903
• fix: use spill writer's schema instead of the first batch schema for spill files #21293 / @alamb @gruuya / [branch-53] fix: use spill writer's schema instead of the first batch schema for … #21451
• fix: use datafusion_expr instead of datafusion crate in spark  #21043 (comment) / @rluvaton / [branch-53] fix: use datafusion_expr instead of datafusion crate in spark bitmap/… #21452
• regression: ProjectionPushdown failed with project index 2 out of bounds, max field 2 #21459 / @haohuaijin / [branch-53] fix: FilterExec should drop projection when apply projection pushdown #21492
• [branch-52] Restore Sort unparser guard for correct ORDER BY placement (#20658) #21358 / @alamb / [branch-53] Restore Sort unparser guard for correct ORDER BY placement (#20658) #21523
• [BRANCH-52] fix: foreign inner ffi types #21439 / @alamb / [branch-53] fix: foreign inner ffi types (#21439) #21524

[hareshkh]

@alamb : Can we add the following issues to this release please?

• Filter pushdown past children with fetch limits causes correctness bug #21063 (cc @shivbhatia10)
  • Backport PR: [branch-53] Fix push_down_filter for children with non-empty fetch fields (#21057) #21142
• Substrait join consumer produces incorrect logical plan #21124 (cc @gabotechs)
  • Backport PR: [branch-53] Substrait join consumer should not merge nullability of join keys (#21121) #21162

[alamb]

I am happy to run this release unless someone else wants to

I'll plan to get it ready for next week sometime

[comphead]

I am happy to run this release unless someone else wants to

I can continue with 53ish to get a full picture on release and hotfixes

[alamb]

Sold! I'll assign it you you @comphead

Thank you

I will then help with some of the backports with myfreed up time

[comphead]

I'm planning to make all backports today

[alamb]

Thanks @comphead -- We should also make sure that anything backported into 52.5.0 is ported to branch-53. I will do that review tomorrow and make any additional backports needed

[comphead]

• fix: do not recompute hash join exec properties if not required #20900 was applied before, prob its missing from change log

• fix: use spill writer's schema instead of the first batch schema for spill files #21293 backport [branch-53] fix: use spill writer's schema instead of the first batch schema for … #21451

• fix: use datafusion_expr instead of datafusion crate in spark  #21043 backport [branch-53] fix: use datafusion_expr instead of datafusion crate in spark bitmap/… #21452

• Remove the statistics() api in execution plan #20319 exists, missing from change log

• feat: Add null-aware anti join support #19635 is huge, needs to be merged properly

[haohuaijin]

can we add this to the list？

• regression: ProjectionPushdown failed with project index 2 out of bounds, max field 2 #21459

[alamb]

can we add this to the list？

Added