Skip to content

feat(cli): add opt-in Spark function registration - #24179

Open
goutamadwant wants to merge 2 commits into
apache:mainfrom
goutamadwant:feat-cli-spark-functions
Open

feat(cli): add opt-in Spark function registration#24179
goutamadwant wants to merge 2 commits into
apache:mainfrom
goutamadwant:feat-cli-spark-functions

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

datafusion-cli does not currently register the datafusion-spark function library, so Spark-compatible functions cannot be explored interactively. Registering them by default would silently override core DataFusion functions with the same names and could change existing CLI behavior.

What changes are included in this PR?

  • Add a --spark CLI flag.
  • Register Spark-compatible scalar, aggregate, and window functions only when the flag is enabled.
  • Add CLI integration coverage for both the default and opt-in paths.
  • Document the flag and its function-override behavior.

Are these changes tested?

Yes. The following checks pass:

  • cargo test -p datafusion-cli --test cli_integration spark_functions_require_opt_in
  • cargo test -p datafusion-cli --all-targets
  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • ./ci/scripts/doc_prettier_check.sh
  • ./ci/scripts/license_header.sh
  • ./ci/scripts/typos_check.sh
  • ./ci/scripts/rust_docs.sh

Are there any user-facing changes?

Yes. Users can pass --spark to enable Spark-compatible functions for the CLI session. The option is disabled by default, so existing CLI behavior is unchanged. When enabled, Spark-compatible functions override DataFusion functions with the same name.

@codecov-commenter

codecov-commenter commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.05%. Comparing base (c1cb715) to head (5e1213d).

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #24179   +/-   ##
=======================================
  Coverage   81.05%   81.05%           
=======================================
  Files        1106     1106           
  Lines      382210   382210           
  Branches   382210   382210           
=======================================
+ Hits       309812   309816    +4     
+ Misses      54091    54083    -8     
- Partials    18307    18311    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the datafusion-spark function library available in datafusion-cli behind an opt-in flag

2 participants