Skip to content

feat: implement substrait join filter support - #6868

Merged
alamb merged 4 commits into
apache:mainfrom
nseekhao:substrait-join-filter
Jul 9, 2023
Merged

alamb merged 4 commits into
apache:mainfrom
nseekhao:substrait-join-filter

Conversation

@nseekhao

@nseekhao nseekhao commented Jul 6, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6866 .

Rationale for this change

Explained in issue.

What changes are included in this PR?

  • Parse DF join filter into Substrait's post_join_filter in producer
  • Parse post_join_filter into join filter in consumer
  • Test cases:
    • non-empty join expression, non-empty filter
    • empty join expression, non-empty filter
    • query with exists clause

Are these changes tested?

Yes.

Please note that if a test case includes subquery aliases, the results will not be correct due to issue #6867 .

Are there any user-facing changes?

No

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Jul 6, 2023
@nseekhao

nseekhao commented Jul 6, 2023

Copy link
Copy Markdown
Contributor Author

Does anyone know what's the difference between the tests/roundtrip_logical_plan.rs file and the tests/cases/roundtrip_logical_plan.rs file?

@nseekhao nseekhao changed the title Substrait join filter support feat: implement substrait join filter support Jul 6, 2023
@andygrove

Copy link
Copy Markdown
Member

@waynexia fyi

@waynexia waynexia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great, thanks 👍

@waynexia

waynexia commented Jul 9, 2023

Copy link
Copy Markdown
Member

Does anyone know what's the difference between the tests/roundtrip_logical_plan.rs file and the tests/cases/roundtrip_logical_plan.rs file?

I just realized we have two different case files 🤣 It looks like this is a mistake from #6604. I'll reorganize those test cases later when I split our consumer/producer implementation into functionality-grouped style. I believe there are many duplications...

@alamb
alamb merged commit ce3a0bb into apache:main Jul 9, 2023
@alamb

alamb commented Jul 9, 2023

Copy link
Copy Markdown
Contributor

I just realized we have two different case files 🤣 It looks like this is a mistake from #6604. I'll reorganize those test cases later when I split our consumer/producer implementation into functionality-grouped style. I believe there are many duplications...

I took a quick look at it looks to me like the tests are entirely duplicated. Proposed removing them here: #6894

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait: Add support for joins with filter

4 participants