Release Datafusion 6.0.0

[houqp]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We had some oversights in the 5.0.0 release (#771) causing us not able to release the python binding and datafusion-cli.

Describe the solution you'd like

Release Datafusion 5.1.0 with an improved process to support python binding and cli releasse.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
see #887, #883 and #837

[mmuru]

@houqp: Do we have an ETA on the python binding release? Thanks.

[houqp]

@mmuru probably 2-3 weeks, need to get some legal issues resolved in #920 before we can cut a release tarball and start the voting process.

[mmuru]

@houqp: Thanks for the update. I am trying to build python wheel locally but I noticed in Cargo.toml file the dependency listed
datafusion = { git = "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/apache/arrow-datafusion.git", rev = "4d61196dee8526998aee7e7bb10ea88422e5f9e1" }
It did not get updated. Will maturin develop pick up the latest datafusion code? Please, can you clarify it? Thanks again.

[houqp]

@mmuru it will not, I just sent #967 to handle the datafusion update.

[mmuru]

@houqp: Thanks for the clarification and quick turnaround fix. I verified your changes and found two issues.

1. the requirement.txt is locked to Python 3.8 version. I think, this file should not be checked in the source since the python development version could be different. In my case, it was python 3.7 version.
2. Fixed the issue Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives "type16 not valid error" when schema includes date32[day] type #949. Added support for pyarrow datatypes such as date32, date64 and Timestamp.
   I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

[houqp]

the requirement.txt is locked to Python 3.8 version.

The requirements.txt is supposed to work for all python versions supported by the python binding. if it's broken for python 3.7, could you file a separate issue with the error message? We can continue the troubleshooting there.

I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

That's great, I recommend you send a separate PR based off the branch in #967 for this, or collaborate on #969 to get that issue addressed.

[mmuru]

@houqp: Sure, created #975. Ping me if you need more information.

[houqp]

@andygrove @alamb @Dandandan @jorgecarleitao @nevi-me given that we have had many major breaking changes merged in since the 5.x release, I am thinking maybe it's better to skip 5.1 and go 6.0 after #1010 gets merged. What do you think?

[alamb]

merged in since the 5.x release

I think using 6.0 is a good idea. I also don't think we need to wait for #1010 to be merged for a release, if we need to get the python binding / cli out sooner

[houqp]

sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

[alamb]

sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

It sounds like we are aiming to release arrow 6.0 in 2 weeks or so