Release a new version of DataFusion to crates.io

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The current version of datafusion on crates.io is 4.0.0 (which among other things doesn't work with the newly released arrow 5.0.0)

Describe the solution you'd like
It would be great if we released a DataFusion 5.0.0 (or some other number if we want to diverge from arrow) -- doing so would likely involve porting the arrow-rs release scripts, from https://github.com/apache/arrow-rs/tree/master/dev/release and then sending it to the dev mailing list for a formal vote

Starting this ticket to gather some feedback

[houqp]

@andygrove @jorgecarleitao I couldn't find the exact commit for the 4.0.0 version we have in crates.io, Do you know where can we find it? It would be good to retroactively create the 4.0.0 tag for change log automation. I am guessing the commit is around ddaea81?

@jorgecarleitao do we want to bundle the python release with the main datafusion release as well? Or do we want to release python binding in its own cadence? I think if the python binding wants to have a more frequent release cadence than the core, then it's better to have a separate release process for it. Otherwise it would be less work to bundle it with the core Datafusion release.

[alamb]

I think 4.0.0 was released when datafusion was in the https://github.com/apache/arrow repo, specifically https://github.com/apache/arrow/tree/f959141ece4d660bce5f7fa545befc0116a7db79

I dug that version out of the email thread for the 4.0.0 RC3 release:

https://lists.apache.org/thread.html/rfb4e0065347235571cd0e0e71a7fa38c9652f07e27f98f89b4398feb%40%3Cdev.arrow.apache.org%3E

[andygrove]

We also need to consider what to do about Ballista release cadence and version numbers. My preference would be to release DataFusion and Ballista together. Ballista version is currently 0.5.0-SNAPSHOT and I think that releasing as 0.5.0 would be appropriate.

[jorgecarleitao]

That would be awesome, I would +1 on aligning to 0.5.0, also for Python Datafusion (i.e. bump from 0.2.1 to 0.5.0); I think we do have a problem with DataFusion, that is already 5.0.0 on crates.io?

[alamb]

FWIW, the most recent release to crates.io for datafusion is 4.0.0: https://crates.io/crates/datafusion

Given that DataFusion doesn't have the associated arrow ecosystem I think using an entirely different versioning scheme is fine for DataFusion and Ballista - I don't really have a strong opinion on versions for DataFusion (e.g I would be happy to have it be 0.5.0 or something, though I don't know how we would handle the existing versions)

[alamb]

BTW @houqp / @jimexist I am happy to help here / run whatever scripts / share my experience with doing the release process with arrow-rs -- just let me know what I can do to help

[houqp]

I also prefer we go with our own versioning scheme to better adhere to semantic versioning at the very least for ballista and datafusion python binding.

For datafusion the ship has already sailed unfortunately. If we are to release datafusion and ballista together, how do we manage the version differences? Do we just name the same tarball with different names? Or do we actually produce two different release tarballs with project specific sources?

Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @jimexist perhaps you could help start building the automation for that?

[houqp]

Here is what I think could work for us, release all project sources in a single signed repo source tarball, which gets uploaded to apache svn. Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.

For example, in the upcoming release, we could have datafusion-5.0.0, ballista-0.5.0 and datafusion python 0.3.0 (or 0.5.0 if we want to align it with ballista) published from the same source tarball apache-arrow-datafusion-5.0.0.tar.gz. Name of the source tarball won't matter much here since I don't expect downstream consumers of these projects to use it directly. Instead they would use crates.io or pypi with the proper project name and version.

Consequence of this is every time we need to release a new version of either datafusion, python binding or ballista, we would need to vote and release a new version the datafusion repo as a whole.The repo source release should be pretty light weight given we don't need to do maintenance releases in current state of the project. All we have to do is to run create-tarball.sh , send to dev list for vote, then finally run release-tarball.sh.

Changelog for each project will need to be generated separately before we propose a source release in apache svn.

WDYT?

[houqp]

on a side note, turns out the tree we used for 4.0.0 release (https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79) is not in our repo.

Should I push this tree to our repo and tag it with 4.0.0? Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.

[jimexist]

Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @jimexist perhaps you could help start building the automation for that?

@jorgecarleitao any previous work already done this area?

also there's how and where docs are supposed to be hosted, e.g.

• jax uses read the docs, so is arrow which is likely the easiest
• pandas has its own
• requests has a different style but is also hosted by read the docs

[alamb]

Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.

I think that would work nicely and I think it makes a lot of sense.

Should I push this tree to our repo and tag it with 4.0.0?

I would recommend not doing this (because it will effectively bring along the (large) history of the arrow repo with it, effectively making the arrow-datafusion repo several times larger).

Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.

I think tagging 31dd3cd as 4.0.0 is fine (and I did something similar in arrow-rs) -- the official apache release policy, at least as I understand and discussed on the arrow-dev mailing lists, was centered around the tarballs as the artifacts. Since the 4.0.0 release / announcement was built from https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79 if anyone wants to know exactly what was in the release they can use that reference. tagging 31dd3cd in this repo to compute the changelog seems like it would be fine

[houqp]

@jimexist there were some previous discussion on how to host a website for datafusion on the dev list: https://lists.apache.org/thread.html/r0ed76cc60cdf651e8cf5c82a21cc64114c1f6d174dc5487434bd32ef%40%3Cdev.arrow.apache.org%3E.

Read the docs is certainly the route with the least amount of work, but I am not 100% sure if it's something allowed for apache projects. The dev mailing list recommended hosting our docs as a sub path under the main arrow website. This incurs more work since this is not a turnkey solution. But it has the added benefit of us leveraging the arrow brand for marketing Datafusion and potentially improve SEO. We might be able to reuse the same automation that's used to generate https://arrow.apache.org/docs/python/api.html for datafusion too.