py_common API

[UebelAndre]

Description of the feature request:

Similar to cc_common is there a py_common available?

What underlying problem are you trying to solve with this feature?

I would like to have some rules which generate additional actions but would otherwise be interchangeable with a py_library. I've done similar things before using the cc_common API but cannot find one for python.

Which operating system are you running Bazel on?

Any

What is the output of bazel info release?

release 5.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[comius]

cc @rickeylev

py_common is not available at the moment. We're doing rewrite on Python rules to Starlark, starting with the internal ones and then the ones native in Bazel. Implementing py_common now would interfere with the process, however I see no problem distilling it from Starlark implementation when it's finished.

As such I marked this as P4, but will upgrade it to P2 once Starlarkification is done.

[UebelAndre]

@comius is there a tracking issue for porting the python rules to starlark?

[UebelAndre]

@comius @rickeylev friendly ping. Is there a tracking issue for the port to starlark?

[rickeylev]

Sorry, didn't see this. No, I don't think there's a GitHub issue tracking it, yet. I'd create one now but am on vacation for a bit. Feel free to create one and assign it to me, though. Could you tell more about the problem you're trying to solve, and features you want? Internally, we have 3-5+ cases that could be described as "create a drop in replacement for py library", but each has some small differences that need different "plugin" points. So I'm curious to know which part of rule behavior your logic needs to intercept/customize/keep/drop

…

On Mon, Jul 4, 2022, 10:21 AM UebelAndre ***@***.***> wrote: @comius <https://github.com/comius> @rickeylev <https://github.com/rickeylev> friendly ping. Is there a tracking issue for the port to starlark? — Reply to this email directly, view it on GitHub <#1647>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEXQ6SZVUZOCCUFVSAFYLTVSMMSFANCNFSM5WSSU56Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[UebelAndre]

Could you tell more about the problem you're trying to solve, and features you want? Internally, we have 3-5+ cases that could be described as "create a drop in replacement for py library", but each has some small differences that need different "plugin" points. So I'm curious to know which part of rule behavior your logic needs to intercept/customize/keep/drop

I think the case that would be most impactful to me is being able to take my cc_binary-like rule and update it to provide an interface to python targets I have without needing to make a tree of macros as some of the attributes are hard to manage when select is used for some attributes. I'd come across the desire to do this a couple of times before writing this issue. In most cases I have a web of macros that could easily be condensed to a single rule.

@rickeylev would you be able to enumerate the cases you're aware of?

[rickeylev]

fyi, i created bazelbuild/bazel#15897 as a tracking issue for the starlark rewrite

[rickeylev]

A quick brain dump of some known points of customization:

• Being able to invoke pyc compilation as part of library building; similarly where to put such files in the output
• Writing out the executable script (internally, we replace this with a C executable; aspect does something to create venv-startup wrapper thing; internally we use a different stub script)
• Attaching aspects to various attributes and still acting as a regular py_library (useful for e.g. type checkers)
• Replicating env and args expansion for a "packaging" rule around a binary (e.g., you should be able to drop-in replace a regular py_binary with a par-based, pyoxidze-based, etc binary rule and have its args/env behavior be the same)
• Merging PyInfo objects (i.e. PyInfo equiv of cc_common.merge_cc_infos)
• Populating/propagating providers that a regular py_library would (this allows more flexibility and interoperability between the canonical py_library and custom variants of it)
• Being able to create compatible attribute definitions (e.g., the "stamp" attribute varies for tests vs binaries, or could vary by a particular type of binary; most variants of the canonical py_X rules will want to support largely the same attrs with the same behavior as the canonical rule)
• Maybe also helpers to help identify where to put files in runfiles

[rickeylev]

From Phillip Shrader over slack, the sort of customization they do at Aurora:

Our current approach is: bazel-generated wrapper calls into a shell script we wrote that sets up some hermeticity things and eventually calls path/to/python3 -SsB path/to/hermetic_python3.py actual/file/of/interest.py . And then path/to/hermetic_python3.py does a few things like clearing sys.path[0] and a few other hermeticity things and then eventually delegates to actual/file/of/interest.py .
So another potential interest of mine is to maybe combine the bazel generated wrapper with our shell script to reduce the number of steps involved in setting up all the hermetic stuff.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[UebelAndre]

Cc @rickeylev this is still desired but maybe the issue can be moved to rules_puthon? Now that Bazel 7 is out and we can use the starlark rules there?