protobuf type stubs obscure other imports from google.*

[nullie]

See this issue: microsoft/pyright#2113

[srittau]

We need to mark this package as "partial" by adding a py.typed file that contains the string partial\n (PEP 561). We need to add this functionality to the stub uploader, and also add a field to METADATA.toml.

[romanofski]

@srittau does this need to be fixed first before adding any more packages in the google namespace? I'd be happy to give it a crack. It seems easy to knock off. Would it be possible to point me to the stub uploader code? I've grepped through the source code and can't see where the magic happens :)

[JelleZijlstra]

It's in https://github.com/typeshed-internal/stub_uploader. Probably https://github.com/typeshed-internal/stub_uploader/blob/main/scripts/build_wheel.py is what needs to change.

[romanofski]

I spend some time on this today. Apart from the weird problem I ran with pyright, I've found that the way we currently package those namespace packages leads to problems when uninstalling them.

E.g. if I install types-protobuf and an additional google namespace package like types-google-cloud-ndb from #5821 they'll clobber each others contents in google-stubs. Perhaps the METADATA.toml isn't of significance to Python (I don't know), but at the point someone will uninstall at least one package, it'll wipe out one of the __init__.pyi files leading to import problems.

Now I wonder whether any google namespace stub packages will have to be packaged together to avoid such a scenario. Or they will have to share a common directory under stubs. Just spitballing here what a possible solution could look like.

[srittau]

It seems, google-cloud-ndb depends on protobuf, so we could just make the types package depend on each other like this. This way, protobuf has the "common" files, and google-cloud-ndb doesn't.

[nullie]

Maybe there should be a separate namespace package on which other packages would depend

[srittau]

Reading PEP 561 again, I'm not sure, partial will do the trick:

Many stub packages will only have part of the type interface for libraries completed, especially initially. For the benefit of type checking and code editors, packages can be "partial". This means modules not found in the stub package SHOULD be searched for in parts four and five of the module resolution order above, namely inline packages and typeshed.

Steps 4 and 5 are inline packages and typeshed, respectively. This means that partial can't be used to merge two third-party packages, if I understand correctly. Maybe @ethanhs can clarify.

[jakebailey]

That's my personal understanding (from our implementation in pyright/Pylance); the spec doesn't really allow for more than one partial package at a time (especially given that they must have py.typed at the root, which I don't think is possible across multiple distributions?). I believe we end up using the partial stub folder that's first in the search paths, but maybe that's not true for namespace packages.

By the same rules, a partial stub also can't override a py.typed library. Both things combined make our use of partial stubs in Pylance (working around hard-to-analyze libraries without fully typing them) a little fragile depending on the rest of the ecosystem.

[srittau]

Maybe there should be a separate namespace package on which other packages would depend

This is also a good idea.

[emmatyping]

If I remember correctly, technically PEP 561 talks about the order to resolve modules aka scipy.linalg not just entire packages (just scipy), so I believe you can have stubs that are both partial that implement different parts of a larger package, they just can't implement stubs for the same module.

[jakebailey]

Can you publish two partial distributions if py.typed needs to be at the module root of both? That's the bit that I'm finding restrictive (as I can only construct a situation where this works when they are in different import roots and therefore both provide py.typed, but that isn't how site-modules works).

[emmatyping]

I'm afraid I'm not sure I understand your scenario. Are you saying two namespace packages would interfere because they both need the py.typed at their root, and that would be an error on installation?

[jakebailey]

Yes, that's what I'm describing. This would affect both typeshed and regular packages published to PyPI, as they all get placed in site-packages.

[romanofski]

I've actually had a look why google.cloud.ndb depends on protobuf. It seems it's possible to deserialise an ndb.Model instance from a protobuffer.
Public interface wise there is actually nothing in common, except one single private function:

def _entity_from_protobuf(protobuf):
    """Deserialize an entity from a protobuffer.

    Args:
        protobuf (google.cloud.datastore_v1.types.Entity): An entity protobuf
            to be deserialized.

    Returns:
        .Model: The deserialized entity.
    """
    ds_entity = helpers.entity_from_protobuf(protobuf)
    return _entity_from_ds_entity(ds_entity)

So arguably the types do not depend on each other, but I think the possibility for them to be installed in the same site-packages is real.

The idea of having a common namespace package is neat, but I doubt it would scale very well if there are more packages to be typed.