More ReDOS fixes!

[Rocketknight1]

This PR replaces three more problematic regexes. In all cases I eliminate the regex entirely.

When parsing Python files for import statements we now rely on ast.parse() instead. This is the most complex code but it seems to pass tests.

In chat.py I think I figured out what the regex was doing and wrote a Python method to duplicate its functionality - cc @gante to confirm I got it right!

Finally, the change in configuration_utils.py was very straightforward; we could just use str.startswith() and str.endswith() instead.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Happy with config changes!
imports should be tested in terms of speed. Ast is a bit slow as I remember it but for something lile this we can cache / optimize!
cc @LysandreJik

[Rocketknight1]

ast.parse() seems quite performant for me! I parsed modeling_phi4.py (2.3k LOC) in ~10ms.

[gante]

@Rocketknight1 yeah, the chat changes look good to me

[Rocketknight1]

One more performance note: Since most Transformers code is contained within classes, we can speed up walking the ast tree a lot if we just don't recurse into classes. This will miss imports inside classes, but I don't know if we count those as required imports or not, and probably we don't want to?

However, I don't think this is really that important - parsing and walking the tree takes << 1s for me. I'm not even sure it's slower than the regex solution!

[ArthurZucker]

SGTM then! Parsing a single file is fast!

[Rocketknight1]

cc @Michellehbn merged! Should be in the next release and then we can close out those issues

[Spico197]

@Rocketknight1

Hi Matt, I find the ast-style parsing would introduce an error if a function is ast.Attribute rather than a ast.Name. Here's a minimal reproduction:

import torch.nn as nn


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 10)

    def forward(self, x):
        if x.size() == 1:   # error
            raise ValueError("x.size() == 1")
        return self.linear(x)

When I try to run get_imports("debug_model.py"), it would lead to this error. Here I changed the code a little bit to help debugging. Maybe we should add an additional if-else judgment to check if a func is ast.Name ?

[Rocketknight1]

Hi @Spico197, yes, we noticed that bug too! My fault - I fixed it here: #37245

[Spico197]

Hi @Spico197, yes, we noticed that bug too! My fault - I fixed it here: #37245

Thanks for the prompt reply! Really impressive fast feedback~