Add docs around how Sourcebot handles rate-limiting against code hosts

[brendan-kellam]

See:

• https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#checking-the-status-of-your-rate-limit

Discussed in #316

^{Originally posted by dsculptor May 27, 2025}

Excessive PAT token usage can suspend your github account!

One of my colleagues had her github account suspended. The most likely reason we think is because of excessive API usage via her PAT token in our sourcebot instance. Our self-hosted instance had 15+ public repos indexed (mainly AI SDKs).

Although, github has not shared any details on the exact reasons for suspension; everything else about her account / activity had been normal since forever.

💡 Ideas

• Would request soucebot team to put reasonable warnings e.g link to section H in the docs for using PATs.
• Suggest workarounds.
• Rate limiting / controls on pull frequency can be added as well.

Github Terms Section H

[msukkari]

Potentially related to this issue for GitLab #422