github-to-sqlite should handle rate limits better #51

Activity

1. simonw

   added

   enhancementNew feature or requestNew feature or request

   on Sep 17, 2020
2. simonw

   changed the title [-]github-to-sqlite get should follow rate limits[/-] [+]github-to-sqlite should handle rate limits better[/+] on Nov 30, 2020
3. simonw commented on Nov 30, 2020

   

   simonw

   on Nov 30, 2020

   CollaboratorAuthor

   More actions

   This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true

     File "/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite", line 33, in <module>
       sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')())
     File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
       return self.main(*args, **kwargs)
     File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
       rv = self.invoke(ctx)
     File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
       return ctx.invoke(self.callback, **ctx.params)
     File "/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
       return callback(*args, **kwargs)
     File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py", line 142, in issue_comments
       for comment in utils.fetch_issue_comments(repo, token, issue):
     File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 380, in fetch_issue_comments
       for comments in paginate(url, headers):
     File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 472, in paginate
       raise GitHubError.from_response(response)
   github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
   Error: Process completed with exit code 1.
   
4. simonw

   added a commit that references this issue on Nov 30, 2020

   Sleep longer, hopefully help workaround #51

   fce9738
5. daniel-butler commented on Jan 30, 2021

   

   daniel-butler

   on Jan 30, 2021

   Last edited by daniel-butler

   Contributor

   More actions

   I don't have much experience with github's rate limiting. In my day job we use the tenacity library to handle http errors we get.
6. hydrosquall commented on Aug 9, 2022

   

   hydrosquall

   on Aug 9, 2022

   More actions

   I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits.

   I've had decent success with the Airbyte github extractor (aside from one data quality issue airbytehq/airbyte#15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive.

   https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

   Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children PyGithub/PyGithub#1989 , but they weren't working as expected.
7. chapmanjacobd commented on Oct 14, 2022

   

   chapmanjacobd

   on Oct 14, 2022

   More actions

   also, it says that authenticated requests have a much higher "rate limit". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a "rate limit" (although I guess that is semantic equivalence)

   You would want to use x-ratelimit-reset

   time.sleep(r['x-ratelimit-reset'] + 1 - time.time())
   

   But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only github-to-sqlite get is using the --auth= CLI flag