Repository navigation
Copying very large repo (30k issues, 33k PRs) #84
Description
I want to keep a local copy of metadata for qgis/QGIS, which is quite large with 30k issues and 33k pull requests. Has anyone used github-to-sqlite for anything like that?
The GitHub API has a rate limit of 5,000 API calls per hours, for authenticated users. And I think it takes one call for each issue - am I right? And github-to-sqlite does not offer only fetching only some subset of the issues, and furthermore does not commit to sqlite before all issues are fetched (so in case a call is interrupted, e.g. by the rate limit, the fetched data is simply discarded).
So I don't see any way that I can actually use github-to-sqlite for this (unless I pay for such privileges to github). Can you confirm?
Sincerely, Thomas
Activity
velle commentedon Jul 28, 2025
I was wrong about that. As far as I can tell from looking at the code, all data is fetched via the REST API (as opposed to the GraphQL API) using pagination, with 100 rows per call. So 33000 issues is only 330 API calls.
Fetching all issues from qgis/QGIS took roughly two hours:
But to my surprise around 62k rows were inserted into the
issuestable, as opposed to the expected 33. I found out that pull requests are also included. If I don't count those, then the numbers match.