Skip to content

Add incremental sync support to starred #90

Open

Description

@tekumara

Is your feature request related to a problem?

Running github-to-sqlite starred fetches the user's entire star list on every run, even when most or all of those stars are already in the database. This makes repeated syncs slow and uses unnecessary GitHub API requests for users with large star lists.

Describe the feature

Add an opt-in incremental mode, such as --incremental, that uses the database's existing stars for the specified user to avoid fetching the full history.

For example:

github-to-sqlite starred --incremental ~/.local/state/github-to-sqlite/github.db tekumara

On an empty database, incremental mode should do a full initial import. On later runs, it should fetch recent stars and stop once it has reached stars already stored in the database. Existing full-fetch behavior should remain available.

Additional context

GitHub's starred-repositories endpoint supports sorting by when a repository was starred, but does not provide a since cursor. Incremental mode would therefore need to paginate through recent results and use the database to determine when it has caught up. It should account for multiple stars sharing a timestamp so that it doesn't skip any.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions