Skip to content

[Refactor][core]Data inflation when using postgres #8142

Description

@narrowizard

What and why to refactor

As a software engineer, I am using devlake to collect developer data since long time ago. Recently we find that the _devlake_subtasks table occurs a data inflation. It used 750MB after a week from we upgrading devlake v1.0, but there were only 1000 records in the table.

Describe the solution you'd like

Solutions from @klesh

  1. Reduce the update rate to _devlake_subtasks table when collect data.
  2. Store progress info in memory firstly and write it to db in a fixed rate.

Related issues

No

Additional context

  • The issue was introduced in v1.0.
  • We are using postgres.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions