Skip to content

Multithreaded NT/NQ to Jelly conversions #240

Description

@Ostrzyciel

When converting NT/NQ to Jelly, the main bottleneck is parsing the input in NT/NQ. This can't really be sped up, it's as fast as it will get.

We could however, run the conversion in parallel. If we split the input into chunks of, say, 100k triples, we could send these chunks to parallel converters. These would have to be either completely independent with checkpointing... or we could pipeline their outputs through a transcoder to recompress the output.

When splitting we can seek by an arbitrary offset in the file (like a few MB) and then look for the next \n. This will require extensive buffering, so memory usage will be significantly higher.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions