Skip to content

Speed up create_batch_from_map #338

Description

@Dandandan

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently .to_array() is called on each scalar value that creates a single row array, which is very inefficient:

  • There is overhead for generating the array structure for each row.
  • The single-row arrays are concatenated afterwards at the end, which is slow and would be unnecessary if they are created with more values at once
  • Intermediate Vecs are generated, causing more memory usage / allocations / fragmentation.

I expect this should speed up some db-benchmark queries (group by queries with smaller groups) considerably and may decrease memory usage by quite a bit.

Describe the solution you'd like
Iterate over the values and emit arrays of batch_size elements at once.
Or as a first step just do it for all of the values (as is the case currently) - and emit smaller batches in a later PR.

To do it with batch_size there should be some state and/or the groups should be removed from the map.

Describe alternatives you've considered
n/a

Additional context
n/a

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions