Skip to content

Write to Zarr store using h5py-like interface - #28

Merged
magland merged 42 commits into
mainfrom
write
Apr 4, 2024
Merged

magland merged 42 commits into
mainfrom
write

Conversation

@magland

@magland magland commented Mar 25, 2024

Copy link
Copy Markdown
Collaborator

@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@oruebel

oruebel commented Mar 25, 2024

Copy link
Copy Markdown

Just to clarify, if I understand this correctly, lindi.LindiH5pyFile is here backed by a Zarr store, i.e., it is representing a Zarr file as if it where h5py and as such, pynwb.NWBHDF5IO is here actually writing to Zarr. I.e., in some sense this would be in a form a replacement forZarrIO in hdmf_zarr.

store = zarr.DirectoryStore(zarr_dirname)
zarr.group(store=store) # create a root group
with lindi.LindiH5pyFile.from_zarr_store(store, mode='a') as client:
with pynwb.NWBHDF5IO(file=client, mode='r+') as io:
io.write(nwbfile) # type: ignore

@magland

magland commented Mar 25, 2024

Copy link
Copy Markdown
Collaborator Author

Just to clarify, if I understand this correctly, lindi.LindiH5pyFile is here backed by a Zarr store, i.e., it is representing a Zarr file as if it where h5py and as such, pynwb.NWBHDF5IO is here actually writing to Zarr. I.e., in some sense this would be in a form a replacement forZarrIO in hdmf_zarr.

store = zarr.DirectoryStore(zarr_dirname)
zarr.group(store=store) # create a root group
with lindi.LindiH5pyFile.from_zarr_store(store, mode='a') as client:
with pynwb.NWBHDF5IO(file=client, mode='r+') as io:
io.write(nwbfile) # type: ignore

@oruebel Yes that's right.

Comment thread examples/example_edit_nwb.py
Comment thread tests/test_fletcher32.py
Comment thread tests/test_remote_data.py Outdated
Comment thread tests/test_zarr_write.py Outdated
magland and others added 2 commits April 2, 2024 19:21
Co-authored-by: Ryan Ly <rly@lbl.gov>
Co-authored-by: Ryan Ly <rly@lbl.gov>
Comment thread tests/test_copy.py Outdated
Comment thread examples/example_create_zarr_nwb.py Outdated
Comment thread lindi/conversion/attr_conversion.py Outdated
Comment thread lindi/conversion/attr_conversion.py Outdated
Comment thread lindi/conversion/attr_conversion.py Outdated
Comment thread lindi/conversion/nan_inf_ninf.py Outdated
@rly

rly commented Apr 4, 2024

Copy link
Copy Markdown
Contributor

The suggested changes around np.float128/np.complex256 and the change of mode="a" to mode="r+" were necessary to make examples/example_create_zarr_nwb.py run on my machine.

Comment thread lindi/LindiH5pyFile/write/LindiH5pyGroupWrite.py
@rly

rly commented Apr 4, 2024

Copy link
Copy Markdown
Contributor

I did not do heavy testing of this PR, but I read through it all, and as far as I can tell, it makes sense and looks good. Nicely done!

HDMF supports a lot of weird edge cases. I'll see if I can think up more tests to what you already have here.

magland and others added 3 commits April 4, 2024 08:28
Co-authored-by: Ryan Ly <rly@lbl.gov>
Co-authored-by: Ryan Ly <rly@lbl.gov>
@magland

magland commented Apr 4, 2024

Copy link
Copy Markdown
Collaborator Author

I did not do heavy testing of this PR, but I read through it all, and as far as I can tell, it makes sense and looks good. Nicely done!

HDMF supports a lot of weird edge cases. I'll see if I can think up more tests to what you already have here.

Thanks Ryan!

Comment thread lindi/conversion/attr_conversion.py Outdated
@magland
magland merged commit b2c2788 into main Apr 4, 2024
@magland
magland deleted the write branch April 4, 2024 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants