Skip to content

Add metadata to dtype when kind is 'O' - #47

Merged
magland merged 4 commits into
mainfrom
dtype-meta-vlen
Apr 19, 2024
Merged

magland merged 4 commits into
mainfrom
dtype-meta-vlen

Conversation

@magland

@magland magland commented Apr 16, 2024 •

Copy link
Copy Markdown
Collaborator

In order to get pynwb validation to pass I needed to make the tweak of this PR. It seems that h5py uses a vlen field in metadata for a numpy dtype, and pynwb inspects that. I don't understand exactly what this is about. But when I set it to bytes in this case, the validation passes.

Here's a test validation script where things are passing with this PR

import pynwb
from pynwb import CORE_NAMESPACE
from pynwb.validate import ValidatorMap
import h5py
import remfile
import lindi


def test_validate(io: pynwb.NWBHDF5IO):
    namespace = CORE_NAMESPACE
    validator = ValidatorMap(namespace=io.manager.namespace_catalog.get_namespace(name=namespace))
    builder = io.read_builder()
    validation_errors = validator.validate(builder=builder)
    if len(validation_errors) == 0:
        print('Validation passed!')
    else:
        print('Validation failed!')
        for ve in validation_errors:
            print(ve)


if __name__ == "__main__":
    # https://neurosift.app/?p=/nwb&dandisetId=000409&dandisetVersion=draft&url=https://api.dandiarchive.org/api/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/download/

    print('')
    print('Validating with HDF5 URL...')
    h5_url = 'https://api.dandiarchive.org/api/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/download/'
    remf = remfile.File(h5_url)
    h5f = h5py.File(remf, mode='r')
    with pynwb.NWBHDF5IO(file=h5f, mode='r', load_namespaces=True) as io:
        test_validate(io)

    print('')
    print('Validating with LINDI URL...')
    lindi_url = 'https://lindi.neurosift.org/dandi/dandisets/000409/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/zarr.json'
    ff = lindi.LindiH5pyFile.from_reference_file_system(lindi_url)
    with pynwb.NWBHDF5IO(file=ff, mode='r', load_namespaces=True) as io:
        test_validate(io)

@magland
magland requested a review from rly April 16, 2024 16:03
Base automatically changed from rfs-json to main April 16, 2024 17:15
@codecov-commenter

codecov-commenter commented Apr 16, 2024 •

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.34%. Comparing base (3c4562f) to head (d269450).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #47   +/-   ##
=======================================
  Coverage   82.33%   82.34%           
=======================================
  Files          28       28           
  Lines        1970     1971    +1     
=======================================
+ Hits         1622     1623    +1     
  Misses        348      348           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rly rly left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Let's try to validate all NWB files on DANDI and see if this will work. We should look more closely into why the logic is what it is in PyNWB...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants