Repository navigation

Add S3 (with Minio) netCDF open/read and processing and exploratory tests - #89

Merged

valeriupredoi merged 61 commits into

mainfrom

real_world_s3_tests

Jun 15, 2023

Commits on Jun 9, 2023

ignore s3 exploratory tests in normal runs
valeriupredoi
committed
9f7783d
add s3 test run GA workflow
valeriupredoi
committed
8069cef
change GA test feature branch
valeriupredoi
committed
d854997
add s3 exploratory test repo
valeriupredoi
committed
ea2cf98
correct path to minio scripts
valeriupredoi
committed
e84f21b
run s3 AS container
valeriupredoi
committed
fc5766a
run Active component too
valeriupredoi
committed
37ca643
temp deactivate check on physical file in active class
valeriupredoi
committed
cccc8d2
shove in use s3
valeriupredoi
committed
d0b9805
run with s3 file
valeriupredoi
committed
1c10fb8
try load netCDF from s3
valeriupredoi
committed
405b5eb
pass correct s3 uri
valeriupredoi
committed
60e1cb3
of course I forgot import
valeriupredoi
committed
bf280ab
try without s3 prefit for 3 file path
valeriupredoi
committed
68a7777
possible correct s3 path
valeriupredoi
committed
0fa23f2
possible correct s3 path
valeriupredoi
committed
66dce92
try with context manager
valeriupredoi
committed
de1939d
use s3fs directly
valeriupredoi
committed
6c78fce
add client server arg
valeriupredoi
committed
fee58b3
try without localhost
valeriupredoi
committed
a1d3ac0
add xarray for test
valeriupredoi
committed
f076c6c
add xarray for test
valeriupredoi
committed
1efe1e7
use xarray for test
valeriupredoi
committed
6981740
wrong engine
valeriupredoi
committed
482ffb5
try mfdataset
valeriupredoi
committed
f17d08f
forget about xarray
valeriupredoi
committed
ab0674b
forget about xarray
valeriupredoi
committed
77bfef9
hat does that s3 file like look like
valeriupredoi
committed
5b4f64e
how does that s3 file look like
valeriupredoi
committed
6003b74

Commits on Jun 12, 2023

asdd h5netcdf to env
valeriupredoi
committed
71051e4
asdd h5netcdf to env
valeriupredoi
committed
624c849
try h5netcdf layer
valeriupredoi
committed
30c33d3
dont call netcdf4
valeriupredoi
committed
514b1b6
skeip filers xtraction
valeriupredoi
committed
9dcb03d
fix test
valeriupredoi
committed
1e0ee8d
tweak call
valeriupredoi
committed
83e3e85
switch to s3fs
valeriupredoi
committed
04ee231
give it a slash free name
valeriupredoi
committed
b8f2761
bleghh
valeriupredoi
committed
4af7c4c
pytested test
valeriupredoi
committed
c7b4fef
refine a bit
valeriupredoi
committed
f2a1478
corrected check for file
valeriupredoi
committed
1a43bf1
posix to str paths
valeriupredoi
committed
20a806e
made an s3 loader function
valeriupredoi
committed
4b5c2d8
blegh colon dude
valeriupredoi
committed
770a9d0
return something too
valeriupredoi
committed
621f85d
load s3 connection params from config
valeriupredoi
committed
945b32f

Commits on Jun 14, 2023

add test data file
valeriupredoi
committed
0d3492f
add netCDF real file test
valeriupredoi
committed
ea91080
missing import
valeriupredoi
committed
d1fa7fb
pass correct var
valeriupredoi
committed
5272d7d
fail to visualize results
valeriupredoi
committed
4c5b311
fixed test
valeriupredoi
committed
5ae2666
correct version of test
valeriupredoi
committed
b0dc572
add to local s3 unit test
valeriupredoi
committed
4ef10cb
add mention in test
valeriupredoi
committed
2fb70fd
correct file passed eh
valeriupredoi
committed
273f034
simple raiser test
valeriupredoi
committed
8bf1454

Commits on Jun 15, 2023

remove unnecessary test
valeriupredoi
committed
04f29a4
start a unit test for storage types
valeriupredoi
committed
5c7f86b
Merge branch 'main' into real_world_s3_tests
valeriupredoi
committed
f401deb