NodeTypeValidationError when opening with zarr.open_zarr but xarray.open_zarr suceeds

[rbavery]

I'm trying to investigate/learn how dask chunks get translated to Zarr v3 shards (or chunks?). So I'm working with an example where I save a Dask array to a zarr and then try to open it with zarr-python to inspect the shard size. From this comment it sounds like dask chunks get translated to shards: pydata/xarray#9938 (reply in thread)

import xarray as xr import dask.array as da import numpy as np import zarr ar = da.random.random((4096, 4096), chunks=(64, 64)) dset = xr.DataArray(ar, coords={"x": range(4096), "y": range(4096)}).to_dataset( name="variables" ) encoding = {"variables": {"shards": (2048, 2048)}} dset["variables"].encoding = encoding dset.to_zarr("test.zarr", zarr_format=3, mode="w") rt = xr.open_zarr("test.zarr") # has a chunk size same as dask chunk size, not the shard size import zarr zarr.open_array(store="test.zarr") #fails

The error is

zarr.open_array(store="test.zarr") 

--------------------------------------------------------------------------- NodeTypeValidationError Traceback (most recent call last) Cell In[23], line 2 1 import zarr ----> 2 zarr.open_array(store="test.zarr") File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/api/synchronous.py:1289, in open_array(store, zarr_version, path, storage_options, **kwargs) 1259 def open_array( 1260 store: StoreLike | None = None, 1261 *, (...) 1265 **kwargs: Any, 1266 ) -> Array: 1267 """Open an array using file-mode-like semantics. 1268 1269 Parameters (...) 1286 The opened array. 1287 """ 1288 return Array( -> 1289 sync( 1290 async_api.open_array( 1291 store=store, 1292 zarr_version=zarr_version, 1293 path=path, 1294 storage_options=storage_options, 1295 **kwargs, 1296 ) 1297 ) 1298 ) File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/sync.py:163, in sync(coro, loop, timeout) 160 return_result = next(iter(finished)).result() 162 if isinstance(return_result, BaseException): --> 163 raise return_result 164 else: 165 return return_result File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/sync.py:119, in _runner(coro) 114 """ 115 Await a coroutine and return the result of running it. If awaiting the coroutine raises an 116 exception, the exception will be returned. 117 """ 118 try: --> 119 return await coro 120 except Exception as ex: 121 return ex File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:1261, in open_array(store, zarr_version, zarr_format, path, storage_options, **kwargs) 1258 _warn_write_empty_chunks_kwarg() 1260 try: -> 1261 return await AsyncArray.open(store_path, zarr_format=zarr_format) 1262 except FileNotFoundError: 1263 if not store_path.read_only and mode in _CREATE_MODES: File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/array.py:898, in AsyncArray.open(cls, store, zarr_format) 875 """ 876 Async method to open an existing Zarr array from a given store. 877 (...) 895 <AsyncArray memory://... shape=(100, 100) dtype=int32> 896 """ 897 store_path = await make_store_path(store) --> 898 metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format) 899 # TODO: remove this cast when we have better type hints 900 _metadata_dict = cast(ArrayV3MetadataDict, metadata_dict) File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/array.py:210, in get_array_metadata(store_path, zarr_format) 207 assert zarr_json_bytes is not None 208 metadata_dict = json.loads(zarr_json_bytes.to_bytes()) --> 210 parse_node_type_array(metadata_dict.get("node_type")) 212 return metadata_dict File ~/wherobots-executors-demo/.venv/lib/python3.12/site-packages/zarr/core/metadata/v3.py:64, in parse_node_type_array(data) 62 if data == "array": 63 return "array" ---> 64 raise NodeTypeValidationError("node_type", "array", data) NodeTypeValidationError: Invalid value for 'node_type'. Expected 'array'. Got 'group'. 

any pointers on how to open this test.zarr with zarr-python? am I missing something about how I should save or open the zarr store?

[rbavery]

I was using open_array instead of open woopsidaisy

[ljstrnadiii]

Looks like sharding is not used here and must be declared another way.

import zarr x = zarr.open_group("test.zarr") np.array(x['variables']) x['variables'].shards -> None