issues

The zarr backend uses the fill_value of zarrs .zarray key as if it would be the _FillValue according to CF-Conventions:

https://github.com/pydata/xarray/blob/1a7b285be676d5404a4140fc86e8756de75ee7ac/xarray/backends/zarr.py#L373

I think this interpretation of the fill_value is wrong and creates problems. Here's why:

The zarr v2 spec is still a little vague, but states that fill_value is

A scalar value providing the default value to use for uninitialized portions of the array, or null if no fill_value is to be used.

Accordingly this value should be used to fill all areas of a variable which are not backed by a stored chunk with this value. This is also different from what CF conventions state (emphasis mine):

The scalar attribute with the name _FillValue and of the same type as its variable is recognized by the netCDF library as the value used to pre-fill disk space allocated to the variable. This value is considered to be a special value that indicates undefined or missing data, and is returned when reading values that were not written.

The difference between the two is, that fill_value is only a background value, which just isn't stored as a chunk. But _FillValue is (possibly) a background value and is interpreted as not being valid data. In my opinion, this mix of _FillValue and missing_value could be considered a defect in the CF-Conventions, but probably that's far to late as many depend on this.

Thinking of an example, when storing a density field (i.e. water droplets forming clouds) in a zarr dataset, it might be perfectly valid to set the fill_value to 0 and then store only chunks in regions of the atmosphere where clouds are actually present. In that case, 0 (i.e. no drops) would be a perfectly valid value, which just isn't stored. As most parts of the atmosphere are indeed cloud-free, this may save quite a bunch of storage. Other formats (e.g. OpenVDB) commonly use this trick.

The issue gets worse when looking into the upcoming zarr v3 spec where fill_value is described as:

Provides an element value to use for uninitialised portions of the Zarr array.

If the data type of the Zarr array is Boolean then the value must be the literal false or true. If the data type is one of the integer data types defined in this specification, then the value must be a number with no fraction or exponent part and must be within the range of the data type.

For any data type, if the fill_value is the literal null then the fill value is undefined and the implementation may use any arbitrary value that is consistent with the data type as the fill value.

[...]

Thus for boolean arrays, if the fill_value would be interpreted as a missing value indicator, only (missing, True) or (False, missing) arrays could be represented. A (False, True) array would not be possible. The issue applies similarly for integer types as well.

What happened?

python import numpy as np import xarray as xr ds = xr.Dataset({"a": ("x", [3.], {"_FillValue": np.nan})}) m = {} ds.to_zarr(m) ds.to_zarr(m, append_dim="x") raises ValueError: failed to prevent overwriting existing key _FillValue in attrs. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

What did you expect to happen?

I'd expect this to just work (effectively concatenating the dataset to itself).

Anything else we need to know?

appears also for region writes

The same issue appears for region writes as in: python import numpy as np import dask.array as da import xarray as xr ds = xr.Dataset({"a": ("x", da.array([3.,4.]), {"_FillValue": np.nan})}) m = {} ds.to_zarr(m, compute=False, encoding={"a": {"chunks": (1,)}}) ds.isel(x=slice(0,1)).to_zarr(m, region={"x": slice(0,1)}) raises

ValueError: failed to prevent overwriting existing key _FillValue in attrs. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

there's a workaround

The workaround (deleting the _FillValue in subsequent writes): python m = {} ds.to_zarr(m) del ds.a.attrs["_FillValue"] ds.to_zarr(m, append_dim="x") seems to do the trick.

There are indications that the result might still be broken, but it's not yet clear how to reproduce them (see comments below).

This issue has been split off from #6069

When storing a dask-chunked dataset to zarr, the size of the last chunk in each dimension does not matter, as this single last chunk will be written to any number of zarr chunks, but none of the zarr chunks which are being written to will be accessed by any other dask chunk.

• [x] Closes #6255
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc'ing @rabernat who seems to have worked on this lately.

What happened?

I'm trying to write a dataset which is (dask-) chunked in large chunks into zarr which should be chunked in smaller chunks. The dask chunks are intentionally chosen to be integer multiples of the zarr chunks, such that there will never be two dask chunks which may be written into a single zarr chunk.

When trying to write such a dataset using to_zarr, the following exception appears:

NotImplementedError: Final chunk of Zarr array must be the same size or smaller than the first. Specified Zarr chunk encoding['chunks']=(1,), for variable named 'a' but (2, 2) in the variable's Dask chunks ((2, 2),) are incompatible with this encoding. Consider either rechunking using `chunk()`, deleting or modifying `encoding['chunks']`, or specify `safe_chunks=False`.

What did you expect to happen?

I'd expect the write to "just work".

Minimal Complete Verifiable Example

python import xarray as xr ds = xr.Dataset({"a": ("x", [1, 2, 3, 4])}).chunk({"x": 2}) m = {} ds.to_zarr(m, encoding={"a": {"chunks": (1,)}})

Relevant log output

No response

Anything else we need to know?

I believe that the expected behaviour is according to this design choice: # DESIGN CHOICE: do not allow multiple dask chunks on a single zarr chunk # this avoids the need to get involved in zarr synchronization / locking # From zarr docs: # "If each worker in a parallel computation is writing to a separate # region of the array, and if region boundaries are perfectly aligned # with chunk boundaries, then no synchronization is required."

But I believe that this if-statement is not needed and should be removed. The if-statement compares the size of the last dask-chunk within each dimenstion to the zarr-chunk size. There are three possible cases, which (as far as I understand) should all be just fine: * the dask-chunk is smaller than the zarr chunk: one dask chunk will write into one (smaller, last) zarr chunk * the dask-chunk is equal than the zarr chunk: one dask chunk will write into one zarr chunk * ther dask-chunk is larger than the zarr chunk: one dask chunk will write into multiple zarr chunks. None of these zarr chunks will be touched by any other dask-chunk as all previous dask chunks are aligned to zarr-chunk boundaries.

Note: If that if-statement goes away, this one may go away as well (was introduced in #4312).

Environment

If a dask array is chunked along one dimension but not chunked along another, any manually specified zarr chunk size should be valid, but before this patch, this resulted in an error.

• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8 (only for modified sections)
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

netCDF3 doesn't know unsigned while OPeNDAP doesn't know signed (bytes). Depending on which backend source is used, the original data is stored with the wrong signedness and needs to be decoded based on the _Unsigned attribute. While the netCDF3 variant is already implemented, this commit adds the symmetric case covering OPeNDAP.

• [x] Closes #4954
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

netCDF3 only knows signed bytes, but there's a convention of adding an attribute _Unsigned=True to the variable to be able to store unsigned bytes non the less. This convention is handled at this place by xarray.

OPeNDAP only knows unsigned bytes, but there's a hack which is used by the thredds server and the netCDF-c library of adding an attribute _Unsigned=False to the variable to be able to store signed bytes non the less. This hack is not handled by xarray, but maybe should be handled symmetrically at the same place (i.e. if .kind == "u" and unsigned == False).

As descibed in the "hack", netCDF-c handles this internally, but pydap doesn't. This is why the engine="netcdf4" variant returns (correctly according to the hack) negative values and the engine="pydap" variant doesn't. However, as xarray returns a warning at exactly the location referenced above, I think that this is the place where it should be fixed.

If you agree, I could prepare a PR to implement the fix.

```python In [1]: import xarray as xr

In [2]: xr.open_dataset("https://observations.ipsl.fr/thredds/dodsC/EUREC4A/PRODUCTS/testdata/netcdf_testfiles/test_NC_BYTE_neg.nc", engine="netcdf4") Out[2]: <xarray.Dataset> Dimensions: (test: 7) Coordinates: * test (test) float32 -128.0 -1.0 0.0 1.0 2.0 nan 127.0 Data variables: empty

In [3]: xr.open_dataset("https://observations.ipsl.fr/thredds/dodsC/EUREC4A/PRODUCTS/testdata/netcdf_testfiles/test_NC_BYTE_neg.nc", engine="pydap") /usr/local/lib/python3.9/site-packages/xarray/conventions.py:492: SerializationWarning: variable 'test' has _Unsigned attribute but is not of integer type. Ignoring attribute. new_vars[k] = decode_cf_variable( Out[3]: <xarray.Dataset> Dimensions: (test: 7) Coordinates: * test (test) float32 128.0 255.0 0.0 1.0 2.0 nan 127.0 Data variables: empty ```