issues

What happened?

Several processes read and write to a xarray stored in .zarr format, on a network. The write operations write to existing regions. These regions are not aligned to chunks, therefore I use a ProcessSynchronizer. The ProcessSynchronizer points to a local folder on SSD, separate from the actual stored array.

After several hundreds of read/write I get permission errors like below. So far I have failed to reproduce the error with a MCVE.

The file 0 that gave a permission error is the chunk of coordinates of a certain dimension in the dimension folder dim_yyy: dim_yyy |-- .zarray |-- .zattrs `-- 0

What did you expect to happen?

No permission error.

Minimal Complete Verifiable Example

```Python I have failed so far to reproduce the error with an MVCE. Here my attempt.

from pathlib import Path

import dask.array as da import pandas as pd import xarray as xr from dask.distributed import Client from zarr.sync import ProcessSynchronizer

if name == "main": path_store = Path(aaa) path_synchronizer = Path(bbb) # must exist, and not same location as store

# create and save a datset to zarr
s0, s1, s2 = 10, 10, 10
temperature = da.random.random((s0, s1, s2), chunks=[s0, s1, s2])
precipitation = da.random.random((s0, s1, s2), chunks=[s0, s1, s2])
lon = da.random.random((s0, s1))
lat = da.random.random((s0, s1))
time = pd.date_range("2014-09-06", periods=s2)
reference_time = pd.Timestamp("2014-09-05")
ds = xr.Dataset(
    data_vars=dict(
        temperature=(["x", "y", "time"], temperature),
        precipitation=(["x", "y", "time"], precipitation),
    ),
    coords=dict(
        lon=(["x", "y"], lon),
        lat=(["x", "y"], lat),
        time=time,
        reference_time=reference_time,
    ),
    attrs=dict(description="Weather related data."),
)
print(f"{ds=}")
ds.to_zarr(path_store, mode="w")

def read_write(path_store: Path):
    """lazily opens the dataset, then writes into a region. Comment/uncomment to use synchronizer"""
    synchronizer = ProcessSynchronizer(path_synchronizer)
    for b in range(100):
        # open the saved dataset
        # xr.open_zarr(path_store, synchronizer=synchronizer)
        ds = xr.open_zarr(path_store)

        # process a region
        dst = (
            ds.temperature.isel(x=slice(0, 5), y=slice(0, 5), time=slice(0, 5))
            .to_dataset()
            .load()
        )
        dst["temperature"] = -dst["temperature"]
        dst = dst.drop_vars(["time", "reference_time"])

        # save the region to the zarr store
        dst.to_zarr(
            path_store,
            region={
                "x": slice(0, 5),
                "y": slice(0, 5),
                "time": slice(0, 5),
            },
            # synchronizer=synchronizer,
        )

# independent processes that perform read and write operations
with Client(processes=True) as client:
    futures = [client.submit(read_write, path_store) for a in range(1000)]
    client.gather(futures)

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python return xr.open_zarr(path, synchronizer=synchronizer) File "C:\anaconda3\lib\site-packages\xarray\backends\zarr.py", line 787, in open_zarr ds = open_dataset( File "C:\anaconda3\lib\site-packages\xarray\backends\api.py", line 539, in open_dataset backend_ds = backend.open_dataset( File "C:\anaconda3\lib\site-packages\xarray\backends\zarr.py", line 862, in open_dataset ds = store_entrypoint.open_dataset( File "C:\anaconda3\lib\site-packages\xarray\backends\store.py", line 43, in open_dataset ds = Dataset(vars, attrs=attrs) File "C:\anaconda3\lib\site-packages\xarray\core\dataset.py", line 604, in __init__ variables, coord_names, dims, indexes, _ = merge_data_and_coords( File "C:\anaconda3\lib\site-packages\xarray\core\merge.py", line 575, in merge_data_and_coords return merge_core( File "C:\anaconda3\lib\site-packages\xarray\core\merge.py", line 755, in merge_core collected = collect_variables_and_indexes(aligned, indexes=indexes) File "C:\anaconda3\lib\site-packages\xarray\core\merge.py", line 365, in collect_variables_and_indexes variable = as_variable(variable, name=name) File "C:\anaconda3\lib\site-packages\xarray\core\variable.py", line 168, in as_variable obj = obj.to_index_variable() File "C:\anaconda3\lib\site-packages\xarray\core\variable.py", line 624, in to_index_variable return IndexVariable( File "C:\anaconda3\lib\site-packages\xarray\core\variable.py", line 2844, in __init__ self._data = PandasIndexingAdapter(self._data) File "C:\anaconda3\lib\site-packages\xarray\core\indexing.py", line 1420, in __init__ self.array = safe_cast_to_index(array) File "C:\anaconda3\lib\site-packages\xarray\core\indexes.py", line 177, in safe_cast_to_index index = pd.Index(np.asarray(array), **kwargs) File "C:\anaconda3\lib\site-packages\xarray\core\indexing.py", line 524, in __array__ return np.asarray(array[self.key], dtype=None) File "C:\anaconda3\lib\site-packages\xarray\backends\zarr.py", line 68, in __getitem__ return array[key.tuple] File "C:\anaconda3\lib\site-packages\zarr\core.py", line 821, in __getitem__ result = self.get_basic_selection(pure_selection, fields=fields) File "C:\anaconda3\lib\site-packages\zarr\core.py", line 947, in get_basic_selection return self._get_basic_selection_nd(selection=selection, out=out, File "C:\anaconda3\lib\site-packages\zarr\core.py", line 990, in _get_basic_selection_nd return self._get_selection(indexer=indexer, out=out, fields=fields) File "C:\anaconda3\lib\site-packages\zarr\core.py", line 1285, in _get_selection self._chunk_getitem(chunk_coords, chunk_selection, out, out_selection, File "C:\anaconda3\lib\site-packages\zarr\core.py", line 1994, in _chunk_getitem cdata = self.chunk_store[ckey] File "C:\anaconda3\lib\site-packages\zarr\storage.py", line 1085, in __getitem__ return self._fromfile(filepath) File "C:\anaconda3\lib\site-packages\zarr\storage.py", line 1059, in _fromfile with open(fn, 'rb') as f: PermissionError: [Errno 13] Permission denied: 'xxx.zarr\\dim_yyy/0'

Anything else we need to know?

No response

Environment

What happened?

Same example as in DataArray.idxmax but instead we look at the "y" dimension. The starting "y" coordinates are of type int: [-1,0,1]

The return values of argmax are of type int64: good. The return values of idxmax are of type float64: bad.

What did you expect to happen?

If no fillna operation must occur, then the return values of idxmax should be the same type as from the input.

Else, the return type might change to a new type depending on the type of the filled value.

Minimal Complete Verifiable Example

Python array = xr.DataArray( [ [2.0, 1.0, 2.0, 0.0, -2.0], [-4.0, np.NaN, 2.0, np.NaN, -2.0], [np.NaN, np.NaN, 1.0, np.NaN, np.NaN], ], dims=["y", "x"], coords={"y": [-1, 0, 1], "x": np.arange(5.0) ** 2}, ) print(array.argmax(dim="y").dtype) print(array.idxmax(dim="y").dtype)

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python In [41]: print(array.argmax(dim="y").dtype) int64

In [42]: print(array.idxmax(dim="y").dtype) float64 ```

Anything else we need to know?

Suggestions: - change these two lines: if skipna or (skipna is None and array.dtype.kind in na_dtypes): # Put the NaN values back in after removing them into if (skipna or (skipna is None and array.dtype.kind in na_dtypes)) and allna.any(): # Put the NaN values back in after removing them, if any

• or maybe instead, it is a bug from DataArray.where: this res = res.where(~allna, fill_value) should not change the array type if not allna.any()? Actually, it is a known limitation of where: #3570

Environment

What happened?

I used DataArray.transpose with transpose_coords=True to change the coords order from startings_dims = "dim_0", "dim_1", "dim_2" to reordered_dims = "dim_2", "dim_1", "dim_0".

The order of dims was correctly transposed but the order of coords remained unchanged.

What did you expect to happen?

I expected the transposed coords to be in the new order:

reordered_dims = "dim_2", "dim_1", "dim_0"

Minimal Complete Verifiable Example

```Python import numpy as np import pandas as pd import xarray as xr

np.random.seed(0) temperature = np.random.randn(4, 4, 3) dim_0_values = [1, 2, 3, 4] dim_1_values = [5, 6, 7, 8] dim_2_values = pd.date_range("2014-09-06", periods=3) starting_dims = "dim_0", "dim_1", "dim_2"

da = xr.DataArray( data=temperature, dims=starting_dims, coords=dict( dim_0=dim_0_values, dim_1=dim_1_values, dim_2=dim_2_values, ), attrs=dict( description="Ambient temperature.", units="degC", ), )

print(f"{da.dims=}") print(f"{da.coords.keys()=}")

reordered_dims = "dim_2", "dim_1", "dim_0" print(f"{da.transpose(reordered_dims).dims=}") print(f"{da.transpose(reordered_dims).coords.keys()=}") print(f"{da.transpose(*reordered_dims, transpose_coords=True).coords.keys()=}") ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python da.dims=('dim_0', 'dim_1', 'dim_2') da.coords.keys()=KeysView(Coordinates: * dim_0 (dim_0) int32 1 2 3 4 * dim_1 (dim_1) int32 5 6 7 8 * dim_2 (dim_2) datetime64[ns] 2014-09-06 2014-09-07 2014-09-08) da.transpose(*reordered_dims).dims=('dim_2', 'dim_1', 'dim_0') da.transpose(*reordered_dims).coords.keys()=KeysView(Coordinates: * dim_0 (dim_0) int32 1 2 3 4 * dim_1 (dim_1) int32 5 6 7 8 * dim_2 (dim_2) datetime64[ns] 2014-09-06 2014-09-07 2014-09-08) da.transpose(*reordered_dims, transpose_coords=True).coords.keys()=KeysView(Coordinates: * dim_0 (dim_0) int32 1 2 3 4 * dim_1 (dim_1) int32 5 6 7 8 * dim_2 (dim_2) datetime64[ns] 2014-09-06 2014-09-07 2014-09-08)

Anything else we need to know?

No response

Environment

What is your issue?

line 23 An ellipsis (...) can be use -> An ellipsis (...) can be used