MCVE Code Sample

```python import datetime

import xarray as xr

date = datetime.datetime(2019, 1, 1, 1, 10)

Reading in 2 min time stepped MRMS data

ds = xr.open_rasterio(dir_path) ds.name = "mrms" ds["time"] = date ds = ds.expand_dims("time") ds = ds.to_dataset()

ds.to_zarr("fin_zarr", compute=False, mode="w-")

date = datetime.datetime(2019, 1, 1, 1, 12)

Reading in 2 min time stepped MRMS data

This can be the same file since we are adding time manually

ds = xr.open_rasterio(dir_path) ds.name = "mrms" ds["time"] = date ds = ds.expand_dims("time") ds = ds.to_dataset()

ds.to_zarr("fin_zarr", compute=False, mode="a", append_dim="time") ```

Expected Output

<xarray.Dataset> Dimensions: (band: 1, time: 1, x: 7000, y: 3500) Coordinates: * band (band) int64 1 * y (y) float64 55.0 54.99 54.98 54.97 ... 20.04 20.03 20.02 20.01 * x (x) float64 -130.0 -130.0 -130.0 -130.0 ... -60.03 -60.02 -60.01 * time (time) datetime64[ns] 2019-01-01T01:10:00 Data variables: mrms (time, band, y, x) uint8 255 255 255 255 255 ... 255 255 255 255 appended by this in a ds.to_zarr() <xarray.Dataset> Dimensions: (band: 1, time: 1, x: 7000, y: 3500) Coordinates: * band (band) int64 1 * y (y) float64 55.0 54.99 54.98 54.97 ... 20.04 20.03 20.02 20.01 * x (x) float64 -130.0 -130.0 -130.0 -130.0 ... -60.03 -60.02 -60.01 * time (time) datetime64[ns] 2019-01-01T01:12:00 Data variables: mrms (time, band, y, x) uint8 255 255 255 255 255 ... 255 255 255 255 should look like below <xarray.Dataset> Dimensions: (band: 1, time: 2, x: 7000, y: 3500) Coordinates: * band (band) int64 1 * time (time) datetime64[ns] 2019-01-01T01:10:00 2019-01-01T01:12:00 * x (x) float64 -130.0 -130.0 -130.0 -130.0 ... -60.03 -60.02 -60.01 * y (y) float64 55.0 54.99 54.98 54.97 ... 20.04 20.03 20.02 20.01 Data variables: mrms (time, band, y, x) uint8 dask.array<shape=(2, 1, 3500, 7000), chunksize=(1, 1, 438, 1750)>

Problem Description

The outout looks like this: <xarray.Dataset> Dimensions: (band: 1, time: 2, x: 7000, y: 3500) Coordinates: * band (band) int64 1 * time (time) datetime64[ns] 2019-01-01T01:10:00 2019-01-01T01:10:00 * x (x) float64 -130.0 -130.0 -130.0 -130.0 ... -60.03 -60.02 -60.01 * y (y) float64 55.0 54.99 54.98 54.97 ... 20.04 20.03 20.02 20.01 Data variables: mrms (time, band, y, x) uint8 dask.array<shape=(2, 1, 3500, 7000), chunksize=(1, 1, 438, 1750)>

Where the minutes are repeated for the whole hour until a new hour is appended. It seems to not be handling minutes correctly.

Output of xr.show_versions()

Description

If you create a Zarr with a single timestep and then append to the time dimension of that Zarr in subsequent writes then the appended timestamps are likely to be wrong. This only seems to happen if the time dimension is datetime64.

Minimal Complete Verifiable Example

Create a really simple Dataset:

python times = pd.date_range("2000-01-01 00:35", periods=8, freq="6H") da = xr.DataArray(coords=[times], dims=["time"]) ds = da.to_dataset(name="foo")

Write just the first timestep to a new Zarr store:

python ZARR_PATH = "test.zarr" ds.isel(time=[0]).to_zarr(ZARR_PATH, mode="w")

So far, so good!

Now things get weird... let's append the remainder of ds to the Zarr store:

python ds.isel(time=slice(1, None)).to_zarr(ZARR_PATH, append_dim="time")

This throws a warning, which is probably relevant:

/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/xarray/core/dataset.py:2037: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs return to_zarr(

What happened

Let's load the Zarr and print the contents on the time coord:

python ds_loaded = xr.open_dataset(ZARR_PATH, engine="zarr") print(ds_loaded.time) <xarray.DataArray 'time' (time: 8)> array(['2000-01-01T00:35', '2000-01-01T00:35', '2000-01-01T00:35', '2000-01-02T00:35', '2000-01-02T00:35', '2000-01-02T00:35', '2000-01-03T00:35', '2000-01-03T00:35'], dtype='datetime64[ns]') Coordinates: * time (time) datetime64[ns] 2000-01-01T00:35:00 ... 2000-01-03T00:35:00

(I've removed the seconds and milliseconds to make it a bit easier to read)

The first and fifth time coords (2000-01-01T00:35 and 2000-01-02T00:35) are correct. None of the others are correct!

The encoding is not appropriate (see #3942)... notice that the units is days since..., which clearly can't represent sub-day resolution:

python print(ds_loaded.time.encoding) {'chunks': (1,), 'preferred_chunks': {'time': 1}, 'compressor': Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0), 'filters': None, 'units': 'days since 2000-01-01 00:35:00', 'calendar': 'proleptic_gregorian', 'dtype': dtype('int64')}

What you expected to happen

The correct time coords are: python print(ds.time) <xarray.DataArray 'time' (time: 8)> array(['2000-01-01T00:35', '2000-01-01T06:35', '2000-01-01T12:35', '2000-01-01T18:35', '2000-01-02T00:35', '2000-01-02T06:35', '2000-01-02T12:35', '2000-01-02T18:35'], dtype='datetime64[ns]') Coordinates: * time (time) datetime64[ns] 2000-01-01T00:35:00 ... 2000-01-02T18:35:00

Anything else we need to know?

There are three workarounds that I'm aware of:

1) When first creating the Zarr, write two or more timesteps into the Zarr. Then you can append any number of timesteps to the Zarr and everything works fine. 2) Convert the time coords to Unix epoch, represented as ints. 3) Manually set the encoding before the first write (as suggested in https://github.com/pydata/xarray/issues/3942#issuecomment-610444090). For example:

python ds.isel(time=[0]).to_zarr( ZARR_PATH, mode="w", encoding={ 'time': { 'units': 'seconds since 1970-01-01' } } )

Related issues

It's possible that the root cause of this issue is #3942.

And I think #3379 is another symptom of this issue.

Environment

• [x] towards #8844

In the tests, we've been calling np.cross with vectors of 2 or 3 dimensions, numpy>=2 will deprecate 2D vectors (plus, we're now raising on warnings). Thus, we 0-pad the inputs before generating the expected result (which generally should not change the outcome of the tests).

For a later PR: add tests to check if xr.cross works if more than a single dimension is present, and pre-compute the expected result. Also, for property-based testing: the cross-product of two vectors is perpendicular to both input vectors (use the dot product to check that), and its length (l2-norm) is the product of the lengths of the input vectors.

• [ ] Towards #8856
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Per https://github.com/pydata/xarray/issues/8844#issuecomment-2089427222, for the time being this marks test_use_cftime_false_standard_calendar_in_range as an expected failure under NumPy 2. Hopefully we'll be able to fix the upstream issue in pandas eventually.

I considered wedging this into core/ops.py, but it didn't look like it fit there.

This is a basic lift and shift from datatree_/ops.py to core/datatree_ops.py

I did fix the document addendum injection and added a couple of tests.

• [x] Contributes to migration step for miscellaneous modules in Track merging datatree into xarray #8572
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

- [x] towards #8844

• replace numpy.core.finfo with numpy.finfo
• add dtype and copy parameters to all definitions of __array__

In trying to install packages that are compatible with numpy>=2 I added several projects that are built in CI without build isolation (so that they will be built with the nightly version of numpy). That was a temporary workaround, so we should start thinking about cleaning this up.

As it seems numcodecs is now compatible (or uses less of numpy in compiled code, not sure), this is an attempt to see if CI works if we use the version from conda-forge.

bottleneck and cftime now build against numpy>=2.0.0rc1, so we can stop building them without build isolation.

Don't just xfail, and not only on windows, since it can crash the worker

The example in the doc-string of the Dataset class prior to this commit uses an example array whose size is 2 x 2 x 3 with the first two dimensions labeled "x" and "y" and the final dimension labeled "time". This was confusing due to the fact that "x" and "y" are just arbitrary names for these axes and that no reason is given for the data to be organized in a 2x2x3 array instead of a 2x2 matrix. This commit clarifies the example.

Additionally, this PR contains updates to the documentation, specifically the user-guide/data-structures.rst file; the updates bring the documentation examples into alignment with the doc-string change. Unfortunately, I wasn't able to build the documentation, so this will need to be checked. (I followed the instructions here, but despite cfgrib working fine, I got an error about how it wasn't a valid engine.)

See issue #8970 for more information.

• [X] Closes #8970

What is your issue?

The example code in the documentation for the Dataset class (e.g., here) is probably clear to those who study Earth and Atmospheric Sciences, but it makes no sense to me. Here is the code:

```python np.random.seed(0) temperature = 15 + 8 * np.random.randn(2, 2, 3) precipitation = 10 * np.random.rand(2, 2, 3) lon = [[-99.83, -99.32], [-99.79, -99.23]] lat = [[42.25, 42.21], [42.63, 42.59]] time = pd.date_range("2014-09-06", periods=3) reference_time = pd.Timestamp("2014-09-05")

ds = xr.Dataset( data_vars=dict( temperature=(["x", "y", "time"], temperature), precipitation=(["x", "y", "time"], precipitation), ), coords=dict( lon=(["x", "y"], lon), lat=(["x", "y"], lat), time=time, reference_time=reference_time, ), attrs=dict(description="Weather related data."), ) ```

To be clear, I understand each individual line of code, but I don't understand why there is both a latitude/longitude and an x/y in this example or how they are supposed to be related to each other (and there do not appear to be any additional details about this dataset's intended structure). Probably due to this lack of clarity I'm having a hard time wrapping my head around what the x/y coordinates and the lat/lon coordinates are supposed to demonstrate about xarray here, or how the x/y and lat/lon values are represented in the data structure. Are the x and y coordinates in a map projection of some kind? I have worked successfully with Datasets in the past, but as someone who doesn't work with geospatial data, I find myself more confused about Datasets after reading this example than before.

I suspect that all that is needed is a clear description of what these data are supposed to represent, how they are intended to be used, and how x/y and lat/lon are related. If someone can explain this to me, I'd be happy to submit a PR for the docs.

this PR builds towards

• https://github.com/pydata/xarray/pull/8870
• https://github.com/pydata/xarray/pull/8856

the primary objective is to partially address

1. Implement fall back .oindex, .vindex properties on BackendArray base class. These will simply rewrap the key tuple with the appropriate *Indexer object, and pass it on to __getitem__ or __setitem__. These methods will also raise DeprecationWarning so that external backends will know to migrate to .oindex, and .vindex over the next year.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What is your issue?

I've recently been hit with this when experimenting with xr.dot and xr.corr — xr.dot takes dims, and xr.cov takes dim. Because they each take multiple arrays as positional args, kwargs are more conventional.

Should we standardize on one of these?

I think this is the final set of methods

• [x] Closes #6646
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Notes inline. Would fix #4587

FWIW one thing I'd be keen for to do generally — though maybe this isn't the place to start it — is handle warnings in the test suite when we add a new warning — i.e. filter them out where we expect them.

In this case, that would be the loading the netCDF files that have duplicate dims.

Otherwise warnings become a huge block of text without much salience. I mostly see the 350 lines of them and think "meh mostly units & cftime", but then something breaks on a new upstream release that was buried in there, or we have a supported code path that is raising warnings internally.

(I'm not sure whether it's possible to generally enforce that — maybe we could raise on any warnings coming from within xarray? Would be a non-trivial project to get us there though...)

Originally posted by @max-sixty in https://github.com/pydata/xarray/issues/8491#issuecomment-1834615826

1. Now that numba supports 3.12.
2. Disabled pint on the main environment since it doesn't work. Pint is still installed in the all-but-dask env, which is still runs python 3.11 for this reason.

What happened?

When I try to run a modified version of the example from the dropna documentation (see below), it creates a never terminating process. To reproduce it I added a rolling operation before dropping nans and then run 4 processes using the standard library multiprocessing Pool class on DaskArrays. Running the rolling + dropna in a for loop finishes as expectedly in no time.

What did you expect to happen?

There is nothing obvious to me why this wouldn't just work unless there is a weird interaction between the Dask threads and the different processes. Using Xarray+Dask+Multiprocessing seems to work for me on other functions, it seems to be this particular combination that is problematic.

Minimal Complete Verifiable Example

```Python import xarray as xr import numpy as np from multiprocessing import Pool

datasets = [xr.Dataset( { "temperature": ( ["time", "location"], [[23.4, 24.1], [np.nan if i>1 else 23.4, 22.1 if i<2 else np.nan], [21.8 if i<3 else np.nan, 24.2], [20.5, 25.3]], ) }, coords={"time": [1, 2, 3, 4], "location": ["A", "B"]}, ).chunk(time=2) for i in range(4)]

def process(dataset): return dataset.rolling(dim={'time':2}).sum().dropna(dim="time", how="all").compute()

This works as expected

dropped = [] for dataset in datasets: dropped.append(process(dataset))

This seems to never finish

with Pool(4) as p: dropped = p.map(process, datasets) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

I am still running on 2023.08.0 see below for more details about the environment

Environment

Turns out pydap also needs to build with numpy>=2. Until it does, we should remove it from the upstream-dev environment. Also, numcodecs build-depends on setuptools-scm.

And finally, the h5py nightlies might support numpy>=2 (h5py>=3.11 supposedly is numpy>=2 compatible), so once again I'll try and see if CI passes.

• [x] towards #8844

Notes are inline.

• [x] Closes https://github.com/pydata/xarray/issues/8494
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Done with some help from an LLM — quite good for doing tedious tasks that we otherwise wouldn't want to do — can paste in all the warnings output and get a decent start on rules for exclusions

While I don't feel super confident that this is designed to protect against any bugs, it does solve the immediate problem in #8371, by hoisting the encoding check above the code that runs for only new variables. The encoding check is somewhat implicit, so this was an easy thing to miss prior.

• [x] Closes #8371,
• [x] Closes #8882
• [x] Closes #8876
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

cc @hmaarrfk @kmuehlbauer

Passed: https://github.com/pydata/xarray/actions/runs/4115923717/jobs/7105298426 Failed: https://github.com/pydata/xarray/actions/runs/4115946392/jobs/7105345290

Versions: h5netcdf 1.1.0 pyhd8ed1ab_0 conda-forge h5py 3.8.0 nompi_py310h5555e59_100 conda-forge hdf4 4.2.15 h7aa5921_5 conda-forge hdf5 1.12.2 nompi_h48135f9_101 conda-forge

``` =================================== FAILURES =================================== ___ test_open_mfdataset_manyfiles[h5netcdf-20-True-5-5] ______ [gw1] darwin -- Python 3.10.9 /Users/runner/micromamba-root/envs/xarray-tests/bin/python

readengine = 'h5netcdf', nfiles = 20, parallel = True, chunks = 5 file_cache_maxsize = 5

@requires_dask
@pytest.mark.filterwarnings("ignore:use make_scale(name) instead")
def test_open_mfdataset_manyfiles(
    readengine, nfiles, parallel, chunks, file_cache_maxsize
):
    # skip certain combinations
    skip_if_not_engine(readengine)

    if ON_WINDOWS:
        pytest.skip("Skipping on Windows")

    randdata = np.random.randn(nfiles)
    original = Dataset({"foo": ("x", randdata)})
    # test standard open_mfdataset approach with too many files
    with create_tmp_files(nfiles) as tmpfiles:
        writeengine = readengine if readengine != "pynio" else "netcdf4"
        # split into multiple sets of temp files
        for ii in original.x.values:
            subds = original.isel(x=slice(ii, ii + 1))
            if writeengine != "zarr":
                subds.to_netcdf(tmpfiles[ii], engine=writeengine)
            else:  # if writeengine == "zarr":
                subds.to_zarr(store=tmpfiles[ii])

        # check that calculation on opened datasets works properly

      with open_mfdataset(
            tmpfiles,
            combine="nested",
            concat_dim="x",
            engine=readengine,
            parallel=parallel,
            chunks=chunks if (not chunks and readengine != "zarr") else "auto",
        ) as actual:

/Users/runner/work/xarray/xarray/xarray/tests/test_backends.py:3267:

/Users/runner/work/xarray/xarray/xarray/backends/api.py:991: in open_mfdataset datasets, closers = dask.compute(datasets, closers) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py:599: in compute results = schedule(dsk, keys, kwargs) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py:89: in get results = get_async( /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:511: in get_async raise_exception(exc, tb) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:319: in reraise raise exc /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:224: in execute_task result = _execute_task(task, data) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py:119: in _execute_task return func((_execute_task(a, cache) for a in args)) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py:72: in apply return func(args, kwargs) /Users/runner/work/xarray/xarray/xarray/backends/api.py:526: in open_dataset backend_ds = backend.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:417: in open_dataset ds = store_entrypoint.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/store.py:32: in open_dataset vars, attrs = store.load() /Users/runner/work/xarray/xarray/xarray/backends/common.py:129: in load (decode_variable_name(k), v) for k, v in self.get_variables().items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf.py:220: in get_variables return FrozenDict( /Users/runner/work/xarray/xarray/xarray/core/utils.py:471: in FrozenDict return Frozen(dict(args, *kwargs)) /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:221: in <genexpr> (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:200: in open_store_variable elif var.compression is not None: /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/h5netcdf/core.py:394: in compression return self._h5ds.compression

self = <[AttributeError("'NoneType' object has no attribute '_root'") raised in repr()] Variable object at 0x151378970>

@property
def _h5ds(self):
    # Always refer to the root file and store not h5py object
    # subclasses:

  return self._root._h5file[self._h5path]

E AttributeError: 'NoneType' object has no attribute '_h5file'

```

What happened?

When selecting variables from a Dataset using a view on dict keys, the type returned is a DataArray, whereas the same using a list is a Dataset.

What did you expect to happen?

The type returned should be a Dataset.

Minimal Complete Verifiable Example

```Python import xarray as xr

d = {"a": ("dim", range(1, 4)), "b": ("dim", range(2, 5))}

data = xr.Dataset(d) select_dict = data[d.keys()] select_list = data[list(d)]

reveal_type(select_dict) reveal_type(select_list) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python $ mypy test.py test.py:9: note: Revealed type is "xarray.core.dataarray.DataArray" test.py:10: note: Revealed type is "xarray.core.dataset.Dataset" Success: no issues found in 1 source file

Anything else we need to know?

No response

Environment

What happened:

The xarray.ufuncs.angle is broken. From the help docstring one may use option deg=True to have the result in degrees instead of radians (which is consistent with numpy.angle function). Yet results show that this is not the case. Moreover specifying deg=True or deg=False leads to the same result with the values in radians.

What you expected to happen:

To have the result of xarray.ufuncs.angle converted to degrees when option deg=True is specified.

Minimal Complete Verifiable Example:

```python

Put your MCVE code here

import numpy as np import xarray as xr

ds = xr.Dataset(coords={'wd': ('wd', np.arange(0, 360, 30, dtype=float))})

Z = xr.ufuncs.exp(1j * xr.ufuncs.radians(ds.wd)) D = xr.ufuncs.angle(Z, deg=True) # YIELDS INCORRECT RESULTS if not np.allclose(ds.wd, (D % 360)): print(f"Issue with angle operation: {D.values%360} instead of {ds.wd.values}" \ + f"\n\tERROR xr.ufuncs.angle(Z, deg=True) gives incorrect results !!!")

D = xr.ufuncs.degrees(xr.ufuncs.angle(Z)) # Works OK if not np.allclose(ds.wd, (D % 360)): print(f"Issue with angle operation: {D%360} instead of {ds.wd}" \ + f"\n\tERROR xr.ufuncs.degrees(xr.ufuncs.angle(Z)) gives incorrect results!!!")

D = xr.apply_ufunc(np.angle, Z, kwargs={'deg': True}) # Works OK if not np.allclose(ds.wd, (D % 360)): print(f"Issue with angle operation: {D%360} instead of {ds.wd}" \ + f"\n\tERROR xr.apply_ufunc(np.angle, Z, kwargs={{'deg': True}}) gives incorrect results!!!") ```

Anything else we need to know?:

Though xarray.ufuncs has a deprecated warning stating that the numpy equivalent may be used, this is not true for numpy.angle. Example:

```python import numpy as np import xarray as xr

ds = xr.Dataset(coords={'wd': ('wd', np.arange(0, 360, 30, dtype=float))})

Z = np.exp(1j * np.radians(ds.wd)) print(Z) print(f"Is Z an XArray? {isinstance(Z, xr.DataArray)}")

D = np.angle(ds.wd, deg=True) print(D) print(f"Is D an XArray? {isinstance(D, xr.DataArray)}") `` If this code is run, the result ofnumpy.angle(xarray.DataArray)is not a DataArray object, contrary to other numpy operations (for all versions of xarray I've used). Hence thexarray.ufuncs.angle` is a great option, if it was not for the current problem.

Environment:

No issues with xarray versions 0.16.2 and 0.17.0. This error happens from 0.18.0 onwards, up to 0.19.0 (recentmost).

What happened?

I am experiencing an error when using the array.interp function. The error message indicates that the function signature does not match with scipy interpn.

It 's linked to scipy update 1.10.0 (2023/01/03).

What did you expect to happen?

I would interpolate 2D data of numpy float64 : two data lattitudes and longitudes following <xarray.DataArray (row: 32, col: 32)>. da is a xarray dataset : <xarray.Dataset> Dimensions: (lat: 721, lon: 1441) Coordinates: * lat (lat) float64 90.0 89.75 89.5 89.25 ... -89.25 -89.5 -89.75 -90.0 * lon (lon) float64 0.0 0.25 0.5 0.75 1.0 ... 359.2 359.5 359.8 360.0 Data variables: hgt (lat, lon) >f4 13.61 13.61 13.61 13.61 ... -29.53 -29.53 -29.53 Attributes:

Minimal Complete Verifiable Example

Python interpolated_da = da.interp( { "x": xr.DataArray(x, dims=("x", "y")), "y": xr.DataArray(y, dims=("x", "y")), } )

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python interpolated_da = da.interp( venv/lib/python3.8/site-packages/xarray/core/dataset.py:3378: in interp variables[name] = missing.interp(var, var_indexers, method, kwargs) venv/lib/python3.8/site-packages/xarray/core/missing.py:639: in interp interped = interp_func( venv/lib/python3.8/site-packages/xarray/core/missing.py:764: in interp_func return _interpnd(var, x, new_x, func, kwargs) venv/lib/python3.8/site-packages/xarray/core/missing.py:788: in _interpnd rslt = func(x, var, xi, kwargs) venv/lib/python3.8/site-packages/scipy/interpolate/_rgi.py:654: in interpn return interp(xi) venv/lib/python3.8/site-packages/scipy/interpolate/_rgi.py:336: in call result = evaluate_linear_2d(self.values,

??? E TypeError: No matching signature found

_rgi_cython.pyx:19: TypeError ```

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None python: 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 5.4.0-135-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.0

xarray: 2022.12.0 pandas: 1.5.2 numpy: 1.22.4 scipy: 1.10.0 netCDF4: 1.6.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.4 cfgrib: None iris: None bottleneck: None dask: 2022.12.1 distributed: 2022.12.1 matplotlib: 3.6.2 cartopy: None seaborn: None numbagg: None fsspec: 2022.11.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.6.3 pip: 22.3.1 conda: None pytest: 7.2.0 mypy: None IPython: 8.7.0 sphinx: 5.3.0 None

Hello,

I have successfully opened an OpenDAP URL with ds = xarray.open_dataset(url) However, after selecting a subset with ds = ds.isel(time=0) and attempting to load the data with ds.load(), I get the following error:

HTTP Status 400 – Bad Request: Invalid character found in the request

target. The valid characters are defined in RFC 7230 and RFC 3986

I suspect the reason is that square brackets are passed in the URL when attempting to load: ...zg_6hrPlevPt_MIROC6_historical_r1i1p1f1_gn_185001010600-185101010000.nc.dods?zg.zg[0][0:6][0:127][0:255]] because of the index selection with .isel()

In fact, some servers do forbid square brackets: https://www.unidata.ucar.edu/mailing_lists/archives/thredds/2020/msg00056.html

Would it be possible to provide an option to encode URLs? ( [ becomes %5B, and ] becomes %5D )

Or, instead of loading directly with ds.load(), is there a way for me to retrieve the URL with offending brackets that is generated automatically by xarray, encode it myself, and then use ds2 = xarray.load_dataset(encoded_url) to load?

Thank you for your help!

What happened?

When trying to plot a 2d DataArray that has one of the 2 coordinates as datetime with da.plot.imshow, the following error is returned:

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

I know that I can use pcolormesh instead but on large arrays, imshow is much faster. It also behaves nicer with transparency and interpolation so or regularly sampled data, I find imshow a better choice.

Here is a minimal working example:

```python import numpy as np from xarray import DataArray from pandas import date_range

time = date_range('2020-01-01', periods=7, freq='D') y = np.linspace(0, 10, 11) da = DataArray( np.random.rand(time.size, y.size), coords=dict(time=time, y=y), dims=('time', 'y') )

da.plot.imshow(x='time', y='y') ```

What did you expect to happen?

I suggest the following solution which can be added after https://github.com/pydata/xarray/blob/4da7fdbd85bb82e338ad65a532dd7a9707e18ce0/xarray/plot/plot.py#L1366

python left, right = map(date2num, (left, right))

and then adding: python ax.xaxis_date() plt.setp(ax.get_xticklabels(), rotation=30, ha='right')

Minimal Complete Verifiable Example

```Python import numpy as np from xarray import DataArray from pandas import date_range

creating the data

time = date_range('2020-01-01', periods=7, freq='D') y = np.linspace(0, 10, 11) da = DataArray( np.random.rand(time.size, y.size), coords=dict(time=time, y=y), dims=('time', 'y') )

import matplotlib.pyplot as plt from matplotlib.dates import date2num, AutoDateFormatter

from https://github.com/pydata/xarray/blob/4da7fdbd85bb82e338ad65a532dd7a9707e18ce0/xarray/plot/plot.py#L1348

def _center_pixels(x): """Center the pixels on the coordinates.""" if np.issubdtype(x.dtype, str): # When using strings as inputs imshow converts it to # integers. Choose extent values which puts the indices in # in the center of the pixels: return 0 - 0.5, len(x) - 0.5

try:
    # Center the pixels assuming uniform spacing:
    xstep = 0.5 * (x[1] - x[0])
except IndexError:
    # Arbitrary default value, similar to matplotlib behaviour:
    xstep = 0.1

return x[0] - xstep, x[-1] + xstep

Center the pixels:

left, right = _center_pixels(da.time) top, bottom = _center_pixels(da.y)

the magical step

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

left, right = map(date2num, (left, right))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

plotting

fig, ax = plt.subplots() ax.imshow( da.T, extent=(left, right, top, bottom), origin='lower', aspect='auto' )

ax.xaxis_date() plt.setp(ax.get_xticklabels(), rotation=30, ha='right') ```

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python

TypeError Traceback (most recent call last) /var/folders/bj/czjbfh496258q1lc3p01lyz00000gn/T/ipykernel_59425/1460104966.py in <module> ----> 1 da.plot.imshow(x='time', y='y')

~/miniconda3/lib/python3.8/site-packages/xarray/plot/plot.py in plotmethod(_PlotMethods_obj, x, y, figsize, size, aspect, ax, row, col, col_wrap, xincrease, yincrease, add_colorbar, add_labels, vmin, vmax, cmap, colors, center, robust, extend, levels, infer_intervals, subplot_kws, cbar_ax, cbar_kwargs, xscale, yscale, xticks, yticks, xlim, ylim, norm, kwargs) 1306 for arg in ["_PlotMethods_obj", "newplotfunc", "kwargs"]: 1307 del allargs[arg] -> 1308 return newplotfunc(allargs) 1309 1310 # Add to class _PlotMethods

~/miniconda3/lib/python3.8/site-packages/xarray/plot/plot.py in newplotfunc(darray, x, y, figsize, size, aspect, ax, row, col, col_wrap, xincrease, yincrease, add_colorbar, add_labels, vmin, vmax, cmap, center, robust, extend, levels, infer_intervals, colors, subplot_kws, cbar_ax, cbar_kwargs, xscale, yscale, xticks, yticks, xlim, ylim, norm, kwargs) 1208 ax = get_axis(figsize, size, aspect, ax, subplot_kws) 1209 -> 1210 primitive = plotfunc( 1211 xplt, 1212 yplt,

~/miniconda3/lib/python3.8/site-packages/xarray/plot/plot.py in imshow(x, y, z, ax, kwargs) 1394 z[np.any(z.mask, axis=-1), -1] = 0 1395 -> 1396 primitive = ax.imshow(z, defaults) 1397 1398 # If x or y are strings the ticklabels have been replaced with

~/miniconda3/lib/python3.8/site-packages/matplotlib/_api/deprecation.py in wrapper(args, kwargs) 454 "parameter will become keyword-only %(removal)s.", 455 name=name, obj_type=f"parameter of {func.name}()") --> 456 return func(args, kwargs) 457 458 # Don't modify func's signature, as boilerplate.py needs it.

~/miniconda3/lib/python3.8/site-packages/matplotlib/init.py in inner(ax, data, args, kwargs) 1410 def inner(ax, args, data=None, kwargs): 1411 if data is None: -> 1412 return func(ax, *map(sanitize_sequence, args), kwargs) 1413 1414 bound = new_sig.bind(ax, args, *kwargs)

~/miniconda3/lib/python3.8/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, interpolation_stage, filternorm, filterrad, resample, url, **kwargs) 5450 # update ax.dataLim, and, if autoscaling, set viewLim 5451 # to tightly fit the image, regardless of dataLim. -> 5452 im.set_extent(im.get_extent()) 5453 5454 self.add_image(im)

~/miniconda3/lib/python3.8/site-packages/matplotlib/image.py in set_extent(self, extent) 980 self._extent = xmin, xmax, ymin, ymax = extent 981 corners = (xmin, ymin), (xmax, ymax) --> 982 self.axes.update_datalim(corners) 983 self.sticky_edges.x[:] = [xmin, xmax] 984 self.sticky_edges.y[:] = [ymin, ymax]

~/miniconda3/lib/python3.8/site-packages/matplotlib/axes/_base.py in update_datalim(self, xys, updatex, updatey) 2474 """ 2475 xys = np.asarray(xys) -> 2476 if not np.any(np.isfinite(xys)): 2477 return 2478 self.dataLim.update_from_data_xy(xys, self.ignore_existing_data_limits,

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' ```

Anything else we need to know?

No response

Environment

What happened: datetime was in np.datetime64 formet. When converted t datetime.datetime format it returned an int What you expected to happen: `to get a datetime returned Minimal Complete Verifiable Example:

```python

Put your MCVE code here

import xarray as xr import numpy as np import datetime date_frame = xr.DataArray(dims='time',coords={'time':pd.date_range('2000-01-01',periods=365)},data=np.zeros(365)) print('pandas date range (datetime): ',pd.date_range('2000-01-01',periods=365)[0]) print('dataframe datetime converted to datetime (int): ',date_frame.coords['time'].data[0].astype(datetime.datetime)) print("normal numpy datetime64 converted to datetime (datetime): ",np.datetime64(datetime.datetime(2000,1,1)).astype(datetime.datetime)) output: pandas date range (datetime): 2000-01-01 00:00:00 dataframe datetime converted to datetime (int): 946684800000000000 normal numpy datetime64 converted to datetime (datetime): 2000-01-01 00:00:00 ```

if converted to int, it also gives different lengths of int : date_frame: 946684800000000000 946684800000000 normal datetime64^ Anything else we need to know?:

it is also mentioned in this SO thread appears to be a problem in the datetime64....

numpy version 1.20.0 pandas version 1.2.1

Environment:

What happened?

It seems netcdf4 does not work well currently with s3fs the FUSE filesystem layer over S3 compatible storage with either the default netcdf4 engine nor with the h5netcdf.

Here is an example python import numpy as np import xarray as xr from datetime import datetime, timedelta NTIMES=48 start = datetime(2022,10,6,0,0) time_vals = [start + timedelta(minutes=20*t) for t in range(NTIMES)] times = xr.DataArray(data = [t.strftime('%Y%m%d%H%M%S').encode() for t in time_vals], dims=['Time']) v1 = xr.DataArray(data=np.zeros((len(times), 201, 201)), dims=['Time', 'x', 'y']) ds = xr.Dataset(data_vars=dict(times=times, v1=v1)) ds.to_netcdf(path='/my_s3_fs/test_netcdf.nc', format='NETCDF4', mode='w') On my system this code crashes with NTIMES=48, but completes without an error with NTIMES=24.

The output with NTIMES=48 is

``` There are 1 HDF5 objects open!

Report: open objects on 72057594037927936 Segmentation fault (core dumped) ```

I have tried the other engine that handles NETCDF4 in xarray with engine='h5netcdf' and also got a segfault.

A quick workaround seems to be to use the local filesystem to write the NetCDF file and then move the complete file to S3.

python ds.to_netcdf(path='/tmp/test_netcdf.nc', format='NETCDF4', mode='w') shutil.move('/tmp/test_netcdf.nc', '/my_s3_fs/test_netcdf.nc') There are several pieces of software involved here: the xarray package (0.16.1), netcdf4 (1.5.4), HDF5 (1.10.6), and s3fs (1.79). If this is not a bug in my code but in the underlying libraries, most likely it is not an xarray bug, but since it fails with both Netcdf4 engines, I decided to report it here.

What did you expect to happen?

With NTIMES=24 I am getting a file /my_s3_fs/test_netcdf.nc of about 7.8 MBytes. WIth NTIMES=36 I get an empty file. I would expect to have this code run without a segfault and produce a nonempty file.

Minimal Complete Verifiable Example

Python import numpy as np import xarray as xr from datetime import datetime, timedelta NTIMES=48 start = datetime(2022,10,6,0,0) time_vals = [start + timedelta(minutes=20*t) for t in range(NTIMES)] times = xr.DataArray(data = [t.strftime('%Y%m%d%H%M%S').encode() for t in time_vals], dims=['Time']) v1 = xr.DataArray(data=np.zeros((len(times), 201, 201)), dims=['Time', 'x', 'y']) ds = xr.Dataset(data_vars=dict(times=times, v1=v1)) ds.to_netcdf(path='/my_s3_fs/test_netcdf.nc', format='NETCDF4', mode='w')

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python There are 1 HDF5 objects open!

Report: open objects on 72057594037927936 Segmentation fault (core dumped) ```

Anything else we need to know?

No response

Environment

What happened?

When I export a dataset to NetCDF and Zarr, the y axis appears to have been reversed with gdalinfo. I also cannot build a vrt file with the Zarr file since it complains about positive NS axis, but this works fine with the NetCDF file.

Example NetCDF file as input: in.nc.zip

gdalinfo on output NetCDF file: $ gdalinfo NETCDF:out.nc:air_temperature_2m Driver: netCDF/Network Common Data Format Files: out.nc out.nc.aux.xml Size is 949, 1069 Coordinate System is: PROJCRS["unnamed", BASEGEOGCRS["unknown", DATUM["unnamed", ELLIPSOID["Sphere",6371000,0, LENGTHUNIT["metre",1, ID["EPSG",9001]]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433, ID["EPSG",9122]]]], CONVERSION["unnamed", METHOD["Lambert Conic Conformal (2SP)", ID["EPSG",9802]], PARAMETER["Latitude of false origin",63.3, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8821]], PARAMETER["Longitude of false origin",15, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8822]], PARAMETER["Latitude of 1st standard parallel",63.3, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8823]], PARAMETER["Latitude of 2nd standard parallel",63.3, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8824]], PARAMETER["Easting at false origin",0, LENGTHUNIT["metre",1], ID["EPSG",8826]], PARAMETER["Northing at false origin",0, LENGTHUNIT["metre",1], ID["EPSG",8827]]], CS[Cartesian,2], AXIS["easting",east, ORDER[1], LENGTHUNIT["metre",1, ID["EPSG",9001]]], AXIS["northing",north, ORDER[2], LENGTHUNIT["metre",1, ID["EPSG",9001]]]] Data axis to CRS axis mapping: 1,2 Origin = (-1061334.000000000000000,1338732.125000000000000) Pixel Size = (2500.000000000000000,-2500.000000000000000) Metadata: air_temperature_2m#coordinates=longitude latitude air_temperature_2m#grid_mapping=projection_lambert air_temperature_2m#long_name=Screen level temperature (T2M) air_temperature_2m#standard_name=air_temperature air_temperature_2m#units=K air_temperature_2m#_FillValue=9.96921e+36 height1#description=height above ground height1#long_name=height height1#positive=up height1#units=m height1#_FillValue=nan NC_GLOBAL#coordinates=projection_lambert time NETCDF_DIM_EXTRA={height1} NETCDF_DIM_height1_DEF={1,5} NETCDF_DIM_height1_VALUES=2 projection_lambert#earth_radius=6371000 projection_lambert#grid_mapping_name=lambert_conformal_conic projection_lambert#latitude_of_projection_origin=63.3 projection_lambert#longitude_of_central_meridian=15 projection_lambert#standard_parallel={63.3,63.3} x#long_name=x-coordinate in Cartesian system x#standard_name=projection_x_coordinate x#units=m x#_FillValue=nan y#long_name=y-coordinate in Cartesian system y#standard_name=projection_y_coordinate y#units=m y#_FillValue=nan Geolocation: LINE_OFFSET=0 LINE_STEP=1 PIXEL_OFFSET=0 PIXEL_STEP=1 SRS=GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]] X_BAND=1 X_DATASET=NETCDF:"out.nc":longitude Y_BAND=1 Y_DATASET=NETCDF:"out.nc":latitude Corner Coordinates: Upper Left (-1061334.000, 1338732.125) ( 18d10'24.02"W, 72d45'59.56"N) Lower Left (-1061334.000,-1333767.875) ( 0d15'55.60"E, 50d18'23.10"N) Upper Right ( 1311166.000, 1338732.125) ( 54d17'24.85"E, 71d34'43.38"N) Lower Right ( 1311166.000,-1333767.875) ( 33d 2'20.10"E, 49d45' 6.51"N) Center ( 124916.000, 2482.125) ( 17d30' 3.21"E, 63d18' 1.50"N) Band 1 Block=949x1069 Type=Float32, ColorInterp=Undefined Min=236.480 Max=284.937 Minimum=236.480, Maximum=284.937, Mean=269.816, StdDev=9.033 NoData Value=9.96920996838686905e+36 Unit Type: K Metadata: coordinates=longitude latitude grid_mapping=projection_lambert long_name=Screen level temperature (T2M) NETCDF_DIM_height1=2 NETCDF_VARNAME=air_temperature_2m standard_name=air_temperature STATISTICS_MAXIMUM=284.93682861328 STATISTICS_MEAN=269.81614967971 STATISTICS_MINIMUM=236.47978210449 STATISTICS_STDDEV=9.0332172122638 units=K _FillValue=9.96921e+36

gdalinfo on output Zarr file: $ gdalinfo ZARR:out.zarr:/air_temperature_2m:0 Driver: Zarr/Zarr Files: none associated Size is 949, 1069 Origin = (-1061334.000000000000000,-1333767.875000000000000) Pixel Size = (2500.000000000000000,2500.000000000000000) Metadata: coordinates=longitude latitude grid_mapping=projection_lambert long_name=Screen level temperature (T2M) standard_name=air_temperature Corner Coordinates: Upper Left (-1061334.000,-1333767.875) Lower Left (-1061334.000, 1338732.125) Upper Right ( 1311166.000,-1333767.875) Lower Right ( 1311166.000, 1338732.125) Center ( 124916.000, 2482.125) Band 1 Block=475x268 Type=Float32, ColorInterp=Undefined NoData Value=9.96920996838686905e+36 Unit Type: K

The main issue is that the origin and y-axis direction is reversed, as you can see from the origin and pixel size. I have tried taking the CRS from the netcdf and adding it to the Zarr file as a _CRS attribute manually, but this doesn't make any difference to the origin or pixel size.

What did you expect to happen?

Origin, pixel size and corner coords should match those in the netcdf file.

$ gdalinfo ZARR:out.zarr:/air_temperature_2m:0 Driver: Zarr/Zarr Files: none associated Size is 949, 1069 Origin = (-1061334.000000000000000,1338732.125000000000000) Pixel Size = (2500.000000000000000,-2500.000000000000000) Metadata: coordinates=longitude latitude grid_mapping=projection_lambert long_name=Screen level temperature (T2M) standard_name=air_temperature Corner Coordinates: Corner Coordinates: Upper Left (-1061334.000, 1338732.125) ( 18d10'24.02"W, 72d45'59.56"N) Lower Left (-1061334.000,-1333767.875) ( 0d15'55.60"E, 50d18'23.10"N) Upper Right ( 1311166.000, 1338732.125) ( 54d17'24.85"E, 71d34'43.38"N) Lower Right ( 1311166.000,-1333767.875) ( 33d 2'20.10"E, 49d45' 6.51"N) Center ( 124916.000, 2482.125) ( 17d30' 3.21"E, 63d18' 1.50"N) Band 1 Block=475x268 Type=Float32, ColorInterp=Undefined NoData Value=9.96920996838686905e+36 Unit Type: K

Minimal Complete Verifiable Example

```Python import xarray as xr from pyproj import CRS

ds = xr.open_dataset("in.nc")

Optionally take copy CRS to Zarr (produces and error, but does work)

crs_wkt = CRS.from_cf(ds["projection_lambert"].attrs).to_wkt() ds["air_temperature_2m"] = ds["air_temperature_2m"].assign_attrs(_CRS={"wkt": crs_wkt})

ds.to_zarr("out.zarr")

ds.to_netcdf("out.nc") ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

• [x] Closes first step of #8496, would require another PR later to actually change the return type. Also really resolves the second half of #921.
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Is your feature request related to a problem?

Currently, I have something similar to this, where the input_lat is transformed to new_lat (here, +0.25, but in real use case, it's indeterministic).

Since xarray_ufunc doesn't return a dataset with actual coordinates values, I had to return a second output to retain new_lat to properly update the coordinate values, but this second output is shaped time, lat, lon so I have to ds["lat"] = new_lat.isel(lon=0, time=0).values, which I think is inefficient; I simply need it to be shaped lat.

Any ideas on how I can modify this to make it more efficient?

```python import xarray as xr import numpy as np

air = xr.tutorial.open_dataset("air_temperature")["air"] input_lat = np.arange(20, 45)

def interp1d_np(data, base_lat, input_lat): new_lat = input_lat + 0.25 return np.interp(new_lat, base_lat, data), new_lat

ds, new_lat = xr.apply_ufunc( interp1d_np, # first the function air, air.lat, # as above input_lat, # as above input_core_dims=[["lat"], ["lat"], ["lat"]], # list with one entry per arg output_core_dims=[["lat"], ["lat"]], # returned data has one dimension exclude_dims=set(("lat",)), # dimensions allowed to change size. Must be a set! vectorize=True, # loop over non-core dims ) new_lat = new_lat.isel(lon=0, time=0).values ds["lat"] = new_lat ```

Describe the solution you'd like

Either be able to automatically assign the new_lat to the returned xarray object, or allow a 1D dataset to be returned

Describe alternatives you've considered

No response

Additional context

No response

Hello, I have been trying to write zarr files from xarray directly into an s3 store but keep getting errors for missing arrays. It looks like the structure of the zarr archive is created in my s3 bucket, I can see .zarray and .zattrs files but it's missing the 0.0.0, 0.0.1, etc files. I have been able to write the same arrays directly to my disk so don't think it's an issue with the dataset itself.

MCVE Code Sample

```python s3 = s3fs.S3FileSystem(anon=False) store= s3fs.S3Map(root=f's3://my-bucket/data.zarr', s3=s3, check=False)

ds.to_zarr(store=store, consolidated=True, mode='w')

```

Output

The variable name of the array changes by the run, it's not always the same one that it says is missing.

Output of xr.show_versions()

Is your feature request related to a problem?

DataArray and Dataset have an .expand_dims method, but it looks like Variable doesn't.

Describe the solution you'd like

Variable should also have this method, the only difference being that it wouldn't create any coordinates or indexes.

Describe alternatives you've considered

No response

Additional context

No response

• [x] Solves part of #8871 by pulling out part of https://github.com/pydata/xarray/pull/8872#issuecomment-2027571714
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

TODO: - [x] Add new kwarg to DataArray.expand_dims - [ ] Add examples to docstrings? - [x] Check it actually solves the problem in #8872

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This PR continues the overall work of migrating DataTree into xarray.

• xarray/core/datatree_render.py is the renamed version of xarray/datatree_/datatree/render.py.
• xarray/core/extensions.py now contains functionality from xarray/datatree_/datatree/extensions.py.
• xarray/core/formatting.py now contains functionality from xarray/datatree_/datatree/formatting.py.
• xarray/tests/test_datatree.py now contains tests from xarray/datatree_/datatree/tests/test_dataset_api.py.
• xarray/testing/assertions.py now contains functionality from /xarray/datatree_/datatree/testing.py.

I had also meant to get to common.py and what's left of io.py, but I've got a hefty couple of days of meetings ahead, so I wanted to get this progress into PR before that happens. @flamingbear or I can follow up with the remaining things in a separate PR. (Also this PR is already getting a little big, so maybe it's already got enough in it)

• [x] Contributes to migration step for miscellaneous modules in #8572
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

After adding the py38-all-but-dask CI in #3919, it discovered a few backend issues: - zarr: - [x] open_zarr with chunks="auto" always tries to chunk, even if dask is not available (fixed in #3919) - [x] ZarrArrayWrapper.__getitem__ incorrectly passes the indexer's tuple attribute to _arrayize_vectorized_indexer (this only happens if dask is not available) (fixed in #3919) - [x] slice indexers with negative steps get transformed incorrectly if dask is not available https://github.com/pydata/xarray/pull/8674 - rasterio: - ~calling pickle.dumps on a Dataset object returned by open_rasterio fails because a non-serializable lock was used (if dask is installed, a serializable lock is used instead)~

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

Reindexing a dataset without loading it into memory seems to be very slow (about 1000x slower than reindexing after loading into memory).

Here is a minimum working example: ``` times = 100 nlat = 200 nlon = 300

fp = xr.Dataset({"fp": (["time", "lat", "lon"], np.arange(times * nlat * nlon).reshape(times, nlat, nlon))}, coords={"time": pd.date_range(start="2019-01-01T02:00:00", periods=times, freq="1H"), "lat": np.arange(nlat), "lon": np.arange(nlon)})

flux = xr.Dataset({"flux": (["time", "lat", "lon"], np.arange(nlat * nlon).reshape(1, nlat, nlon))}, coords={"time": [pd.to_datetime("2019-01-01")], "lat": np.arange(nlat) + np.random.normal(0.0, 0.01, nlat), "lon": np.arange(nlon) + np.random.normal(0.0, 0.01, nlon)})

fp.to_netcdf("combine_datasets_tests/fp.nc") flux.to_netcdf("combine_datasets_tests/flux.nc")

fp1 = xr.open_dataset("combine_datasets_tests/fp.nc") flux1 = xr.open_dataset("combine_datasets_tests/flux.nc") ```

Then flux1 = flux1.reindex_like(fp1, method="ffill", tolerance=None) takes over a minute, while flux1 = flux1.load().reindex_like(fp1, method="ffill", tolerance=None) is almost instantaneous (timeit says 91ms, including opening the dataset... I'm not sure if caching is influencing this).

Profiling the "reindex without load" cell: ``` 804936 function calls (804622 primitive calls) in 93.285 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function) 1 92.211 92.211 93.191 93.191 {built-in method _operator.getitem} 1 0.289 0.289 0.980 0.980 utils.py:81(_StartCountStride) 6 0.239 0.040 0.613 0.102 shape_base.py:267(apply_along_axis) 72656 0.109 0.000 0.109 0.000 utils.py:429(<lambda>) 72656 0.085 0.000 0.136 0.000 utils.py:430(<lambda>) 72661 0.051 0.000 0.051 0.000 {built-in method numpy.arange} 145318 0.048 0.000 0.115 0.000 shape_base.py:370(<genexpr>) 2 0.045 0.023 0.046 0.023 indexing.py:1334(getitem) 6 0.044 0.007 0.044 0.007 numeric.py:136(ones) 145318 0.044 0.000 0.067 0.000 index_tricks.py:690(next) 14 0.033 0.002 0.033 0.002 {built-in method numpy.empty} 145333/145325 0.023 0.000 0.023 0.000 {built-in method builtins.next} 1 0.020 0.020 93.275 93.275 duck_array_ops.py:317(where) 21 0.018 0.001 0.018 0.001 {method 'astype' of 'numpy.ndarray' objects} 145330 0.013 0.000 0.013 0.000 {built-in method numpy.asanyarray} 1 0.002 0.002 0.002 0.002 {built-in method _functools.reduce} 1 0.002 0.002 93.279 93.279 variable.py:821(_getitem_with_mask) 18 0.001 0.000 0.001 0.000 {built-in method numpy.zeros} 1 0.000 0.000 0.000 0.000 file_manager.py:226(close) ```

The getitem call at the top is from xarray.backends.netCDF4_.py, line 114. Because of the jittered coordinates in flux, I'm assuming that the index passed to netCDF4 is not consecutive/strictly monotonic integers (0, 1, 2, 3, ...). In the past, this has caused issues: https://github.com/Unidata/netcdf4-python/issues/680.

In my venv, netCDF4 was installed from a wheel with the following versions: netcdf4-python version: 1.6.5 HDF5 lib version: 1.12.2 netcdf lib version: 4.9.3-development

This is with xarray version 2023.12.0, numpy 1.26, and pandas 1.5.3.

I will try to investigate more and hopefully simplify the example. (Can't quite justify spending more time on it at work because this is just to tag a version that was used in some experiments before we switch to zarr as a backend, so hopefully it won't be relevant at that point.)

What happened?

Apparently xarray-2024.2.0 requires xarray.datatree_ module but this module isn't included in sdist tarball.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

Python $ tar -tf /tmp/dist/xarray-2024.2.0.tar.gz | grep datatree_ (empty)

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

n/a

This PR removes the directive that strips out the datatree_ directory from the xarray distribution.

It also cleans a few typing errors and removes exceptions for the datatree_ directory for mypy.

It does NOT remove the exception for pre-commit config.

• [X] Closes #8768

FYI @aulemahal, numpy.NaN will be removed in the upcoming numpy=2.0 release.

• [x] follow-up to #8603

• [x] Closes #8665
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

This PR migrates the formatting_html.py module into xarray/core/formatting_html.py as part of the on-going effort to merge xarray-datatree into xarray.

One thing of note is that importing and setting the OPTIONS to "default" in datatree/formatting_html.py (lines) were moved into xarray/core/options.py on #L23 and #L49. So, I did not add them back to xarray/core/formatting_html.py.

• [x] Completes migration step for datatree/formating_htmls.py https://github.com/pydata/xarray/issues/8572
• [x] Tests added
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

Some outstanding points/decisions brought up by this PR: - [ ] Confirm type promotion rules and write them out. As it stands now, if everything is of the same extension array type, it is passed onwards and otherwise is converted to numpy. (related: https://github.com/pydata/xarray/pull/8714) ~- [ ] Acceptance of plum as a dispatch method. Without it, the behavior should be fallen back on from before (cast to numpy types). I am a big fan of dispatching and think it could serve as a model going forward for making support of other data types/arrays more feasible. The other option, I think, would be to just use the underlying array of the ExtensionDuckArray class to decide and then have some central registry that serves as the basis for a decorator (like the api for accessors via _CachedAccessor). That being said, the current defaults are quite good so this is a marginal feature, in all likelihood.~ - [ ] Do we allow just pandas ExtensionArray directly or can we also allow Series?

Possible missing something else! Let me know!

Checklist: - [x] Closes #8463 and Closes #5287 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst - [ ] New functions/methods are listed in api.rst

Is your feature request related to a problem? Please describe. I started writing an ExtensionArray which is basically a Tuple[Array[str], Array[int], Array[int], Array[str], Array[str]]. Its scalar type is a Tuple[str, int, int, str, str].

This is working great in Pandas, I can read and write Parquet as well as csv with it. However, as soon as I'm using any .to_xarray() method, it gets converted to a NumPy array of objects. Also, converting back to Pandas keeps a Series of objects instead of my extension type.

Describe the solution you'd like Would it be possible to support Pandas Extension Types on coordinates? It's not necessary to compute anything on them, I'd just like to use them for dimensions.

Describe alternatives you've considered I was thinking over implementing a NumPy duck array, but I have never tried this and it looks quite complicated compared to the Pandas Extension types.

Is your feature request related to a problem?

We are looking to improve compatibility between AnnData and xarray (see https://github.com/scverse/anndata/issues/744), and so categoricals are naturally on our roadmap. Thus, I think some sort of standard-use categoricals array would be desirable. It seems something similar has come up with netCDF, although my knowledge is limited so this issue may be more distinct than I am aware. So what comes of this issue may solve two birds with one stone, or it may work towards some common solution that can at least help both use-cases (AnnData and netCDF ENUM).

Describe the solution you'd like

The goal would be a standard-use categorical data type xarray container of some sort. I'm not sure what form this can take.

We have something functional here that inherits from ExplicitlyIndexedNDArrayMixin and returns pandas.CategoricalDtype. So let's say this implementation would be at least a conceptual starting point to work from (it also seems not dissimilar to what is done here for new CF types).

Some issues: 1. I have no idea what a standard "return type" for an xarray categorical array should be (i.e., numpy with the categories applied, pandas, something custom etc.). So I'm not sure if using pandas.CategoricalDtype type is acceptable as In do in the linked implementation. Relatedly.... 2. I don't think using pandas.CategoricalDtype really helps with the already existing CF Enum need if you want to have the return type be some sort of numpy array (although again, not sure about the return type). As I understand it, though, the whole point of categoricals is to use integers as the base type and then only show "strings" outwardly i.e., printing, the API for equality operations, accessors etc., while the internals are based on integers. So I'm not really sure numpy is even an option here. Maybe we roll our own solution? 3. I am not sure this is the right level at which to implement this (maybe it should be a Variable? I don't think so, but I am just a beginner here 😄 )

It seems you may want, in addition to the array container, some sort of i/o functionality for this feature (so maybe some on-disk specification?).

Describe alternatives you've considered

I think there is some route via VariableCoder as hinted here i.e., using encode/decode. This would probably be more general purpose as we could encode directly to other data types if using pandas is not desirable. Maybe this would be a way to support both netCDF and returning a pandas.CategoricalDtype (again, not sure what the netCDF return type should be for ENUM).

Additional context

So just for reference, the current behavior of to_xarray with pandas.CategoricalDtype is object dtype from numpy:

```python import pandas as pd df = pd.DataFrame({'cat': ['a', 'b', 'a', 'b', 'c']}) df['cat'] = df['cat'].astype('category') df.to_xarray()['cat']

<xarray.DataArray 'cat' (index: 5)>

array(['a', 'b', 'a', 'b', 'c'], dtype=object)

Coordinates:

* index (index) int64 0 1 2 3 4

```

And as stated in the netCDF issue, for that use-case, the information about ENUM is lost (from what I can read).

Apologies if I'm missing something here! Feedback welcome! Sorry if this is a bit chaotic, just trying to cover my bases.

This PR continues the overall work of migrating DataTree into xarray.

datatree_mapping.py is the renamed version of mapping.py from the datatree repository.

• [x] Closes migration step for mapping.py #8572
• [ ] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Small PR to add a new "method" to convert to and from 360_day calendars. The current two methods (chosen with the align_on keyword) will always remove or add the same day-of-year for all years of the same length.

This new option will randomly chose the days, one for each fifth of the year (72-days period). It emulates the method of the LOCA datasets (see web page and article ). February 29th is always removed/added when the source/target is a leap year.

I copied the implementation from xclim (which I wrote), see code here .

What happened?

A zipped Zarr store is available on s3 bucket that requires authentication.

When using xr.open_dataset, the following exception occurs:

NoCredentialsError: Unable to locate credentials

What did you expect to happen?

I expected the dataset to be openable.

Minimal Complete Verifiable Example

It is difficult for me to describe a MCVE as it requires a remote file on an s3 bucket requiring authentication.

To reproduce fully, one must have access to a zipped zarr on an s3 bucket requiring authentication.

```Python

import xarray as xr

credentials_key = "key" credentials_secret = "secret" credentials_endpoint_url = "endpoint_url" credentials_region_name = "region"

storage_options = dict( key=credentials_key, secret=credentials_secret, client_kwargs=dict( endpoint_url=credentials_endpoint_url, region_name=credentials_region_name, ), )

zip_s3_zarr_path = "zip::s3://path/to/my/dataset.zarr.zip"

xds = xr.open_dataset( zip_s3_zarr_path, backend_kwargs={"storage_options": storage_options}, engine="zarr", group="/", consolidated=True, ) ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Note: the missing kwargs passing is still present in the latest main branch at the time of writing this issue: https://github.com/fsspec/filesystem_spec/blob/37c1bc63b9c5a5b2b9a0d5161e89b4233f888b29/fsspec/implementations/zip.py#L56

Tested on my local environment by editing fsspec itself. The Zip Zarr store on the s3 bucket can then be opened successfully.

Environment

Aims to resolve parts of #8844.

python xarray/tests/test_accessor_str.py::test_case_str: AssertionError: assert dtype('<U26') == dtype('<U30') I'm not sure this is the right location for the fix, at least it fixes those errors. AFAICT this is some issue somewhere inside apply_ufunc where the string dtype size is kept. So this fix removes the size information from the dtype (actually recreating it).

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.2.0 to 4.3.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This PR continues the overall work of migrating DataTree into xarray.

iterators.py does not have direct tests. In discussions with @TomNicholas and @flamingbear, we concurred that other unit tests utilise this functionality.

• [x] Closes migration step for iterators.py #8572
• [ ] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Noticed the **kwargs part of the docstring was mangled - see here

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] follow-up to #8846

pandas recently removed the deprecated unit kwarg to TimedeltaIndex.

• [x] towards #8844

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What is your issue?

Hi all, on a project I am attempting a dataset read using the xarray JSON reference system. Metadata for this file (an ATL03 h5 file) can be found here: https://nsidc.org/sites/default/files/icesat2_atl03_data_dict_v005.pdf

1. Reading a group with variables that have 2 dimensions or less produces no issues. E.g Group "gt1l/heights" (documented as /gtx/heights in the PDF) ds = xr.open_dataset("reference://", engine="zarr", backend_kwargs={ "consolidated": False, "storage_options": {"fo": JSON_PATH}, "group": "gt1l/heights" })
2. Reading a group with a variable that has 3+ dimensions causes the following error. The group "ancillary_data/calibrations/dead_time_radiometric_signal_loss/gt1l" contains variable rad_corr which contains 3 dimensions. ds = xr.open_dataset("reference://", engine="zarr", backend_kwargs={ "consolidated": False, "storage_options": {"fo": JSON_PATH}, "group": "ancillary_data/calibrations/dead_time_radiometric_signal_loss/gt1l" }) ``` { "name": "ValueError", "message": "conflicting sizes for dimension 'phony_dim_1': length 498 on 'width' and length 160 on {'phony_dim_0': 'dead_time', 'phony_dim_1': 'rad_corr', 'phony_dim_2': 'rad_corr'}", "stack": "--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[2], line 1 ----> 1 ds = xr.open_dataset(\"reference://\", engine=\"zarr\", backend_kwargs={ 2 \"consolidated\": False, 3 \"storage_options\": {\"fo\": JSON_PATH}, 4 \"group\": group_path 5 })

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/backends/api.py:539, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, kwargs) 527 decoders = _resolve_decoders_kwargs( 528 decode_cf, 529 open_backend_dataset_parameters=backend.open_dataset_parameters, (...) 535 decode_coords=decode_coords, 536 ) 538 overwrite_encoded_chunks = kwargs.pop(\"overwrite_encoded_chunks\", None) --> 539 backend_ds = backend.open_dataset( 540 filename_or_obj, 541 drop_variables=drop_variables, 542 decoders, 543 kwargs, 544 ) 545 ds = _dataset_from_backend_dataset( 546 backend_ds, 547 filename_or_obj, (...) 555 kwargs, 556 ) 557 return ds

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/backends/zarr.py:862, in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel) 860 store_entrypoint = StoreBackendEntrypoint() 861 with close_on_error(store): --> 862 ds = store_entrypoint.open_dataset( 863 store, 864 mask_and_scale=mask_and_scale, 865 decode_times=decode_times, 866 concat_characters=concat_characters, 867 decode_coords=decode_coords, 868 drop_variables=drop_variables, 869 use_cftime=use_cftime, 870 decode_timedelta=decode_timedelta, 871 ) 872 return ds

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/backends/store.py:43, in StoreBackendEntrypoint.open_dataset(self, store, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta) 29 encoding = store.get_encoding() 31 vars, attrs, coord_names = conventions.decode_cf_variables( 32 vars, 33 attrs, (...) 40 decode_timedelta=decode_timedelta, 41 ) ---> 43 ds = Dataset(vars, attrs=attrs) 44 ds = ds.set_coords(coord_names.intersection(vars)) 45 ds.set_close(store.close)

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/core/dataset.py:604, in Dataset.init(self, data_vars, coords, attrs) 601 if isinstance(coords, Dataset): 602 coords = coords.variables --> 604 variables, coord_names, dims, indexes, _ = merge_data_and_coords( 605 data_vars, coords, compat=\"broadcast_equals\" 606 ) 608 self._attrs = dict(attrs) if attrs is not None else None 609 self._close = None

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/core/merge.py:575, in merge_data_and_coords(data_vars, coords, compat, join) 573 objects = [data_vars, coords] 574 explicit_coords = coords.keys() --> 575 return merge_core( 576 objects, 577 compat, 578 join, 579 explicit_coords=explicit_coords, 580 indexes=Indexes(indexes, coords), 581 )

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/core/merge.py:761, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value) 756 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat) 757 variables, out_indexes = merge_collected( 758 collected, prioritized, compat=compat, combine_attrs=combine_attrs 759 ) --> 761 dims = calculate_dimensions(variables) 763 coord_names, noncoord_names = determine_coords(coerced) 764 if explicit_coords is not None:

File ~/opt/anaconda3/envs/kerchunkc/lib/python3.8/site-packages/xarray/core/variable.py:3208, in calculate_dimensions(variables) 3206 last_used[dim] = k 3207 elif dims[dim] != size: -> 3208 raise ValueError( 3209 f\"conflicting sizes for dimension {dim!r}: \" 3210 f\"length {size} on {k!r} and length {dims[dim]} on {last_used!r}\" 3211 ) 3212 return dims

ValueError: conflicting sizes for dimension 'phony_dim_1': length 498 on 'width' and length 160 on {'phony_dim_0': 'dead_time', 'phony_dim_1': 'rad_corr', 'phony_dim_2': 'rad_corr'}" }

```

1. Now, contrast with reading the same group with the 3+ dimension variables, but using the direct h5 file path. This does not produce an error ds = xr.open_dataset("/Users/katrinasharonin/Downloads/ATL03_20230816235231_08822014_006_01.h5", group="ancillary_data/calibrations/dead_time_radiometric_signal_loss/gt1l")

The JSON reference file has been attached for reference ATL03_REF_NONUTM.json

What happened?

I want to work with a huge dataset stored in hdf5 loaded in chunks. Each chunk contains part of my data that should be saved to a specific region of zarr files. I need to follow the original order of chunks. I found it a convenient way to use a map_blocks function for this purpose. However, I have missing values in the final zarr file. Some chunks or parts of chunks are not stored.

I used a simplified scenario for code documenting this behavior. The initial zarr file of zeros is filled with ones. There are always some parts where there are still zeros.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import os import shutil import xarray as xr import numpy as np import dask.array as da

xr.show_versions()

zarr_file = "file.zarr" if os.path.exists(zarr_file): shutil.rmtree(zarr_file)

chunk_size = 5 shape = (50, 32, 1000) ones_dataset = xr.Dataset({"data": xr.ones_like(xr.DataArray(np.empty(shape)))}) ones_dataset = ones_dataset.chunk({'dim_0': chunk_size}) chunk_indices = np.arange(len(ones_dataset.chunks['dim_0'])) chunk_ids = np.repeat(np.arange(ones_dataset.sizes["dim_0"] // chunk_size), chunk_size) chunk_ids_dask_array = da.from_array(chunk_ids, chunks=(chunk_size,))

Append the chunk IDs Dask array as a new variable to the existing dataset

ones_dataset['chunk_id'] = (('dim_0',), chunk_ids_dask_array)

Create a new dataset filled with zeros

zeros_dataset = xr.Dataset({"data": xr.zeros_like(xr.DataArray(np.empty(shape)))}) zeros_dataset.to_zarr(zarr_file, compute=False)

def process_chunk(chunk_dataset): chunk_id = int(chunk_dataset["chunk_id"][0]) chunk_dataset_to_store = chunk_dataset.drop_vars("chunk_id")

start_index = chunk_id * chunk_size
end_index = chunk_id * chunk_size + chunk_size

chunk_dataset_to_store.to_zarr(zarr_file, region={'dim_0': slice(start_index, end_index)})
return chunk_dataset

ones_dataset.map_blocks(process_chunk, template=ones_dataset).compute()

Load data stored in zarr

zarr_data = xr.open_zarr(zarr_file, chunks={'dim_0': chunk_size})

Find differences

for var_name in zarr_data.variables: try: xr.testing.assert_equal(zarr_data[var_name], ones_dataset[var_name]) except AssertionError: print(f"Differences in {var_name}:") print(zarr_data[var_name].values) print(ones_dataset[var_name].values) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

I was trying to divide two DataArray objects, but I needed to check for zero values. I though to do something like: python res = np.divide(a, b, out=np.zeros_like(a, dtype=float), where=b != 0) to get zero whenever b == 0, but running this resulted in a crush of infinite recursion. Part of the traceback is shown below. The full traceback is way too long to post here.

What did you expect to happen?

The division would succeed and I'd get a DataArray with the same dimensions as a and b with the correct results and zero whenever b == 0.

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

a = xr.DataArray([-1, 0, 1, 2, 3], dims="x") b = xr.DataArray([0, 0, 0, 2, 2], dims="x") print(np.divide(a, b, out=np.zeros_like(a, dtype=float), where=b != 0)) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Python File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 898, in apply_array_ufunc return func(*args) ^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\arithmetic.py", line 83, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 1283, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 880, in apply_array_ufunc if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\core\computation.py", line 880, in <genexpr> if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\namedarray\pycompat.py", line 92, in is_chunked_array return is_duck_dask_array(x) or (is_duck_array(x) and hasattr(x, "chunks")) ^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\namedarray\utils.py", line 94, in is_duck_dask_array return is_duck_array(x) and is_dask_collection(x) ^^^^^^^^^^^^^^^^^^^^^ File "D:\Portable_Programs\.pyenv\pyenv-win\versions\3.11.5\Lib\site-packages\xarray\namedarray\utils.py", line 68, in is_dask_collection if module_available("dask"): ^^^^^^^^^^^^^^^^^^^^^^^^ RecursionError: maximum recursion depth exceeded while calling a Python object

Anything else we need to know?

No response

Environment

This fix enhances the issue 8860. I did not add any test since I believe it is not needed for this case since we did not add any functionality.

• [X] Closes #8860
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

Passing no data to the Dataset constructor can result in a very unhelpful "tuple index out of range" error when this is a clear case of malformed input that we should be able to catch.

What did you expect to happen?

An error more like "tuple must be of form (dims, data[, attrs])"

Minimal Complete Verifiable Example

Python xr.Dataset({"t": ()})

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python

IndexError Traceback (most recent call last) Cell In[2], line 1 ----> 1 xr.Dataset({"t": ()})

File ~/Documents/Work/Code/xarray/xarray/core/dataset.py:693, in Dataset.init(self, data_vars, coords, attrs) 690 if isinstance(coords, Dataset): 691 coords = coords._variables --> 693 variables, coord_names, dims, indexes, _ = merge_data_and_coords( 694 data_vars, coords 695 ) 697 self._attrs = dict(attrs) if attrs else None 698 self._close = None

File ~/Documents/Work/Code/xarray/xarray/core/dataset.py:422, in merge_data_and_coords(data_vars, coords) 418 coords = create_coords_with_default_indexes(coords, data_vars) 420 # exclude coords from alignment (all variables in a Coordinates object should 421 # already be aligned together) and use coordinates' indexes to align data_vars --> 422 return merge_core( 423 [data_vars, coords], 424 compat="broadcast_equals", 425 join="outer", 426 explicit_coords=tuple(coords), 427 indexes=coords.xindexes, 428 priority_arg=1, 429 skip_align_args=[1], 430 )

File ~/Documents/Work/Code/xarray/xarray/core/merge.py:718, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args) 715 for pos, obj in skip_align_objs: 716 aligned.insert(pos, obj) --> 718 collected = collect_variables_and_indexes(aligned, indexes=indexes) 719 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat) 720 variables, out_indexes = merge_collected( 721 collected, prioritized, compat=compat, combine_attrs=combine_attrs 722 )

File ~/Documents/Work/Code/xarray/xarray/core/merge.py:358, in collect_variables_and_indexes(list_of_mappings, indexes) 355 indexes_.pop(name, None) 356 append_all(coords_, indexes_) --> 358 variable = as_variable(variable, name=name, auto_convert=False) 359 if name in indexes: 360 append(name, variable, indexes[name])

File ~/Documents/Work/Code/xarray/xarray/core/variable.py:126, in as_variable(obj, name, auto_convert) 124 obj = obj.copy(deep=False) 125 elif isinstance(obj, tuple): --> 126 if isinstance(obj[1], DataArray): 127 raise TypeError( 128 f"Variable {name!r}: Using a DataArray object to construct a variable is" 129 " ambiguous, please extract the data using the .data property." 130 ) 131 try:

IndexError: tuple index out of range ```

Anything else we need to know?

No response

Environment

Xarray main

A drive-by PR as I was trying to figure out how these functions works.

What is your issue?

My python script was killed because it took up too much memory. After checking, I found that the problem is the ds.where() function.

The original netcdf file opened from the hard disk takes up about 10 Mb of storage, but when I mask the data that doesn't match according to the latitude and longitude location, the variable ds takes up a dozen GB of memory. When I deleted this variable using del ds, the memory occupied by the script immediately returned to normal.

```

Open this netcdf file.

ds = xr.open_dataset(track)

If longitude range is [-180, 180], then convert to [0, 360].

if np.any(ds[var_lon] < 0): ds[var_lon] = ds[var_lon] % 360

Extract data by longitude and latitude.

ds = ds.where((ds[var_lon] >= region[0]) & (ds[var_lon] <= region[1]) & (ds[var_lat] >= region[2]) & (ds[var_lat] <= region[3]))

Select data by range and value of some variables.

for key, value in range_select.items(): ds = ds.where((ds[key] >= value[0]) & (ds[key] <= value[1])) for key, value in value_select.items(): ds = ds.where(ds[key].isin(value)) ```

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.1.1 to 4.2.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

What happened?

I work with an aircraft emission dataset that is freely available online: emission dataset

During my calculations I eventually convert the DataSet to a DataFrame. My motivation is to avoid unnecessary rows in the DataFrame. Doing some calculations my code returned unexpected results. Eventually I could narrow it down to a DataSet.where(... , drop=True) argument I added along the way, which introduces differences in the data. Here are two examples:

Example 1: Along some dimensions data points vanished if drop=True

Example 2: For other dimensions (these?) data points appeared elsewhere if drop=True

What did you expect to happen?

I expect for my calculations to return the same results, regardless of whether drop=True is active or not.

Minimal Complete Verifiable Example

```Python !wget "https://zenodo.org/records/10818082/files/Emission_Inventory_H2O_Optimized_v0.1_MR3_Fleet_BRU-MYA_2075.nc"

import matplotlib.pyplot as plt import xarray as xr

nc_file = xr.open_dataset('Emission_Inventory_H2O_Optimized_v0.1_MR3_Fleet_BRU-MYA_2075.nc')

fig, axs = plt.subplots(1,2,figsize=(10,4))

nc_file.H2O.where(nc_file.H2O!=0, drop=True).sum(('lon','time')).plot.contour(x='lat',ax=axs[0]) axs[0].set_xlim(-50,90) axs[0].set_title('With drop=True')

nc_file.H2O.where(nc_file.H2O!=0, drop=False).sum(('lon','time')).plot.contour(x='lat',ax=axs[1]) axs[1].set_xlim(-50,90) axs[1].set_title('With drop=False')

plt.tight_layout() plt.show()

fig, axs = plt.subplots(1,2,figsize=(10,4))

nc_file.H2O.where(nc_file.H2O!=0, drop=True).sum(('lat','time')).plot.contour(x='lon',ax=axs[0]) axs[0].set_title('With drop=True')

nc_file.H2O.where(nc_file.H2O!=0, drop=False).sum(('lat','time')).plot.contour(x='lon',ax=axs[1]) axs[1].set_title('With drop=False')

plt.tight_layout() plt.show() ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

I am working on a Windows 11 computer with Python 3.11.9 installed as the only version of Python I am working in Virtual Studio Code I have created and activated a venv virtual environment for my project (not a conda one) I used the pip command to install xarray into the virtual environment I have a test script called Testing.py and the only thing it does is import xarray as xr and then it prints a message to the terminal window This works 100% in python in the VSC terminal window I then issue the following command pyinstaller --onefile Testing.py

This creates a file called Testing.exe for me When I run Testing.exe I get the following error message

**Traceback (most recent call last): File "importlib\metadata__init__.py", line 563, in from_name StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "Testing.py", line 20, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray__init__.py", line 3, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray\testing__init__.py", line 1, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray\testing\assertions.py", line 11, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray\core\duck_array_ops.py", line 36, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray\core\dask_array_ops.py", line 3, in <module> File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module File "xarray\core\nputils.py", line 14, in <module> File "xarray\namedarray\utils.py", line 60, in module_available File "importlib\metadata__init__.py", line 1009, in version File "importlib\metadata__init__.py", line 982, in distribution File "importlib\metadata__init__.py", line 565, in from_name importlib.metadata.PackageNotFoundError: No package metadata was found for numpy [15596] Failed to execute script 'Testing' due to unhandled exception!**

Please, is there anyone who can suggest what I need to do to get around this problem .... I need to distribute my project when it is complete

What happened?

I got the error: from_array() got an unexpected keyword argument 'inline_array' for this line in xarray: data = da.from_array( data, chunks, name=name, lock=lock, inline_array=inline_array, **kwargs )

lines 1230 and 1231 in xarray/core/variable.py

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Update the cache always.

Enforce typing on all tests in test_plot.py and add the remaining type hints.

• [x] Attempt to closes issue described in https://github.com/pydata/xarray/issues/8566#issuecomment-1870472827
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Discussed in https://github.com/pydata/xarray/discussions/8751

• [x] Closes #8753

This seems to have been around since 2016-ish, so presumably our backend code path is passing arrays around, not Variables.

cc @ilan-gold

What is your issue?

I have an era5 dataset stored in GCS bucket as zarr. It contains 273 weather related variables and 4 dimensions. It's an hourly stored data from 1940 to 2023. When I try to open with ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3") it takes 90 seconds to actually finish the open call. The chunk scheme is { 'time': 1 }.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I was curious to see if the hypothesis stateful testing would catch an inconsistent sequence of index manipulation operations like #8646. Turns out rename_vars is basically broken? (filed #8659) :P

PS: this blog post is amazing. E state = DatasetStateMachine() E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: () E Data variables: E *empty* E === E E E > vars: ('1', '1_') E state.add_dim_coord(var=<xarray.Variable (1: 1)> E array([0], dtype=uint32)) E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 1 (1) uint32 0 E Data variables: E 1_ (1) uint32 0 E === E E E > renaming 1 to 0 E state.rename_vars(newname='0') E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 0 (1) uint32 0 E Dimensions without coordinates: 1 E Data variables: E 1_ (1) uint32 0 E === E E E state.teardown()

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

joining the party - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Closes #8252 Closes #3921

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Toward #8510, builds on #8776

This adds a proposal for decoupling the lazy indexing array machinery, indexing adapter machinery, and Variable's setitem and getitem methods, so that the latter can be migrated to NamedArray.

cc @andersy005

• [x] xref #8844

What happened?

When appending to an existing zarr along a dimension (to_zarr(..., mode='a', append_dim="x" ,..)), if the dask chunking of the dataset to append does not align with the chunking of the existing zarr, the resulting consolidated zarr store may have NaNs instead of the actual values it is supposed to have.

What did you expect to happen?

We would expected that zarr append to have the same behaviour as if we concatenate dataset in memory (using concat) and write the whole result on a new zarr store in one go

Minimal Complete Verifiable Example

```Python from distributed import Client, LocalCluster import xarray as xr import tempfile

ds1 = xr.Dataset({"a": ("x", [1., 1.])}, coords={'x': [1, 2]}).chunk({"x": 3}) ds2 = xr.Dataset({"a": ("x", [1., 1., 1., 1.])}, coords={'x': [3, 4, 5, 6]}).chunk({"x": 3})

with Client(LocalCluster(processes=False, n_workers=1, threads_per_worker=2)): # The issue happens only when: threads_per_worker > 1 for i in range(0, 100): with tempfile.TemporaryDirectory() as store: print(store) ds1.to_zarr(store, mode="w") # write first dataset ds2.to_zarr(store, mode="a", append_dim="x") # append first dataset

        rez = xr.open_zarr(store).compute() # open consolidated dataset
        nb_values = rez.a.count().item(0) # count non NaN values
        if nb_values != 6:
            print("found NaNs:")
            print(rez.to_dataframe())
            break

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Python /tmp/tmptg_pe6ox /tmp/tmpm7ncmuxd /tmp/tmpiqcgoiw2 /tmp/tmppma1ieo7 /tmp/tmpw5vi4cf0 /tmp/tmp1rmgwju0 /tmp/tmpm6tfswzi found NaNs: a x 1 1.0 2 1.0 3 1.0 4 1.0 5 1.0 6 NaN

Anything else we need to know?

The example code snippet provided here, reproduces the issue.

Since the issue occurs randomly, we loop in the example for a few times and stop when the issue occurs.

In the example, when ds1 is first written, since it only contains 2 values along the x dimension, the resulting .zarr store have the chunking: {'x': 2}, even though we called .chunk({"x": 3}).

Side note: This behaviour in itself is not problematic in this case, but the fact that the chunking is silently changed made this issue harder to spot.

However, when we try to append the second dataset ds2, that contains 4 values, the .chunk({"x": 3}) in the begining splits the dask array into 2 dask chunks, but in a way that does not align with zarr chunks.

Zarr chunks: + chunk1 : x: [1; 2] + chunk2 : x: [3; 4] + chunk3 : x: [5; 6]

Dask chunks for ds2: + chunk A: x: [3; 4; 5] + chunk B: x: [6]

Both dask chunks A and B, are supposed to write on zarr chunk3 And depending on who writes first, we can end up with NaN on x = 5 or x = 6 instead of actual values.

The issue obviously happens only when dask tasks are run in parallel. Using safe_chunks = True when calling to_zarr does not seem to help.

We couldn't figure out from the documentation how to detect this kind of issues, and how to prevent them from happening (maybe using a synchronizer?)

Environment

• [x] Closes https://github.com/xarray-contrib/xeofs/issues/148
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

The change from https://github.com/pydata/xarray/pull/8672 mostly fixed the issue of serializing a reset multiindex in the backends, but there was an additional niche issue that turned up in xeofs that was causing serialization to still fail on the zarr backend.

The issue is that zarr is the only backend that uses a custom version of encode_cf_variable called encode_zarr_variable, and the way this gets called we don't pass through the name of the variable before running ensure_not_multiindex.

As a minimal fix, this PR just passes name through as an additional arg to the general encode_variable function. See @benbovy's comment that maybe we should actually unwrap the level coordinate in reset_index and clean up the checks in ensure_not_multiindex, but I wasn't able to get that working easily.

The exact workflow this turned up in involves DataTree and looks like this: ```python import numpy as np import xarray as xr from datatree import DataTree

ND DataArray that gets stacked along a multiindex

da = xr.DataArray(np.ones((3, 3)), coords={"dim1": [1, 2, 3], "dim2": [4, 5, 6]}) da = da.stack(feature=["dim1", "dim2"])

Extract just the stacked coordinates for saving in a dataset

ds = xr.Dataset(data_vars={"feature": da.feature})

Reset the multiindex, which should make things serializable

ds = ds.reset_index("feature") dt1 = DataTree() dt2 = DataTree(name="feature", data=ds) dt1["foo"] = dt2

Somehow in this step, dt1.foo.feature.dim1.variable becomes an IndexVariable again

print(type(dt1.foo.feature.dim1.variable))

Works

dt1.to_netcdf("test.nc", mode="w")

Fails

dt1.to_zarr("test.zarr", mode="w") ```

But we can reproduce in xarray with the test added here.

• Updates the docstring for persist to mention that it is not altering the original object.

• Adds a return value to the docstring for compute and persist on both Dataset and DataArray

• [x] Closes #8901

What is your issue?

I am playing around with using Dataset.persist and assumed it would work like .load. I also just looked at the source code and it looks to me like it should indeed replace the original data but I can see both in performance and the dask dashboard that steps are recomputed if I don't use the object returned by .persist which points me towards .persist behaving more like .compute.

In either case, I would make a PR to clarify in the docs whether persists leaves the original data untouched or not.

What happened?

Hi, I have been trying to use a custom function on the rolling reduction method, the original function tries to filter the nan values (any numpy function that I have used that handles nans generates the same problem) to later apply some simple aggregate functions, but it is killing all my workers even when the data is very small (I have 7 workers and all of them have 3 Gb of RAM).

What did you expect to happen?

I would expect less use of memory taking into account the size of the rolling window, the simplicity of the function and the amount of data used on the example.

Minimal Complete Verifiable Example

```Python import numpy as np import dask.array as da import xarray as xr import dask

def f(x, axis): # If I replace np.nansum by np.sum everything works perfectly and the amount of memory used is very small return np.nansum(x, axis=axis)

arr = xr.DataArray( dask.array.zeros( shape=(300, 30000), dtype=float, chunks=(30, 6000) ), dims=["a", "b"], coords={"a": list(range(300)), "b": list(range(30000))} )

arr.rolling(a=252).reduce(f).chunk({"a": 252}).to_zarr("/data/test/test_write", mode="w") ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Python KilledWorker: Attempted to run task ('nansum-overlap-sum-aggregate-sum-aggregate-e732de6ad917d5f4084b05192ca671c4', 0, 0) on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://172.18.0.2:39937. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.

Anything else we need to know?

No response

Environment

updates: - github.com/astral-sh/ruff-pre-commit: v0.2.0 → v0.3.4 - github.com/psf/black-pre-commit-mirror: 24.1.1 → 24.3.0 - github.com/pre-commit/mirrors-mypy: v1.8.0 → v1.9.0

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.1.0 to 4.1.1

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Should have been done as part of the last release https://github.com/pydata/xarray/releases/tag/v2024.03.0

Enforce typing on all tests in test_groupby.py and add the remaining type hints.

Upstream numpy has become a bit more particular with which types you can use for inplace operations. This PR fixes ``` __ TestImshow.test_imshow_rgb_values_in_valid_range __

self = <xarray.tests.test_plot.TestImshow object at 0x7f88320c2780>

def test_imshow_rgb_values_in_valid_range(self) -> None:
    da = DataArray(np.arange(75, dtype="uint8").reshape((5, 5, 3)))
    _, ax = plt.subplots()

  out = da.plot.imshow(ax=ax).get_array()

/home/runner/work/xarray/xarray/xarray/tests/test_plot.py:2034:

/home/runner/work/xarray/xarray/xarray/plot/accessor.py:421: in imshow return dataarray_plot.imshow(self._da, args, kwargs) /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1601: in newplotfunc primitive = plotfunc( /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1853: in imshow alpha = 255

self = masked_array( data=[[[1], [1], [1], [1], [1]],

    [[1],
     [1],
...,
     [1],
     [1],
     [1],
     [1]]],

mask=False, fill_value=np.int64(999999), dtype=uint8) other = 255

def __imul__(self, other):
    """
    Multiply self by other in-place.

    """
    m = getmask(other)
    if self._mask is nomask:
        if m is not nomask and m.any():
            self._mask = make_mask_none(self.shape, self.dtype)
            self._mask += m
    elif m is not nomask:
        self._mask += m
    other_data = getdata(other)
    other_data = np.where(self._mask, other_data.dtype.type(1), other_data)

  self._data.__imul__(other_data)

E numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'multiply' output from dtype('int64') to dtype('uint8') with casting rule 'same_kind'

/home/runner/micromamba/envs/xarray-tests/lib/python3.12/site-packages/numpy/ma/core.py:4415: UFuncTypeError ```

Some curious behaviors seen while debugging: ```python alpha = np.array([1], dtype=np.int8) alpha *= 255 repr(alpha) # 'array([-1], dtype=int8)'

alpha = np.array([1], dtype=np.int16) alpha *= 255 repr(alpha) # 'array([255], dtype=int16)' ```

xref: #8844