• [x] Solves the problem in https://github.com/pydata/xarray/pull/8712#issuecomment-1929037299
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

What happened?

When I set the .encoding['dtype'] attribute before saving a to disk, the actual on-disk representation appears to store a record of the dtype encoding, but when opening it back up in xarray I get the same dtype I had before, not the one specified in the encoding. Is that what's supposed to happen? How does this work? (This happens with both zarr and netCDF.)

What did you expect to happen?

I expected that setting .encoding['dtype'] would mean that once I open the data back up, it would be in the new dtype that I set in the encoding.

Minimal Complete Verifiable Example

```Python air = xr.tutorial.open_dataset('air_temperature')

air['air'].dtype # returns dtype('float32')

air['air'].encoding['dtype'] # returns dtype('int16'), which already seems weird

air.to_zarr('air.zarr') # I would assume here that the encoding actually does something during IO

now if I check the zarr .zarray metadata for the air variable it says

"dtype":"<i2"`

air2 = xr.open_dataset('air.zarr', engine='zarr') # open it back up

air2['air'].dtype # returns dtype('float32'), but I expected dtype('int16')

(the same thing happens also with saving to netCDF instead of Zarr) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

I know I didn't explicitly cast with .asdtype, but I'm still confused as to what the relation between the dtype encoding is supposed to be here.

I am probably just misunderstanding how this is supposed to work, but then this is arguably a docs issue, because here it says "[the encoding dtype field] controls the type of the data written on disk", which I would have thought also affects the data you get back when you open it up again?

Environment

main branch of xarray

What is your issue?

In https://github.com/fsspec/kerchunk/issues/377 the idea came up of using the xarray API to concatenate arrays which represent parts of a zarr store - i.e. using xarray to kerchunk a large set of netCDF files instead of using kerchunk.combine.MultiZarrToZarr.

The idea is to make something like this work for kerchunking sets of netCDF files into zarr stores

```python ds = xr.open_mfdataset( '/my/files*.nc' engine='kerchunk', # kerchunk registers an xarray IO backend that returns zarr.Array objects combine='nested', # 'by_coords' would require actually reading coordinate data parallel=True, # would use dask.delayed to generate reference dicts for each file in parallel )

ds # now wraps a bunch of zarr.Array / kerchunk.Array objects, no need for dask arrays

ds.kerchunk.to_zarr(store='out.zarr') # kerchunk defines an xarray accessor that extracts the zarr arrays and serializes them (which could also be done in parallel if writing to parquet) ```

I had a go at doing this in this notebook, and in doing so discovered a few potential issues with xarray's internals.

For this to work xarray has to: - Wrap a kerchunk.Array object which barely defines any array API methods, including basically not supporting indexing at all, - Store all the information present in a kerchunked Zarr store but without ever loading any data, - Not create any indexes by default during dataset construction or during xr.concat, - Not try to do anything else that can't be defined for a kerchunk.Array. - Possibly we need the Lazy Indexing classes to support concatenation https://github.com/pydata/xarray/issues/4628

It's an interesting exercise in using xarray as an abstraction, with no access to real numerical values at all.

RIP

• [x] Closes #8524

Builds upon #7991 by adding a page to the internals enumerating all the different ways in which xarray is interoperable.

Would be nice if https://github.com/pydata/xarray/pull/6975 were merged so that I could link to it from this new page.

• [x] Addresses comment in https://github.com/pydata/xarray/pull/6975#issuecomment-1246487152
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

What is your issue?

Our docs have a general issue with having lots of related resources that are not necessarily linked together in a useful way. This results in users (including myself!) getting "stuck" in one part of the docs and being unaware of material that would help them solve their specific issue.

To give a concrete example, if a user wants to know about coarsen, there is relevant material:

• In the coarsen class docstring
• On the reshaping page
• On the computations page
• On the "how do I?" page
• On the tutorial repository

Different types of material are great, but only some of these resources are linked to others. Coarsen is actually pretty well covered overall, but for other functions there might be no useful linking at all, or no examples in the docstrings.

The biggest missed opportunity here is the way all the great content on the tutorial.xarray.dev repository is not linked from anywhere on the main documentation site (I believe). To address that we could either (a) integrate the tutorial.xarray.dev material into the main site or (b) add a lot more cross-linking between the two sites.

Identifying sections that could be linked and adding links would be a great task for new contributors.

Is it possible to deliberately drop data associated with a particular DataArray from memory?

Obviously da.close() exists, but what happens if you did for example python ds = open_dataset(file) da = ds[var] da.compute() # something that loads da into memory da.close() # is the memory freed up again now? ds.something() # what about now?

Also does calling python's built-in garbage collector (i.e. gc.collect()) do anything in this instance?

The context of this question is that I'm trying to resave some massive variables (~65GB each) that were loaded from thousands of files into just a few files for each variable. I would love to use @rabernat 's new rechunker package but I'm not sure how easily I can convert my current netCDF data to Zarr, and I'm interested in this question no matter how I end up solving the problem.

I don't currently have a particularly good understanding of file I/O and memory management in xarray, but would like to improve it. Can anyone recommend a tool I can use to answer this kind of question myself on my own machine? I suppose it would need to be able to tell me the current memory usage of specific objects, not just the total memory usage.

(@johnomotani you might be interested)

What happened?

I released xarray version v2023.07.0 last night, but I'm not seeing changes to the documentation reflected in the https://docs.xarray.dev/en/stable/ build. (In particular the Internals section now should have an entire extra page on wrapping chunked arrays.) I can however see the newest additions on https://docs.xarray.dev/en/latest/ build. Is that how it's supposed to work?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Builds upon #7911

• [x] Documentation for #7019
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

What is your issue?

(Follow-up to #7019)

When you create a dask graph of xarray operations, the tasks in the graph get useful names according the name of the DataArray they operate on, or whether they represent an open_dataset call.

Currently for cubed this doesn't work, for example this graph from https://github.com/pangeo-data/distributed-array-examples/issues/2#issuecomment-1533852877:

cc @tomwhite @dcherian

What happened?

to_zarr threw an error when I tried to overwrite part of an existing zarr store.

What did you expect to happen?

With mode w I was expecting it to overwrite part of the store with no complaints.

I expected that because that's what the docstring of to_zarr says:

mode ({"w", "w-", "a", "r+", None}, optional) – Persistence mode: “w” means create (overwrite if exists); “w-” means create (fail if exists); “a” means override existing variables (create if does not exist);

The default mode is "w", so I was expecting it to overwrite.

Minimal Complete Verifiable Example

```Python import xarray as xr import numpy as np np.random.seed(0)

ds = xr.Dataset() ds["data"] = (['x', 'y'], np.random.random((100,100))) ds.to_zarr("test.zarr") print(ds["data"].mean().compute())

returns array(0.49645889) as expected

ds = xr.open_dataset("test.zarr", engine='zarr', chunks={}) ds["data"].mean().compute() print(ds["data"].mean().compute())

still returns array(0.49645889) as expected

ds.to_zarr("test.zarr", mode="a") ```

python <xarray.DataArray 'data' ()> array(0.49645889) <xarray.DataArray 'data' ()> array(0.49645889) Traceback (most recent call last): File "/home/tom/Documents/Work/Code/experimentation/bugs/datatree_nans/mwe_xarray.py", line 16, in <module> ds.to_zarr("test.zarr") File "/home/tom/miniconda3/envs/xrdev3.9/lib/python3.9/site-packages/xarray/core/dataset.py", line 2091, in to_zarr return to_zarr( # type: ignore File "/home/tom/miniconda3/envs/xrdev3.9/lib/python3.9/site-packages/xarray/backends/api.py", line 1628, in to_zarr zstore = backends.ZarrStore.open_group( File "/home/tom/miniconda3/envs/xrdev3.9/lib/python3.9/site-packages/xarray/backends/zarr.py", line 420, in open_group zarr_group = zarr.open_group(store, **open_kwargs) File "/home/tom/miniconda3/envs/xrdev3.9/lib/python3.9/site-packages/zarr/hierarchy.py", line 1389, in open_group raise ContainsGroupError(path) zarr.errors.ContainsGroupError: path '' contains a group

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

I would like to know what the intended result is supposed to be here, so that I can make sure datatree behaves the same way, see https://github.com/xarray-contrib/datatree/issues/168.

Environment

Main branch of xarray, zarr v2.13.3

Initial work on integrating the multi-dimensional dask-powered histogram functionality from xhistogram into xarray. Just working on the skeleton to fit around the histogram algorithm for now, to be filled in later.

• [x] Closes #4610
• [x] API skeleton
• [x] Input checking
• [ ] Internal blockwise algorithm from https://github.com/xgcm/xhistogram/pull/49
• [x] Redirect plot.hist
• [x] da.weighted().hist()
• [ ] Tests added for results
• [x] Hypothesis tests for different chunking patterns
• [ ] Examples in documentation
• [ ] Examples in docstrings
• [x] Type hints (first time trying these so might be wrong)
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst
• [x] Range argument
• [ ] Handle multidimensional bins (for a future PR? - See https://github.com/xgcm/xhistogram/pull/59)
• [ ] Handle np.datetime64 dtypes by refactoring to use np.searchsorted (for a future PR? See discussion)
• [ ] Fast path for uniform bin widths (for a future PR? See suggestion)

Question: da.hist() or da.histogram()?

I realised there is very little documentation on Coarsen.construct, so I added this example.

Unsure whether it should instead live in the page on reshaping and reorganising data though, as it is essentially a reshape operation. EDIT: Now on the reshape page

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

cc @jbusecke @paigem

• [ ] From https://github.com/pydata/xarray/pull/7019#discussion_r968606140
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

We want to encourage people to use and extend xarray, and we already provide testing functions as public API to help with this.

One function I keep using when writing code which uses xarray is xarray.tests.test_dataset.create_test_data(). This is very useful for quickly writing tests for the same reasons that it's useful in xarray's internal tests, but it's not explicitly public API. This means that there's no guarantee it won't change/disappear, which is not ideal if you're trying to write a test suite for separate software. But so many tests in xarray rely on it that presumably it's not going to get changed.

Is there any reason why it shouldn't be public API? Is there something I should use instead?

Also reorganised the logic that deals with combining mixed sets of objects (i.e. named dataarrays, unnamed dataarrays, datasets) that was added in #4696.

TODO - same reorganisation / testing but for combine_nested as well as combine_by_coords. EDIT: I'm going to do this in a separate PR, so that this bugfix can be merged without it.

• [x] Closes #5833
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Rectifies the the issue in #5843 by making DataArray.chunks and Variable.chunks consistent with Dataset.chunks. This would obviously need a deprecation cycle before it were merged.

Currently a WIP - I changed the behaviour but this obviously broke quite a few tests and I haven't looked at them yet.

• [x] Closes #5843
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Updated the release guide to account for what is now automated via github actions, and any other bits I felt could be clearer.

Now only 16 easy steps!

• Motivated by #5232 and #5244
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Problem description

apply_ufunc with dask='parallelized' appears to be trying to act on an empty numpy array when the computation is specified, but before .compute() is called. In other words, a ufunc which just prints the shape of its argument will print (0,0) then print the correct shape once .compute() is called.

Minimum working example

```python import numpy as np import xarray as xr

def example_ufunc(x): print(x.shape) return np.mean(x, axis=-1)

def new_mean(da, dim): result = xr.apply_ufunc(example_ufunc, da, input_core_dims=[[dim]], dask='parallelized', output_dtypes=[da.dtype]) return result

shape = {'t': 2, 'x':3} data = xr.DataArray(data=np.random.rand(*shape.values()), dims=shape.keys()) unchunked = data chunked = data.chunk(shape)

actual = new_mean(chunked, dim='x') # raises the warning print(actual)

print(actual.compute()) # does the computation correctly ```

Result

(0, 0) /home/tnichol/anaconda3/envs/py36/lib/python3.6/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice. out=out, **kwargs) <xarray.DataArray (t: 2)> dask.array<shape=(2,), dtype=float64, chunksize=(2,)> Dimensions without coordinates: t (2, 3) <xarray.DataArray (t: 2)> array([0.147205, 0.402913]) Dimensions without coordinates: t

Expected result

Same thing without the (0,0) or the numpy warning.

Output of xr.show_versions()

(my xarray is up-to-date with master)

MCVE Code Sample

```python import xarray as xr import numpy as np

x_coord = xr.DataArray(data=[0.1, 0.2], dims=['x']) t_coord = xr.DataArray(data=[10, 20], dims=['t'])

da = xr.DataArray(data=np.array([[0, 1], [5, 9]]), dims=['x', 't'], coords={'x': x_coord, 'time': t_coord}) print(da)

da.transpose('time', 'x') Output: <xarray.DataArray (x: 2, t: 2)> array([[0, 1], [5, 9]]) Coordinates: * x (x) float64 0.1 0.2 time (t) int64 10 20

Traceback (most recent call last): File "mwe.py", line 22, in <module> da.transpose('time', 'x') File "/home/tegn500/Documents/Work/Code/xarray/xarray/core/dataarray.py", line 1877, in transpose "permuted array dimensions (%s)" % (dims, tuple(self.dims)) ValueError: arguments to transpose (('time', 'x')) must be permuted array dimensions (('x', 't')) ```

As 'time' is a coordinate with only one dimension, this is an unambiguous operation that I want to perform. However, because .transpose() currently only accepts dimensions, this fails with that error.

This causes bug in other parts of the code - for example I found this by trying to plot this type of dataarray: python da.plot(x='time', hue='x') which gives the same error.

(You can get a similar error also with da.plot(y='time', hue='x').)

If the code which explicitly checks that the arguments to transpose are dims and not just coordinate dimensions is removed, then both of these examples work as expected.

I would like to generalise the transpose function to also accept dimension coordinates, is there any reason not to do this?

• [x] Closes #2662

Explained in #2662. Also renamed some variables slightly for clarity.

Not sure how to add a test without refactoring the groupby into a separate function, as in it's current form the problem only manifests as a (huge) slowdown.

When I try to plot the values in a 1D DataArray against the values in one of its coordinates, it does not behave at all as expected:

```python import numpy as np import matplotlib.pyplot as plt from xarray import DataArray

current = DataArray(name='current', data=np.array([5, 8, 14, 22, 30]), dims=['time'], coords={'time': (['time'], np.array([0.1, 0.2, 0.3, 0.4, 0.5])), 'voltage': (['time'], np.array([100, 200, 300, 400, 500]))})

print(current)

Try to plot current against voltage

current.plot.line(x='voltage') plt.show() ``` Output:

<xarray.DataArray 'current' (time: 5)> array([ 5, 8, 14, 22, 30]) Coordinates: * time (time) float64 0.1 0.2 0.3 0.4 0.5 voltage (time) int64 100 200 300 400 500

Problem description

Not only is 'voltage' not on the x axis, but 'current' isn't on the y axis either!

Expected Output

Based on the documentation (and common sense) I would have expected it to plot voltage on the x axis and current on the y axis.

(using a branch of xarray which is up-to-date with master)