• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@aulemahal would you be able to test against xclim's test suite. I imagine you're doing a bunch of grouped quantiles.

…roadcast=False`` (#8784)"

This reverts commit 11f89ecdd41226cf93da8d1e720d2710849cd23e.

Reverting #8784

Sadly that PR broke a lot of tests by breaking create_test_data with from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

Somehow that code changes whether IndexVariable.values returns a writeable numpy array. I spent some time debugging but couldn't figure it out.

cc @etienneschalk

I haven't looked at this too closely but it appears that this is a way to save MultiIndexed datasets to netCDF. So we may be able to do sparse -> multiindex -> netCDF

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#compression-by-gathering

cc @fujiisoup

Builds on #8560

• [x] Closes #8409
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @max-sixty

``` print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).map_blocks(lambda x: x))))

779354739 -> 47699827

print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).drop_vars(da.indexes).map_blocks(lambda x: x))))

15981508

```

This is a quick attempt. I think we can generalize this to minimize duplication.

The downside is that the graphs are not totally embarrassingly parallel any more. This PR:

vs main:

Is your feature request related to a problem?

Shall we support opt_einsum as an optional backend for xr.dot?

opt_einsum.contract is a drop-in replacement for np.einsum so this monkey-patch works today xr.core.duck_array_ops.einsum = opt_einsum.contract

Describe the solution you'd like

Add a backend kwarg with options "numpy" and "opt_einsum", with the default being "numpy"

Describe alternatives you've considered

We could create a new package but it seems a bit silly.

Additional context

No response

What is your issue?

Here's the output from the Scientific Python Repo Review tool.

There's an online version here.

On mac I run pipx run 'sp-repo-review[cli]' --format html --show err gh:pydata/xarray@main | pbcopy

A lot of these seem fairly easy to fix. I'll note that there's a large number of mypy config suggestions.

General

• Detected build backend: setuptools.build_meta
• Detected license(s): Apache Software License

<table> <tr><th>?</th><th>Name</th><th>Description</th></tr> <tr style="color: red;"> <td>❌</td> <td>PY007</td> <td> Supports an easy task runner (nox or tox)

Projects must have a noxfile.py or tox.ini to encourage new contributors.

</td> </tr> </table>

PyProject

See https://github.com/pydata/xarray/issues/8239#issuecomment-1739363809

<table> <tr><th>?</th><th>Name</th><th>Description</th></tr> <tr style="color: red;"> <td>❌</td> <td>PP305</td> <td> Specifies xfail_strict

xfail_strict should be set. You can manually specify if a check should be strict when setting each xfail.

[tool.pytest.ini_options]
xfail_strict = true

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PP308</td> <td> Specifies useful pytest summary

-ra should be in addopts = [...] (print summary of all fails/errors).

[tool.pytest.ini_options]
addops = ["-ra", "--strict-config", "--strict-markers"]

</td> </tr> </table>

Pre-commit

<table> <tr><th>?</th><th>Name</th><th>Description</th></tr> <tr style="color: red;"> <td>❌</td> <td>PC110</td> <td> Uses black

Use https://github.com/psf/black-pre-commit-mirror instead of https://github.com/psf/black in .pre-commit-config.yaml

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PC160</td> <td> Uses codespell

Must have https://github.com/codespell-project/codespell repo in .pre-commit-config.yaml

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PC170</td> <td> Uses PyGrep hooks (only needed if RST present)

Must have https://github.com/pre-commit/pygrep-hooks repo in .pre-commit-config.yaml

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PC180</td> <td> Uses prettier

Must have https://github.com/pre-commit/mirrors-prettier repo in .pre-commit-config.yaml

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PC191</td> <td> Ruff show fixes if fixes enabled

If --fix is present, --show-fixes must be too.

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>PC901</td> <td> Custom pre-commit CI message

Should have something like this in .pre-commit-config.yaml:

ci:
  autoupdate_commit_msg: 'chore: update pre-commit hooks'

</td> </tr> </table>

MyPy

<table> <tr><th>?</th><th>Name</th><th>Description</th></tr> <tr style="color: red;"> <td>❌</td> <td>MY101</td> <td> MyPy strict mode

Must have strict in the mypy config. MyPy is best with strict or nearly strict configuration. If you are happy with the strictness of your settings already, ignore this check or set strict = false explicitly.

[tool.mypy]
strict = true

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>MY103</td> <td> MyPy warn unreachable

Must have warn_unreachable = true to pass this check. There are occasionally false positives (often due to platform or Python version static checks), so it's okay to ignore this check. But try it first - it can catch real bugs too.

[tool.mypy]
warn_unreachable = true

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>MY104</td> <td> MyPy enables ignore-without-code

Must have "ignore-without-code" in enable_error_code = [...]. This will force all skips in your project to include the error code, which makes them more readable, and avoids skipping something unintended.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>MY105</td> <td> MyPy enables redundant-expr

Must have "redundant-expr" in enable_error_code = [...]. This helps catch useless lines of code, like checking the same condition twice.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

</td> </tr> <tr style="color: red;"> <td>❌</td> <td>MY106</td> <td> MyPy enables truthy-bool

Must have "truthy-bool" in enable_error_code = []. This catches mistakes in using a value as truthy if it cannot be falsey.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

</td> </tr> </table>

Ruff

<table> <tr><th>?</th><th>Name</th><th>Description</th></tr> <tr style="color: red;"> <td>❌</td> <td>RF101</td> <td> Bugbear must be selected

Must select the flake8-bugbear B checks. Recommended:

[tool.ruff]
select = [
  "B",  # flake8-bugbear
]

</td> </tr> </table>

What happened?

MY prototype CRSIndex is broken on the release version: https://github.com/dcherian/crsindex/blob/main/crsindex.ipynb under heading "BROKEN: Successfully align with a copy of itself"

The cell's code is : copy = newds.copy(deep=True) xr.align(copy, newds)

which should always work.

@headtr1ck is https://github.com/pydata/xarray/pull/7140 to blame?

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:43:44) [Clang 13.0.1 ] python-bits: 64 OS: Darwin OS-release: 21.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.10.0 pandas: 1.5.0 numpy: 1.23.3 scipy: 1.9.1 netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.13.3 cftime: 1.6.2 nc_time_axis: 1.4.1 PseudoNetCDF: 3.2.2 rasterio: 1.3.2 cfgrib: 0.9.10.2 iris: 3.3.1 bottleneck: 1.3.5 dask: 2022.9.2 distributed: 2022.9.2 matplotlib: 3.6.1 cartopy: 0.21.0 seaborn: 0.12.0 numbagg: 0.2.1 fsspec: 2022.8.2 cupy: None pint: 0.19.2 sparse: 0.13.0 flox: 0.6.0 numpy_groupies: 0.9.19 setuptools: 65.5.0 pip: 22.2.2 conda: None pytest: 7.1.3 IPython: 8.5.0 sphinx: None

• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

This adds plotting methods to GroupBy objects so that it's easy to plot each group as a facet. I'm finding this super helpful in my current research project.

It's pretty self-contained, mostly just adding map_groupby* methods to FacetGrid. But that's because I make GroupBy mimic the underlying DataArray by adding coords, attrs and __getitem__.

This still needs more tests but I would like feedback on the feature and the implementation.

Example

``` python import numpy as np import xarray as xr

time = np.arange(80) da = xr.DataArray(5 * np.sin(2np.pitime/10), coords={"time": time}, dims="time") da["period"] = da.time.where((time % 10) == 0).ffill("time")/10 da.plot() ```

python da.groupby("period").plot(col="period", col_wrap=4)

python da = da.expand_dims(y=10) da.groupby("period").plot(col="period", col_wrap=4, sharex=False, sharey=True, robust=True)

• [x] Closes #2383
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Should merge after https://github.com/dask/dask/pull/7234 is merged

• [x] Closes #3277, closes #2531, closes #2532, closes #2514
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It parses args and breaks any xarray objects into appropriate blocks before passing them on to the user function.

e.g. ```python da1 = xr.DataArray( np.ones((10, 20)), dims=["x", "y"], coords={"x": np.arange(10), "y": np.arange(20)} ).chunk({"x": 5, "y": 4}) da1

def sumda(da1, da2): #print(da1.shape) #print(da2.shape) return da1 - da2

da3 = (da1 + 1).isel(x=1, drop=True).rename({"y": "k"}) mapped = xr.map_blocks(sumda, da1, args=[da3]) xr.testing.assert_equal(da1-da3, mapped) # passes ```

memory consumption errors as usual.

@keewis I remember you had an idea for using pip instead of conda?

1. python 3.8 is not allowed on RTD (yet)
2. I pinned a few versions (based on the env solution obtained locally). This seems to have fixed the memory problem.

This adds join='override' which checks that indexes along a dimension are of the same size and overwrites those indices with indices from the first object.

Definitely need help, feedback.

• [x] With #3102 this partially helps with #2039, #2217
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

partially addresses #1850 closes #946

I tried to make an "awesome xarray" list by doing a github search for 'xarray'. I only put packages that looked like they were intended for general use. Also, I moved the list from internals.rst to faq.rst.

Let me know if there are any I've missed or any that should be added.