• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Notes: * For duckarrays, use _dtype[_generic] * For actual np.ndarrays, use np.dtype[np.generic] * np.dtype is too specific in general and it's probably not needed in most of their array_api functions. * _DTypeBase-class in np? * Mixing dtypes from np and xp is discouraged: https://github.com/data-apis/array-api/issues/582 * Using asarray seems to be the recommended way.

What happened?

When doing operations on numpy arrays and xarray variables mypy does not understand that the output is always a xarray variable regardless of the order. See example.

What did you expect to happen?

mypy to pass for the example code.

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

x = np.array([1, 2, 4]) v = xr.Variable(["x"], x)

numpy first:

xv = x * v xv.values # error: "ndarray[Any, dtype[bool_]]" has no attribute "values" [attr-defined] if isinstance(xv, xr.Variable): xv.values

variable first:

vx = v * x vx.values if isinstance(vx, xr.Variable): vx.values ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

Seen in #7741

Environment

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

A drive-by PR as I was trying to figure out how these functions works.

This adds a line plotter based on LineCollections, called .lines at the moment.

I wanted to replace darray.plot() with using LineCollection instead. But unfortunately due to how many cases are supported (and tested in xarray) darray.plot() will continue using plt.plot.

xref:

4820

5622

Enforce typing on all tests in test_plot.py and add the remaining type hints.

What happened?

ffill, bfill reindex etc. have tolerance arguments that also supports strings. And we test for it here:

https://github.com/pydata/xarray/blob/2120808bbe45f3d4f0b6a01cd43bac4df4039092/xarray/tests/test_groupby.py#L2016-L2025

But our typing assumes it's floats only: https://github.com/pydata/xarray/blob/2120808bbe45f3d4f0b6a01cd43bac4df4039092/xarray/core/resample.py#L69-L94

What did you expect to happen?

Since our pytests pass, mypy should pass as well.

Minimal Complete Verifiable Example

```python import numpy as np import pandas as pd

import xarray as xr

https://github.com/pydata/xarray/blob/2120808bbe45f3d4f0b6a01cd43bac4df4039092/xarray/tests/test_groupby.py#L2016

Test tolerance keyword for upsample methods bfill, pad, nearest

times = pd.date_range("2000-01-01", freq="1D", periods=2) times_upsampled = pd.date_range("2000-01-01", freq="6h", periods=5) array = xr.DataArray(np.arange(2), [("time", times)])

Forward fill

actual = array.resample(time="6h").ffill(tolerance="12h") expected = xr.DataArray([0.0, 0.0, 0.0, np.nan, 1.0], [("time", times_upsampled)]) xr.testing.assert_identical(expected, actual)

```

Environment

master

Enforce typing on all tests in test_groupby.py and add the remaining type hints.

Upstream numpy has become a bit more particular with which types you can use for inplace operations. This PR fixes ``` __ TestImshow.test_imshow_rgb_values_in_valid_range __

self = <xarray.tests.test_plot.TestImshow object at 0x7f88320c2780>

def test_imshow_rgb_values_in_valid_range(self) -> None:
    da = DataArray(np.arange(75, dtype="uint8").reshape((5, 5, 3)))
    _, ax = plt.subplots()

  out = da.plot.imshow(ax=ax).get_array()

/home/runner/work/xarray/xarray/xarray/tests/test_plot.py:2034:

/home/runner/work/xarray/xarray/xarray/plot/accessor.py:421: in imshow return dataarray_plot.imshow(self._da, args, kwargs) /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1601: in newplotfunc primitive = plotfunc( /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1853: in imshow alpha = 255

self = masked_array( data=[[[1], [1], [1], [1], [1]],

    [[1],
     [1],
...,
     [1],
     [1],
     [1],
     [1]]],

mask=False, fill_value=np.int64(999999), dtype=uint8) other = 255

def __imul__(self, other):
    """
    Multiply self by other in-place.

    """
    m = getmask(other)
    if self._mask is nomask:
        if m is not nomask and m.any():
            self._mask = make_mask_none(self.shape, self.dtype)
            self._mask += m
    elif m is not nomask:
        self._mask += m
    other_data = getdata(other)
    other_data = np.where(self._mask, other_data.dtype.type(1), other_data)

  self._data.__imul__(other_data)

E numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'multiply' output from dtype('int64') to dtype('uint8') with casting rule 'same_kind'

/home/runner/micromamba/envs/xarray-tests/lib/python3.12/site-packages/numpy/ma/core.py:4415: UFuncTypeError ```

Some curious behaviors seen while debugging: ```python alpha = np.array([1], dtype=np.int8) alpha *= 255 repr(alpha) # 'array([-1], dtype=int8)'

alpha = np.array([1], dtype=np.int16) alpha *= 255 repr(alpha) # 'array([255], dtype=int16)' ```

xref: #8844

What happened?

I have just updated my development branch of xarray to latest main. No other changes. When using .chunk() on a Variable xarray crashes.

What did you expect to happen?

No crash

Minimal Complete Verifiable Example

```Python import numpy as np import pandas as pd import xarray as xr

t_size = 8000 t = np.arange(t_size) var = xr.Variable(dims=("T",), data=np.random.randn(t_size)).chunk() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python Traceback (most recent call last):

File "C:\Users\J.W\AppData\Local\Temp\ipykernel_6480\4053253683.py", line 8, in <cell line: 8> var = xr.Variable(dims=("T",), data=np.random.randn(t_size)).chunk()

File "C:\Users\J.W\Documents\GitHub\xarray\xarray\core\variable.py", line 1249, in chunk chunkmanager = guess_chunkmanager(chunked_array_type)

File "C:\Users\J.W\Documents\GitHub\xarray\xarray\core\parallelcompat.py", line 87, in guess_chunkmanager raise ValueError(

ValueError: unrecognized chunk manager dask - must be one of: [] ```

Anything else we need to know?

Likely from #7019.

Environment

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

python import xarray as xr airtemps = xr.tutorial.open_dataset("air_temperature") airtemps.air[:200].isel(lat=10, lon=10).plot.scatter(x="time", hue="time")

Before:

From https://numpy.org/devdocs/numpy_2_0_migration_guide.html#numpy-2-migration-guide

This adds typing to the dict values. Using variables as keys is apparently not permitted by mypy so that part has been removed.

Attempts to fix issues found in #5662.

This PR makes it easier to trigger the upstream CI and creates a mypy test for upstream as well, but only when adding this label so that it doesn't mess with the scheduled runs and those results.

Noticed the need in #7786.

I noticed that the normal notation for fig, axs = plt.subplots(2, 1) wasn't used in facetgrid so did a quick find replace to change that. This feels better for my pedantic brain at least but I'm not sure it's worth the effort?

• [x] Closes #xxxx
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Add scatter plots for dataarrays. Then later remove the dataset version and replace it with a thin wrapper in a similar fashion to #4820.

New stuff: - [x] Legend now displays colors and sizes in a similar fashion to Seaborn. - [x] New parameter z which allows 3d scatter plots.

Example: Using ds = xr.tutorial.scatter_example_dataset():

Waiting on: - https://github.com/numba/numba/issues/8304 - https://github.com/numba/numba/issues/8841 - https://github.com/zarr-developers/numcodecs/issues/377 - https://github.com/Unidata/netcdf4-python/issues/1204 - https://github.com/h5py/h5py/issues/2146

• [ ] Closes #7316
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Feel free to push to this PR if you want.

Add a new what's new section

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

```python import matplotlib.pyplot as plt import xarray as xr

airtemps = xr.tutorial.open_dataset("air_temperature") air = airtemps.air - 273.15 air1d = air.isel(lat=10, lon=10)

plt.figure() air1d.plot() ``` Before:

After:

• [x] Closes #7907
• [x] Closes #7908
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What happened?

The CI seems to consistently crash on test_units.py now: =========================== short test summary info =========================== FAILED xarray/tests/test_units.py::TestVariable::test_aggregation[int32-method_max] - TypeError: no implementation found for 'numpy.max' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestVariable::test_aggregation[int32-method_min] - TypeError: no implementation found for 'numpy.min' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[float64-function_max] - TypeError: no implementation found for 'numpy.max' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[float64-function_min] - TypeError: no implementation found for 'numpy.min' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[int32-function_max] - TypeError: no implementation found for 'numpy.max' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[int32-function_min] - TypeError: no implementation found for 'numpy.min' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[int32-method_max] - TypeError: no implementation found for 'numpy.max' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_aggregation[int32-method_min] - TypeError: no implementation found for 'numpy.min' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_unary_operations[float64-round] - TypeError: no implementation found for 'numpy.round' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataArray::test_unary_operations[int32-round] - TypeError: no implementation found for 'numpy.round' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataset::test_aggregation[int32-method_max] - TypeError: no implementation found for 'numpy.max' on types that implement __array_function__: [<class 'pint.util.Quantity'>] FAILED xarray/tests/test_units.py::TestDataset::test_aggregation[int32-method_min] - TypeError: no implementation found for 'numpy.min' on types that implement __array_function__: [<class 'pint.util.Quantity'>] = 12 failed, 14880 passed, 1649 skipped, 146 xfailed, 68 xpassed, 574 warnings in 737.19s (0:12:17) = For more details: https://github.com/pydata/xarray/actions/runs/5438369625/jobs/9889561685?pr=7955

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Towards the strict goal in #8239.

Add overloads to .get_axis_num because you will get the same type out as you put in.

Seen in #8344.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I noticed that seaborn's plot was responding much faster than xarray's version with the same data. Turn's out seaborn drops any nulls: https://github.com/mwaskom/seaborn/blob/056413d7393e3daec597d430c076e45938d53376/seaborn/relational.py#L399

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Small change to avoid triggering several if-checks unnecessarily.

This allows adding a dimension of an array. Which is relevant when for example using keepdims in NamedArray.reduce.

Normally you can do this with None but that's not really allowed in the getitem documentation currently https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.array.getitem.html

Should follow the standard but with an additional dim argument, dim should be preferred over the axis argument. https://data-apis.org/array-api/draft/API_specification/generated/array_api.expand_dims.html

xref #8344, #8406

Currently using getitem is not allowed for either a arrayfunction or arrayapi according to the type hints. Add support for it.

Issues found in #8344 as more and more typing was added. But getitem is used in main as well which should fail, but manages to pass due to lack of type hints: https://github.com/pydata/xarray/blob/9acc411bc7e99e61269eadf77e96b9ddd40aec9e/xarray/namedarray/core.py#L792

Notes * mypy will now complain for _arrayapi[None] or _arrayapi([None,)], use xp.expand_dims(_arrayapi) for now instead.

https://github.com/numpy/numpy/pull/25022 https://github.com/data-apis/array-api/pull/687

Using a different TypeVar strategy compared to #8281. The idea here is to typevar shape and dtype instead, just like numpy does.

Previously I tried to use the _data array as the TypeVar but that causes all kinds of issues since TypeVar is usually invariant and can't be updated to a new type. Since the dtype changes very frequently when doing array operations it quickly gets difficult to pass along the correct typing.

• This PR adds a from_array function. The intention is to use that function to create NamedArrays when you are passing around ArrayLikes. The init for NamedArray will now just assume the input data is correct. At runtime at least, mypy will catch any non-supported array types. There's some precedent to this:
• numpy.array_api.Array forces to use xp.asarray.
• Cubed assumes the inputs are correct. Has a xp.asarray and a from_array function.
• The ugly fastpath argument is therefore not needed.
• Adds a bunch of type hint classes, duckarray[ShapeType, DType] (corresponding to np.ndarray) or DuckArray[ScalarType] (corresponding to np.typing.NDArray) are the recommended ones.
• It's better to use these kinds of classes over creating is_duck_array functions with typeguards because isinstance also works on the else clause.

• This PR adds some array_api functions, the idea here is that NamedArray could also be array_api compliant.

• [x] Tests added

• [x] Closes #8291

References: https://github.com/tomwhite/cubed/blob/ea885193dd37d27917a24878b51bb086aaef5fb1/cubed/core/ops.py#L34 https://stackoverflow.com/questions/74633074/how-to-type-hint-a-generic-numpy-array https://numpy.org/doc/stable/reference/arrays.scalars.html#scalars https://github.com/numpy/numpy/blob/040ed2dc9847265c581a342301dd87d2b518a3c2/numpy/init.pyi#L1423 https://github.com/numpy/numpy/blob/040ed2dc9847265c581a342301dd87d2b518a3c2/numpy/_typing/_array_like.py#L32 https://stackoverflow.com/questions/69186176/determine-if-subclass-has-a-base-classs-method-implemented-in-python

Check if the Array object from numpy.array_api is a valid duckarray. If it isn't we're likely doing something wrong.

Comments: * array_api standard does not specify a __array__-method. * Move __array__ to _arrayfunction only * We therefore have to use np.asarray(namedarray.data) when comparing ndarrays in the tests

Seen in #8344

Add a test for masked array in order to make sure it works. Be consistent and use the shapetype dtype generics style instead.

Starting to think it's easier to just stick to one style of typing, even though the other option might be a little shorter.

Use _DType_co instead, otherwise the dtype wont be correctly passed along. Quick follow up to #8387.

Fixes the following pyright error on main:

C:\Users\J.W\Documents\GitHub\xarray\xarray\namedarray\core.py:846:39 - error: Argument of type "NDArray[Any]" cannot be assigned to parameter "data" of type "duckarray[_ShapeType_co@NamedArray, _DType_co@NamedArray] | Default" in function "_replace"   Type "NDArray[Any]" cannot be assigned to type "duckarray[_ShapeType_co@NamedArray, _DType_co@NamedArray] | Default"     "NDArray[Any]" is incompatible with protocol "_arrayfunction[_ShapeType_co@NamedArray, _DType_co@NamedArray]"       Type "(dtype: None = ..., /) -> ndarray[Any, _dtype]" cannot be assigned to type "(dtype: _DType@__array__, /) -> ndarray[Any, _DType@__array__]"         Parameter 1: type "_DType@__array__" cannot be assigned to type "None"           Type cannot be assigned to type "None"       Type "(dtype: None = ..., /) -> ndarray[Any, _dtype]" cannot be assigned to type "(dtype: _DType@__array__, /) -> ndarray[Any, _DType@__array__]"         Parameter 1: type "_DType@__array__" cannot be assigned to type "None"           Type cannot be assigned to type "None" ... (reportGeneralTypeIssues)

Fixes:

``` Running Sphinx v6.2.1

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8387/lib/python3.10/site-packages/sphinx/config.py", line 353, in eval_config_file code = compile(f.read(), filename.encode(fs_encoding), 'exec') File "/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/8387/doc/conf.py", line 240 navigation_with_keys=False, ^^^^^^^^^^^^^^^^^^^^^^^^^^ SyntaxError: keyword argument repeated: navigation_with_keys

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8387/lib/python3.10/site-packages/sphinx/cmd/build.py", line 280, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8387/lib/python3.10/site-packages/sphinx/application.py", line 207, in init self.config = Config.read(self.confdir, confoverrides or {}, self.tags) File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8387/lib/python3.10/site-packages/sphinx/config.py", line 177, in read namespace = eval_config_file(filename, tags) File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8387/lib/python3.10/site-packages/sphinx/config.py", line 357, in eval_config_file raise ConfigError(msg % err) from err sphinx.errors.ConfigError: There is a syntax error in your configuration file: keyword argument repeated: navigation_with_keys (conf.py, line 240)

Configuration error: There is a syntax error in your configuration file: keyword argument repeated: navigation_with_keys (conf.py, line 240) Command time: 0s Return: 2

```

Seen in #8387

xref: #8376

Add more chunkedarray typing options. Also clean up some ideas that didn't work out. Using TypeVar for arrays for example has turned out to not work that great because of the very common dtype changes.

If you want to check if an ArrayLike is a chunked array: python if isinstance(data, _chunkedarrayfunction_or_api): data.chunk() However try to rely on mypy in private functions over isinstance checks for better performance: python def _do_compute(data: chunkedduckarray[Any, _DType]) -> duckarray[Any, _DType]: return data.compute()

ndim is usually defined as len(self.shape), which does not care about the values in the shape-tuple. So there's no risk when getting unknown shape sizes from for example a masked dask array.

Reference: https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.ndim.html

Use the normal repr now that NamedArray has one.
Noticed a bug as well, that sometimes the repr attempts to call NamedArray._in_memory which doesn't exist.

Seen in #8344.

Noticed an unnecessary for loop when playing around with self.get_axis_num in #8344.

With more realistic array sizes maybe the improvements in #8339 should be noticeable.

Reverts a few things from #8313. An IndexVariable can only have PandasIndexAdapter as self._data, which makes the for MemoryCachedArray unnecessary.

Seen in #8294.

Seen in #8294.

The issue is the IndexVariable, ExplicitlyIndexedNDArrayMixin lacks .array which is required for IndexVariable, and therefore we need a new minimal class that are common between the two.

Seen in #8294.

These tests implicitly made sure the type was correct in a way that type checkers wont understand. Make it explicit instead.

Seen in #8294.

It's usually better to switch if-checks so that the wider type is defined first. Especially when relying on the implicit type casting like we do here.

This is how our get_array_namespace works: https://github.com/pydata/xarray/blob/dafd726c36e24ac77427513a4a149a6933353b66/xarray/core/duck_array_ops.py#L44-L48

Which usually works. But not for astype.

Using np.array_api doesn't work because you have to use np.array_api.Array instead of np.ndarray:

```python import numpy.array_api as nxp nxp.astype(np.array([1, 2,]), np.dtype(float))

Traceback (most recent call last):

File "C:\Users\J.W\AppData\Local\Temp\ipykernel_8616\23329947.py", line 1, in <cell line: 1> nxp.astype(np.array([1, 2,]), np.dtype(float))

File "C:\Users\J.W\anaconda3\envs\xarray-tests\lib\site-packages\numpy\array_api_data_type_functions.py", line 20, in astype return Array._new(x._array.astype(dtype=dtype, copy=copy))

AttributeError: 'numpy.ndarray' object has no attribute '_array' ```

I found it simpler to just change astype here.

An alternative solution would be to use: https://github.com/data-apis/array-api-compat https://github.com/tomwhite/cubed/pull/317

Seen in #8294.

These methods are not in the Array API. Instead convert the methods to functions in similar fashion as the array api.

https://data-apis.org/array-api/latest/API_specification/index.html

Not sure how to handle array compliant functions with an axis argument (max for example) but that's for a future PR.

The Idea is to avoid as much normalization in the NamedArray class as possible. Different types are handled before initializing instead.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

References: https://github.com/tomwhite/cubed/blob/ea885193dd37d27917a24878b51bb086aaef5fb1/cubed/core/ops.py#L34 https://stackoverflow.com/questions/74633074/how-to-type-hint-a-generic-numpy-array https://numpy.org/doc/stable/reference/arrays.scalars.html#scalars https://github.com/numpy/numpy/blob/040ed2dc9847265c581a342301dd87d2b518a3c2/numpy/init.pyi#L1423 https://github.com/numpy/numpy/blob/040ed2dc9847265c581a342301dd87d2b518a3c2/numpy/_typing/_array_like.py#L32

Mypy issues: https://github.com/python/typing/issues/548

It can be helpful to get a second opinion from another typer checker than mypy.

The CI activates when the run-pyright label is added.

This seems to be fixed according to #6551.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Reducing type errors in #8241

Binding allows typing the .data property.

A smaller PR related to trying to pass T_DuckArray's along xarray's different classes.

The typing of Variable.data has been the very wide Variable.data: Any because of the difficulties defining the typing of a duckarray.

This has led to confusion in downstream functions that uses .data. Sometimes arguments uses Any, np.typing.ArrayLike, np.ndarray or dask array typing which is unnecessarily narrow when xarray also supports for example sparse arrays or cupy arrays.

This PR is a start at cleaning this up by defining a T_DuckArray typevar that symbolizes an array api compliant array.

Matplotlib has started to support typing in main (https://github.com/matplotlib/matplotlib/issues/20504) and mypy is throwing a few errors:

Some guidance how to solve these:

• [xy]ticks in mpl is currently overly narrowly type hinted because I was following the docstring, but I agree that ArrayLike is a better type hint for that, plan on updating (including the docstring) upstream
• [xy]lim originally neglected the case of passing set_xlim((min, max)) as a tuple, but that has been updated. xarray has that type hinted as array like, but mpl has it hinted as a 2-tuple (I think it is currently still of floats, but may be expanded as we more directly address units/categoricals/etc). Willing to debate here, but my starting position is that the "exactly 2 values" is valuable info here, and I think tuple is the only way to do that.
• get_renderer is not actually available on all of our backends, we should maybe see if there is a more preferred way of doing what you are doing here that will work for all backends, but haven't looked into it too closely.
• Module has no attribute <colormap> is another instance of dynamically generated behavior which can't be statically type checked (elegantly, at least), can probably be replaced by mpl.colormaps["<colormap>"] in many cases, which is statically typecheckable
• Anything to do with 3D Axes is not type hinted, perhaps ignore for now (or help us get that type hinted adequately, but it is relatively low priority currently)
• Module has no attribute "dates" we don't currently type hint dates/units things, but it is on my mind, not sure yet if it will be in first release or not though (may at least put a placeholder that gets rid of this error, but treats everything as "Any").

Originally posted by @ksunden in https://github.com/pydata/xarray/issues/7787#issuecomment-1523743471

The suggestion from mpl (specifically @tacaswell) was to use constrained layout for the purpose that xarray currently uses get_renderer, this will ensure that the facetgrid works with all mpl backends.

Originally posted by @ksunden in https://github.com/pydata/xarray/issues/7787#issuecomment-1528091492

I'm also relatively sure that if you are willing to put a floor on the version of Matplotlib you support get_window_extent will use it's internally cached renderer (and when we make it uniformly optional we also fixed the cache invalidation logic).

Originally posted by @tacaswell in https://github.com/pydata/xarray/issues/7787#issuecomment-1528096647

Fixes the mypy error we're seeing in the CI at the moment:

xarray/core/dataset.py:5286: error: Value of type variable "SupportsRichComparisonT" of "sorted" cannot be "Hashable" [type-var]

Can't sort Hashables apparently.

What happened?

I'm not sure if this is a bug or a feature but I was expecting this example to work since the new coord is just a slight rewrite of the original dimension coordinate:

```python import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature")

Change the first time value:

ds["air_new"] = ds.air.copy() air_new_changed = ds.air_new[{"time": 0}] * 3 ds.air_new.loc[air_new_changed.coords] = air_new_changed # Works! :)

Add a another coord along time axis and change

the first time value:

ds["air_new"] = ds.air.copy().assign_coords( {"time_float": ds.time.astype(float)} ) air_new_changed = ds.air_new[{"time": 0}] * 4 ds.air_new.loc[air_new_changed.coords] = air_new_changed # Error! :(

Traceback (most recent call last):

Cell In[25], line 5 ds.air_new.loc[air_new_changed.coords] = air_new_changed

File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\dataarray.py:222 in setitem dim_indexers = map_index_queries(self.data_array, key).dim_indexers

File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\indexing.py:182 in map_index_queries grouped_indexers = group_indexers_by_index(obj, indexers, options)

File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\indexing.py:144 in group_indexers_by_index raise KeyError(f"no index found for coordinate {key!r}")

KeyError: "no index found for coordinate 'time_float'" ```

It's been unclear to me what methods are necessary to implement or not. I think decorating with @abstractmethod will help with that. It's a breaking change though and it could be disruptive.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

References: https://github.com/cvxpy/cvxpy/pull/1810

Temporally skip tests in order to make CI pass again, the troublesome tests are reported in #7971.

What happened?

The upstream CI has been failing for a while. Here's the latest: https://github.com/pydata/xarray/actions/runs/5501368493/jobs/10024902009#step:7:16

python Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/runner/work/xarray/xarray/xarray/__init__.py", line 1, in <module> from xarray import testing, tutorial File "/home/runner/work/xarray/xarray/xarray/testing.py", line 7, in <module> import numpy as np ModuleNotFoundError: No module named 'numpy'

Digging a little in the logs ``` Installing build dependencies: started Installing build dependencies: finished with status 'error' error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [3 lines of output] Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple ERROR: Could not find a version that satisfies the requirement meson-python==0.13.1 (from versions: none) ERROR: No matching distribution found for meson-python==0.13.1 [end of output] ```

Might be some numpy problem?

Should the CI be robust enough to handle these kinds of errors? Because I suppose we would like to get the automatic issue created anyway?

Use https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/ instead.

• [x] Closes #7969

A simple refactor to make it easier to retrieve the proper paths that open_mfdataset uses and passes on the engine.

I've been thinking how to make use of DataTree and one idea I wanted to try was: * Open file (using_find_absolute_path). * Get all groups in the file. * For each group run xr.open_mfdataset(..., group=group)

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• Don't use python for loops for possibly large coords. Rather create a np array once then filter out bad data.
• DuckArrayModule is slightly slow, so cache the first import in a dict instead to speed up later calls.

• Add more typing to be more confident that inputs are valid and then remove redundant checks and conversions.

• [x] Requires #7843

• [x] Requires #7844.
• [x] Requires #7858
• [x] Closes #7833
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• ds.chunks loops all the variables, do it once.
• Faster to create a meta dataframe once than letting dask guess 2000 times.

Following recommendation from: https://github.com/pydata/xarray/pull/7824#discussion_r1196114696

```python

main:

a = tuple(f"dim_{i}" for i in range(500)) %timeit OrderedSet(a) 46 µs ± 2.26 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

This PR:

a = tuple(f"dim_{i}" for i in range(500)) %timeit OrderedSet(a) 28.9 µs ± 476 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) ```

Split out from #7858.

Improve mypy coverage in order to understand better what the functions are actually used for.

• Use fastpaths
• Only transpose multi-dimensional arrays.
• Calculating chunks and shapes can be costly, prefer doing it once.

Newest pint crashes our tests for some reason, pin it for now.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

The docstring seems to imply that it's possible to get in-memory arrays: https://github.com/pydata/xarray/blob/4bb9d9c6df77137f05e85c7cc6508fe7a93dc0e4/xarray/backends/api.py#L732

But it doesn't seem possible because of: https://github.com/pydata/xarray/blob/4bb9d9c6df77137f05e85c7cc6508fe7a93dc0e4/xarray/backends/api.py#L899

This PR removes that or check, changes the default to chunk={}, and fixes the failing tests.

• [x] Noticed in #5689
• [ ] Closes #7792
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [ ] Closes #7243
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Fixes typing errors when using newest mypy version.

• [x] Closes #7270

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Seems https://github.com/actions/labeler/issues/112 is fixed. So using false should work.

xref: #7431

Is your feature request related to a problem?

Use more up-to-date typing styles on all files. Will reduce number of imports and avoids big diffs when doing relatively minor changes because pre-commit/pyupgrade has been triggered somehow.

Related to #6240

Describe the solution you'd like

Add from __future__ import annotations on files with a lot of typing. Let pyupgrade do the rest.

Describe alternatives you've considered

No response

Additional context

No response

• [x] Closes #7314

• [x] Closes #7268

• Matplotlib adds default colors with plt.plot(x, y) when y is a 2D array.

• Matplotlib prioritizes c over default colors with plt.plot(x, y, c="blue") when y is a 2D array.
• Seaborn prioritizes hue over c, but all plot axes has to be explicit.

The plot1d function now follows closer to seaborns example and require explicitly defining optional dimensions such as hue/markersize. Positives and negatives: +++ Reliable performance, won't accidentally choose large and heavy arrays to plot. - Loss of dimension information for default values, will essentially flatten the n-dimensional arrays. - Other xarray plot functions guesses quite a bit as well, API is not consistent.

Required dimensions like the x-axis will still be guessed since the mpl errors weren't very intuitive.

While at it: * Make it possible to prioritize certain mpl kwargs when hue/markersize isn't defined. * Fix integer coords in plot1d and facetgrid, use .coords instead of getitem.

• [x] Closes #7313
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Turns out that there's a call to .set_dims that forces a broadcast on the numpy coordinates.

• [x] Closes #6811
• [x] Tests added, see #7474.
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Debugging script:

Related to #7472.

Was hunting some low-hanging performance fruits when reading in files.

• Use Variable(..., fastpath=True) for cases when a Variable has been unpacked and modified slightly.
• Don't check if variable is variable in decode_cf_variable
• Don't import DataArray until necessary in as_compatible_data.

• Add typing to touched files to make sure only Variables are used.

• [ ] Closes #xxxx

• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#indexing-examples

I believe there's a typo in the BASIC indexing support example: ```python

shall support integers

backend_array._raw_indexing_method(1, 1) ```

Should be: ```python

shall support integers

backend_array._raw_indexing_method((1, 1)) ```

Suggestion of possible fixes: * Make sure it is a typo. * Create a valid custom MyBackendArray and initialize it. So it is easier to tell if it's a typo. * Add type hinting so mypy can easier catch these errors.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

I was making sure the test in #6832 failed on main, when it did I wrote a few lines in the what's new file but forgot switching back to the other branch and accidentally pushed directly to main. :(

Describe the solution you'd like

I think it's best if we require a pull request for merging. We seem to pretty much do this anyway.

Seems to be this setting if I understand correctly:

Describe alternatives you've considered

No response

Additional context

No response

This tests xr.open_dataset without any slow file reading that can quickly become the majority of the performance time.

Related to #7374.

Timings for the new ASV-tests: ```

[ 50.85%] ··· dataset_io.IOReadCustomEngine.time_open_dataset ok [ 50.85%] ··· ======== ============ chunks
-------- ------------ None 265±4ms
{} 1.17±0.02s ======== ============ [ 54.69%] ··· dataset_io.IOReadSingleFile.time_read_dataset ok [ 54.69%] ··· ========= ============= ============= -- chunks
--------- --------------------------- engine None {}
========= ============= ============= scipy 4.81±0.1ms 6.65±0.01ms netcdf4 8.41±0.08ms 10.9±0.2ms ========= ============= ============= ``` From the IOReadCustomEngine test we can see that chunking datasets with many variables (2000+) is considerably slower.

• Workaround for Pull Request Labeler The PR labeler keeps removing manually added labels. xref: https://github.com/actions/labeler/issues/112
• ASV benchmarks starts also when topic-performance label is added now. Bot is allowed to change this one, but not run-benchmarks.

I saw dask has started using absolute imports in https://github.com/dask/dask/pull/8796. I find it much more readable and there's a nice pre-commit for it as well.

Easiest way to deal with the merge conflicts is probably to just accept your changes and let pre-commit fix them afterwards.

This reverts some of commit 6e77f5e8942206b3e0ab08c3621ade1499d8235b and #7204.

Apparently using it on all folders is not a good idea, follow pandas example.

dask has added py.typed files so now the ignores shouldn't be needed anymore: https://github.com/dask/dask/pull/8854 https://github.com/dask/distributed/pull/5328 As does numpy and pint

pycompat.py doesn't error anymore, so it's good to type check that one as well. Fixed also a python 3.8 related error in it.

.coords were still using DataArray types switch to T_DataArray instead.

• [x] Fixes mypy error seen in #7277

It's a little tricky getting the mypy version, so adding it to the show_versions function as well.

Fixes these errors that I've been seeing locally for a while:

python !mypy C:\Users\J.W\Documents\GitHub\xarray\xarray\core\coordinates.py --ignore-missing-imports C:\Users\J.W\Documents\GitHub\xarray\xarray\core\pycompat.py:48: error: Incompatible types in assignment (expression has type "Tuple[]", variable has type "Tuple[Any]") C:\Users\J.W\Documents\GitHub\xarray\xarray\core\pycompat.py:50: error: Incompatible types in assignment (expression has type "Optional[Module]", variable has type "Optional[Literal['dask', 'pint', 'cupy', 'sparse']]") Found 2 errors in 1 file (checked 1 source file)

Not sure why the CI isn't catching these?

• default markersize values of widths are 18 to 72.
• plt.scatter default markersize is 36.
• plt.plot default linewidth is 6.

With main we get 18 for constant arrays, but 36 if markersize was undefined. This seems a bit inconsistent to me. This PR adds a default value instead for constant arrays

Follow up to #7272.

Found an issue when constant values was used. Now if constant values are found it'll default to the minimum width value instead.

• [x] Closes #6722
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst