issues

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

We encountered the issue that we should plot units as (unit) instead of [unit]. This PR enables us to do exactly this, easier to change this at the source ;)

I think setting this as a global option is the correct approach, but feel free to propose alternatives :)

Support passing instances of BackendEntryPoints as the engine argument.

Closes #8447

Then instead of passing a long list of options to the open_dataset method directly, you can also configure the entrypoint in the constructor and pass it as the engine.

It would look something like this: python engine = NetCDF4BackendEntrypoint(mode="a", clobber=False) ds = xr.open_dataset("some_file.nc", engine=engine) While this is actually even more lines of code, the main advantage is to have better discoverability of the options.

TODO:

• [x] Adapt netcdf4 backend
• [x] Adapt h5netcdf backend
• [x] Find out if h5netcdf backend should have "autoclose" and "mode" options (https://github.com/pydata/xarray/pull/8520#pullrequestreview-1769368001_)
• [x] What to do with "decode_vlen_strings" option in h5netcdf (was this deprecated?)
• [x] Adapt zarr backend
• [x] Adapt scipy backend
• [x] Adapt pydap backend
• [ ] output_grid seems to be always set to True? is this intentional, why not remove it instead?
• [x] ~verify and user_charset are non-existent in pydap?~ > I still had pydap version 3.2, in 3.4 they exist...
• [x] typing is only my first impression. Not easy if upstream libs are untyped :/
• [x] ~Adapt pynio backend~ > Won't adapt because deprecated
• [x] Fix docstrings to include init options
• [x] Check if lock=True is allowed > Not allowed, otherwise scipy backend breaks
• [ ] Change default to lock=True instead of None? Maybe a later PR?
• [ ] Rename XXXBackendEntrypoint > XXXBackend ?
• [x] ~The autoclose argument seems to do nothing?~ > Actually it is used in BaseNetCDF4Array, all good
• [x] ~Move group to open_dataset instead of backend option?~ > Its not really a decoder either. Not sure, for now leave it in the init...
• [ ] Improve _resolve_decoders_kwargs, this function has a lot of implicit assumtions? Maybe remove open_dataset_parameters alltogether?
• [x] Add tests for passing backend directly via engine argument
• [x] open_dataset now has **kwargs to support backwards compatibility. Probably we should raise if unsupported stuff is added (i.e. typos) otherwise this could be confusing? (i.e. see test in zarr that checks for deprecated auto_chunk)

• [x] Closes #8546
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Probably we should add a whole bunch of tests for this. For now only testing the constructor.

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR improves the typing of align. Before: the type of the inputs was reduced to the common superclass and the return type was the same. This often required casts or ignores when mixing classes (e.g. da, ds = xr.align(da, ds). Now: the return types are exactly the same as the input types if the number of passed arguments is <=5.

Only downside: it requires some ugly overloads with type ignores on align. Maybe someone knows how to type this better?

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Mini PR to improve the typing of curvefit. Using dict is problematic since it is invariant, while Mapping is covariant.

• [x] Closes #8448
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I guess we update manually for now?

• xref https://github.com/pydata/xarray/issues/8238

This is highly WIP and probably everything is broken right now... Just creating this now, so other people don't work on the same :) Feel free to continue here with me.

@pydata/xarray 1. what do we do with commonly used functions, is it ok to copy them? 2. Moving the typed ops requires a lot of functions to be added to NamedArray, is there a consensus of what we want to move? Is it basically everything? 3. Slowly the utils module is becomming a graveyard of stuff we dont want to put elsewhere, maybe we should at least move the typing stuff over to a types module.

• [x] Towards #8199
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This aims at making the dimenion type a generic parameter. I thought I will start with NamedArray when testing this out because it is much less interconnected.

• [x] Closes #7802
• [x] Tests added

Mainly deprecated "closed" argument in date_range and passing pd.MultiIndex directly to the constructor.

• [x] Related to #7780
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6775
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

• [x] Closes #7478
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

Don't know yet how to effectively test this, I guess there is some tricky solution with some mocking.

• [x] Closes #7141

• [x] Closes #7479

I don't know how I feel about using private numpy classes that might change anytime. Maybe within a if TYPE_CHECKING block it is not too bad?

• [x] Closes #4975

This little config change will make black remove trailing commas when they are not necessary to fit something into a single line.

It is a pure design choice but personally I like the clean up it does when function signatures simplify (although this happens rarely with more and more type hints added).

I can understand that some people prefer the manual control over what is multiline and what is not. Feel free to vote on it :)

For me it adds cheap LOCs so it looks like I am working hard, haha.

• [x] Closes #7340
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

This PR adds some typing to several abstract base classes that are used in xarray.

Most of it is working, only one major point I could not figure out:

What is the type of NDArrayMixin.array??? I would appreciate it if someone that has more insight into this would help me.

Several minor open points:

• What is the return value of ExplicitlyIndexed.__getitem__
• What is the return value of ExplicitlyIndexed.transpose
• What is the return value of AbstractArray.data
• Variable.values seems to be able to return scalar values which is incompatible with the AbstractArray definition.

Overall it seems that typing has helped to find some problems again :)

Mypy should fail for tests, I did not adopt them yet, want to solve the outstanding issues first.

• [x] Closes #7312
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

Turns out the default value of arguments is important for overloads, haha.

This PR adds a utils.parse_dims function for parsing one or more dimensions. Currently every function that accepts multiple dimensions does this by itself.

I decided to first see if it would be useful to centralize the dimension parsing and collect inputs before adding it to other functions.

• [x] Potentially closes #7181
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

I have passed along some more memo dicts, which could prevent some double deep-copying of the same data (don't know how exactly, but who knows :P) Also, I have found some copy calls that did not pass along the deep argument (I am not sure if that breaks things, lets find out). And finally I have found some places where shallow copies are enough.

All together it should improve the performance a lot when copying things around.

This PR adds the mypy option "warn_unused_ignores" which will raise an error if a # type: ignore is used where it is no longer necessary.

Should enable us to keep our types updated.

I am not sure if this will lead to many issues whenever e.g. numpy changes/improves their typing, so we might get errors whenever there is a new version. Maybe it is not that bad, or maybe we can also remove the option again and only do it manually from time to time?

• [x] Related to #7270

• [x] Hopefully Closes #6726
• [x] ~Tests added~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

• [x] Closes #7156
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

• [x] Closes #7275
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~New functions/methods are listed in api.rst~

• [x] Closes #7049
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #7158

Technically does not close all fails, but if we close it, the CI will open a new issue anyway and the discussion is not relevant anymore :)

related to https://github.com/pydata/xarray/issues/6726

• silences some user warning from renaming dims.
• fix coverage config

• [x] Closes #6949
• [x] Tests added (typing)
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6818
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

Alternative to https://github.com/pydata/xarray/pull/6821

Forgot to add this to #7112

Also, correct the date of the 2022.09 release in whats-new, see #7135

• [x] Closes #7111
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

xarray.testing.assert_identical and probably more do not work yet.

Not sure if that is the correct approach (to simply use a mypy flag) but lets see what people think about it.

While adding type hints to test_backends I noticed that the open_dataset method and the abstract BackendEntrypoint were missing stream types as inputs.

I was playing around a bit with a mypy plugin and this was the best I could come up with. Unfortunately the mypy docu about the plugins is not very detailed...

This plugin makes mypy recognize the user defined accessors.

There is a quite severe bug in there (due to my lack of understanding of mypy internals probably) which makes it work only on the first run but when you change a line in your code and run mypy again it will crash... (you can delete the cache to make it work one more time again :)

Any chance that a mypy expert can figure this out? haha

• [x] Closes #2835 (Even though technically it fixes the DataArray version of the issue)
• [x] ~~Tests added ~~ will be done via #7086
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

related to #2839

Mainly to annoy @Illviljan :)

tiny PR that disables coverage on types.py and all typing-only imports a.la. if TYPE_CHECKING: ...

This PR adds the ellipsis typehint to reductions (only where they behave differently from None to reduce overhead). Follow up on https://github.com/pydata/xarray/pull/7017#issuecomment-1243927061

Additionally I was changing a lot of "one or more dimensions" typehints to str | Iterable[Hashable] (See https://github.com/pydata/xarray/issues/6142). Some code changes were necessary to support this fully. Before several things were not working with actual hashable dimensions that are not strings.

• [x] Closes #7078
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

This PR adds an Ellipsis typehint to some functions.

Interestingly mypy did not complain at the tests before, I assume it is because "..." is Hashable or something like that?

I don't know what to do with reductions, since they also support ellipsis, but it is basically the same as using None. Therefore, I assume it is not necessary to expose this feature.

Did I miss any functions where ellipsis is supported? It is hard to look for "..."... xD

• [x] Closes #6875
• [x] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

This PR removes several warnings from the tests and improves their typing on the way.

This PR tries to get rid of several warnings in the tests.

I could not get rid of RuntimeWarning: All-NaN slice encountered for tests with dask. Does anyone know why is that? pytest.mark.filterwarnings does not seem to capture them...

• [x] Closes #6931
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

Some typing on the way :)

This PR removes the _SupportsDType dependency from numpy and introduces its own.

Closes https://github.com/pydata/xarray/issues/6818

• [x] Closes #6739
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

I have taken the freedom to move all rolling related tests into their own testing module. https://github.com/pydata/xarray/pull/6730 should then take care of the (by now) copy-pasted da and ds fixtures.

• [x] Closes #5458
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

On the way, I have added the support for changing the name and dims/coords in the same rename call.

Also took the freedom to fix some unrelated typing problems.

This PR moves all DataArrayRolling and DatasetRolling tests to their own module.

See request https://github.com/pydata/xarray/pull/6744#issuecomment-1182169308

• [x] Closes #6741
• [x] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] ~~New functions/methods are listed in api.rst~~

This PR adds typing support for groupby, coarsen, rolling, weighted and resample.

There are several open points:

1. Coarsen is missing type annotations for reductions like max, they get added dynamically.
2. The Groupby group-key type is quite wide. Does anyone have any idea on how to type it correctly? For now it is still Any.
3. Several function signatures were inconsistent between the DataArray and Dataset versions (looking at you: map). I took the freedom to align them (required for mypy), hopefully this does not break too much.
4. I was moving the generation functions from DataWithCoords to DataArray and Dataset, which adds some copy-paste of code (I tried to keep it minimal) but was unavoidable for typing support. (Adds the bonus that the corresponding modules are now only imported when required).

• [x] Closes #6714
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

Currently returns a Mapping from variable names to dtypes for ALL variables in the Dataset, including coordinates. Possibly better to only return data_vars dtypes? Give me your thoughts on that, it is easy to change.

• [x] Closes #6707
• [x] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I don't really know how to test this... Is it ok to leave it untested?

• [x] Closes #6227
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6623
• [x] Tests added

• [x] Closes #6632

So far we have not found a good way of typing dynamically added custom backends (installed via pip). So the only fallback option is to allow str.

I have decided to leave the Literal as type, since it gives the user a nice list of supported backends in the editor.

• [x] Closes #5945
• [x] Tests added

Feel free to wait with merging after the 2022.05.0.dev0 release :)

• This PR adds typing to all Dataset methods (only the ones in Dataset directly and DataWithCoords).
• On the way it fixes several typing issues and adds small code changes for more consistent typing (hopefully does not break anything).
• All test_dataset tests are now typed.
• As usual there are still some mypy bugs open that prevent all typing changes to be final (especially https://github.com/python/mypy/issues/12846). I choose to remove the TypeVars in these occasions to prevent false positives on the user side.
• Sorry for updating so many files at once, mostly these are the changes of adding from __future__ import annotations and automatic fixes from pyupdate.

If anyone has some typing expertise: I think that the way align is typed is wrong. The current implementation only works for aligning a sequence of objects of the same type, but not mixed DataArrays and Datasets. In some placed I had to add some "#type: ignore"s... I should probably open an issue for that.

• [x] Tests added

This PR improves typing of DataArray class methods. Main change is that T_DataArray is used whenever possible.

I have left everything untouched that would cause problems with mypy (there are some mypy bugs) or require larger typing efforts in other areas.

Main problems are argmin and argmax.

• [x] Closes #6628
• [x] Tests added

forgot to add defaults to the dtype argument, so the typing defaulted back to the Union definition...

This is initial try to get type hints for str and dt accessors.

I think there is no way of accessing the class at class scope (or is there?), so I had to use plain "DataArray" as the generic type of the accessors. I think that is acceptable for now.

The hack of DatetimeAccessor vs TimedeltaAccessor in the CombinedDatetimelikeAccessor.__new__ is something static typing can handle, so at type-checking time all properties of both accessors are available. If someone has a better idea?

Maybe a common interface class for accessors would be also beneficial?

This PR adds a mypy badge to the README.

Also, nicer alt texts for all other badges.

• [x] Closes #5382

Mypy is not able to compute the overloads of to_netcdf properly (too many Unions). I had to add some # type: ignore

I am not sure about the types of * concat_dim * combine_attrs: the Callable part is still with Anys

(partial) typing for functions full_like, zeros_like, ones_like.

I could not figure out how to properly use TypeVars so many things are "hardcoded" with overloads.

I have added a DTypeLikeSave to npcompat, not sure that this file is supposed to be edited.

Problem1: TypeVar["T", Dataset, DataArray, Variable] can only be one of these three, but never with Union[Dataset, DataArray] which is used in several other places in xarray. Problem2: The official mypy support says: use TypeVar["T", bound=Union[Dataset, DataArray, Variable] but the the isinstance(obj, Dataset) could not be correctly resolved (is that a mypy issue?).

So if anyone can get it to work with TypeVars, feel free to change it. :)

• [x] Closes #6600
• [x] Tests added

• [x] Closes #6597
• [x] Tests added

Attempt to fix the typing issues in xr.polyval.

Some problems are still occuring and require a type: ignore. They seem more like mypy issues to me.

• [x] Closes #6554
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Additionally I have added new literal types for error handling (Only applied to functions related to isel such that mypy stops complaining).

• [x] Closes #6526, #6411
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst