Code Sample

```python arr = xr.DataArray( np.ones(3), dims=('x',), coords={ 'x': ('x', np.array([1, 3, 6])), } ) list(arr.groupby('x'))

[(1, <xarray.DataArray ()> array(1.) Coordinates: x int64 1), (3, <xarray.DataArray ()> array(1.) Coordinates: x int64 3), (6, <xarray.DataArray ()> array(1.) Coordinates: x int64 6)] ```

Problem description

The dimension x disappear. I have done some tests and it seems that this problem raise only with strictly ascending coordinates. For example in this case it works correctly:

```python arr = xr.DataArray( np.ones(3), dims=('x',), coords={ 'x': ('x', np.array([2, 1, 0])), } ) list(arr.groupby('x'))

[(0, <xarray.DataArray (x: 1)> array([1.]) Coordinates: * x (x) int64 0), (1, <xarray.DataArray (x: 1)> array([1.]) Coordinates: * x (x) int64 1), (2, <xarray.DataArray (x: 1)> array([1.]) Coordinates: * x (x) int64 2)] ```

Expected Output

```python arr = xr.DataArray( np.ones(3), dims=('x',), coords={ 'x': ('x', np.array([1, 3, 6])), } ) list(arr.groupby('x'))

[(1, <xarray.DataArray (x: 1)> ar1ay([1.]) Coordinates: * x (x) int64 1), (3, <xarray.DataArray (x: 1)> array([1.]) Coordinates: * x (x) int64 3), (6, <xarray.DataArray (x: 1)> array([1.]) Coordinates: * x (x) int64 6)] ```

Output of xr.show_versions()

When open_dataset() fails because no working engines are found, it suggests installing the dependencies of the compatible internal backends, providing explicitly the list. - [x] closes #5302 - [x] Tests added - [x] Passes pre-commit run --all-files

Fix open_dataset regression, expands ~ in filepath_or_obj when necessary. I have checked the behaviour of the engines. It seems that pynio already expands ~.

• [x] Closes #5098
• [x] Passes pre-commit run --all-files

Currently, a broken engine breaks xarray.open_dataset. I have added a try except to avoid this problem.

Old behaviour:

```python

ds = xr.open_dataset('example.nc') Traceback (most recent call last):

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code exec(code_obj, self.user_global_ns, self.user_ns)

File "<ipython-input-3-0c694cae8262>", line 1, in <module> arr = xr.open_dataset("example.nc")

File "/Users/barghini/devel/xarray/xarray/backends/api.py", line 495, in open_dataset backend = plugins.get_backend(engine)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 115, in get_backend engines = list_engines()

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 97, in list_engines return build_engines(pkg_entrypoints)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 84, in build_engines external_backend_entrypoints = backends_dict_from_pkg(pkg_entrypoints)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 58, in backends_dict_from_pkg backend = pkg_ep.load()

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/pkg_resources/init.py", line 2450, in load return self.resolve()

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/pkg_resources/init.py", line 2456, in resolve module = import(self.module_name, fromlist=['name'], level=0)

File "/Users/barghini/devel/xarray-sentinel/xarray_sentinel/sentinel1.py", line 13 ERROR ^ SyntaxError: invalid syntax ```

New behaviour:

```python

ds = xr.open_dataset('example.nc') /Users/barghini/devel/xarray/xarray/backends/plugins.py:61: RuntimeWarning: Engine sentinel-1 loading failed: name 'ERROR' is not defined warnings.warn(f"Engine {name} loading failed:\n{ex}", RuntimeWarning) ```

• [x] Tests added
• [x] Passes pre-commit run --all-files

• add backend documentation

• rename store_spec in filename_or_objin backend entrypoint method guess_can_open

• [x] Related #4803

I just want to do a small recap of the current proposals for the class AbstractDataStore refactor discussed with @shoyer, @jhamman, and @alexamici.

Proposal 1: Store returns: - xr.Variables with the list of filters to apply to every variable - dataset attributes - encodings

Xarray applies to every variable only the filters selected by the backend before building the xr.Dataset.

Proposal 2: Store returns: - xr.Variables with all needed filters applied (configured by xarray), - dataset attributes - encodings

Xarray builds the xr.Dataset

Proposal 3: Store returns: - xr.Dataset

Before going on I'd like to collect pros and cons. For my understanding:

Proposal 1

pros: - the backend is free to decide which representation to provide. - more control on the backend (? not necessary true, the backend can decide to apply all the filters internally and provide xarray and empty list of filters to be applied) - enable / disable filters logic would be in xarray. - all the filters (applied by xarray) should have a similar interface. - maybe registered filters could be used by other backends

cons: - confusing backend-xarray interface. - more difficult to define interfaces. More conflicts (registered filters with the same name...) - need more structure to define this interface, more code to maintain.

Proposal 2

pros: - interface backend-xarray is clearer / backend and xarray have well different defined tasks. - interface would be minimal and easier to implement - no intermediate representations - less code to maintain

cons: - less control on filters. - more complex explicit definition of the interface (every filter must understand what decode_times means in their case) - more complexity inside the filters

The minimal interface would be something like that: py class AbstractBackEnd: def __init__(self, path, encode_times=True, ..., **kwargs): # signature of open_dataset raise NotImplementedError def get_variables(): """Return a dictionary of variable name and xr.Variable""" raise NotImplementedError def get_attrs(): """returns """ raise NotImplementedError def get_encoding(): """ """ raise NotImplementedError def close(self): pass

Proposal 3

pros w.r.t. porposal 2: - decode_coordinates is done by the backend as the other filters.

cons?

Any suggestions?

The backend read-support refactor is drawing to a close and we should start to add the documentation to explain how to implement new backends.

We should: - decide where to put the documentation - decide a title - define a brief list of the main points to discuss in the documentation.

For the first point, I suggest putting the documentation in "Internal". For the second one, I suggest: "How to add a new backend"

Concerning the third point, in the following there is a list of the topics, that I suggest:: - BackendEntrypoint Description (BackendEntrypoint is the main interface with xarray, it's a container of functions to be implemented and attributes: guess_can_open, open_dataset, open_dataset_parameters, [guess_can_write], [dataset_writer]) - How to add the backend as an external entrypoint. - Description of the function contained in BackendEntrypoint to be implemented. In particular, for open_dataset we have two option to describe:
- No Lazy it returns a dataset containing numpy arrays. - Lazy it returns a dataset containing BackendArrays: - BackendArrays description: - thread-safe __getitem__ - Pickable (use CachingFileManager) - indexing.IndexingSupport

Ensure that backend list are always sorted in the same way. In particular: - the standards backend are always the first in the following order: "netcdf4", "h5netcdf", "scipy" - all the other backends a sorted in lexicographic order.

the changes involve two files (plugins.py and test_plugins.py) and they include: - add utility function for sorting backends sort_backends - Update tests - Small changes in variables/functions names.

• [x] Tests added
• [x] Passes pre-commit run --all-files

Port all the engines to the new API apiv2. Note: - test_autoclose_future_warning has been removed because in apiv2.py autoclose has been removed - in open_backend_dataset_psedonetcdf currently is still used **format_kwargs and the signature is defined explicitly

• [x] Related to https://github.com/pydata/xarray/issues/4309
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8

Remove close_on_error in store.py. This change involves only apiv2. Currently, api_v2.open_dataset can take in input a store instead of a file. In case of error, xarray closes the store. Xarray should manage the closure of a store that has been instantiated externally. This PR correct this behaviour in apiv2

• [x] Related https://github.com/pydata/xarray/pull/4673
• [x] Passes isort . && black . && mypy . && flake8

This PR aims to avoid conflicts during the transition period between the old backend implementation and the new plugins. During the transition period will coexist both external backend plugins and internal ones. Currently, if two plugins with the same name are detected, we just pick one randomly. It would be better to be sure to use the external one.

Main changes: - Remove from setup.cfg - Store in the internal backend and stored in the dictionary in plugins.py. The dictionary is updated with the external plugins detected by pkg_resources. - Move the class BackendEntrypoints in common.py to resolve a circular import.
- Add a test

• [x] Related to https://github.com/pydata/xarray/issues/4309
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

This PR fixes https://github.com/pydata/xarray/issues/4708. It's a very small change. Changes: - In dataset._check_chunks_compatibility now it doesn't raise a warning if the last chunk % preferred_chunk != 0. - Update tests - Style: rename a variable inside dataset._check_chunks_compatibility

• [x] Closes https://github.com/pydata/xarray/issues/4708
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

• 4646 add tests on chunking without using a with statement, causing unexpected warnings.

• Add filterwarnings in test_plugins.test_remove_duplicates tests and backend_tests.test_chunking_consintency
• [x] Tests fixex
• [x] Passes isort . && black . && mypy . && flake8

This PR remove autoclose option from open_dataset (both api.py and apiv2.py) and the corresponding test test_autoclose_future_warning from test.py autoclose=True option was deprecated in https://github.com/pydata/xarray/pull/2261 since xarray now uses a LRU cache to manage open file handles.

• [x] Related to https://github.com/pydata/xarray/issues/4309 and https://github.com/pydata/xarray/pull/2261,
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8

Related to https://github.com/pydata/xarray/issues/4496 Add encodings["preferred_chunks"] in zarr, used in open_dataset instead of encodings["chunks"].

• [x] Related to #https://github.com/pydata/xarray/issues/4496
• [x] Passes isort . && black . && mypy . && flake8

It's the last part of, and closes #4595. Here we unify the code for chunking in apiv2.open_dataset. Note the code unification is only a refactor, there aren't functional changes since the zarr chunking has been already aligned with the others.

• [x] Related to https://github.com/pydata/xarray/issues/4496
• [x] Passes isort . && black . && mypy . && flake8

Part of https://github.com/pydata/xarray/pull/4595 The changes involve only open_dataset(..., engine=zarr) (and marginally open_zarr), in particular, _get_chunks has been modified to fit #4496 (comment) option 1 chunking behaviour and align open_dataset chunking with dataset.chunk:

• with auto it uses dask auto-chunking (if a preferred_chunking is defined, dask will take it into account as done in dataset.chunk)
• with -1 it uses dask but no chunking.
• with {} it uses the backend encoded chunks (when available) for on-disk data (xr.open_dataset) and the current chunking for already opened datasets (ds.chunk)

Add some test - [x] Releted to pydata#4496 - [x] Tests added - [x] Passes isort . && black . && mypy . && flake8 - [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

The aim is to split the PR https://github.com/pydata/xarray/pull/4595 in small PRs. In this smaller PR there aren't changes in xarry interfaces, it's only a small code refactor: - Move get_chunks from zarr.py to dataset.py - Align apiv2 to apiv1: in apiv2 replace zarr.ZarrStore.maybe_chunk with dataset._maybe_chunk and zarr.ZarrStore.get_chunk with dataset._get_chunks. - remove zarr.ZarrStore.maybe_chunk and zarr.ZarrStore.get_chunks (no more used)

• [x] Related #4496
• [x] Passes isort . && black . && mypy . && flake8
• No user visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

Proposal for the new API of open_dataset(). It is implemented in apiv2.py and it doesn't modify the current behavior of api.open_dataset(). It is something in between the first and second alternative suggested at https://github.com/pydata/xarray/issues/4490#issue-715374721, see the related quoted text:

Describe alternatives you've considered

For the overall approach:

1. We could keep the current design, with separate keyword arguments for decoding options, and just be very careful about passing around these arguments. This seems pretty painful for the backend refactor, though.
2. We could keep the current design only for the user facing open_dataset() interface, and then internally convert into the DecodingOptions() struct for passing to backend constructors. This would provide much needed flexibility for backend authors, but most users wouldn't benefit from the new interface. Perhaps this would make sense as an intermediate step?

Instead of a class for the decoders, I have added a function: resolve_decoders_kwargs.
resolve_decoders_kwargs performs two tasks: - If decode_cf is False, it sets to False all the decoders supported by the backend (using inspect). - It filters out the None decoder keywords.

So xarray manages the keyword decode_cf and passes on only the non-default decoders to the backend. If the user sets to a non-None value a decoder not supported by the backend, the backend will raise an error.

With this implementation drop_variable should be always supported by the backend. But I think this could be implemented easely by all the backends. I wouldn't group it with the decoders: to me, it seems to be more a filter rather than a decoder.

The behavior decode_cf is unchanged.

PRO: - the user doesn't need to import and instantiate a class. - users get the argument completion on open_dataset. - the backend defines directly in open_backend_dataset_${engine} API the accepted decoders. - xarray manages decode_cf, not the backends.

Missing points: - deccode_cf should be renamed decode. Probably, the behavior of decode should be modified for two reason: - currently If decode_cf is False, it sets the decoders to False, but there is no check on the other values. The accepted values should be: None (it keeps decoders default values), True (it sets all the decoders to True), False (it sets all the decoders to False). - currently we can set both a decoder and decode_cf without any warning. , but the - Deprecate backend_kwargs (or kwargs). - Separate mask_and_scale?

I think that we need a different PR for the three of them.

• [x] related to https://github.com/pydata/xarray/issues/4490#
• [x] Passes isort . && black . && mypy . && flake8

Currently, the interface between the backend and xarray is the class/container BackendEntrypoint, that must be instantiated by the backend. With this pull request, BackendEntrypoint is replaced by AbstractBackendEntrypoint. The backend will inherit from this class.

Reason for these changes: - This type of interface is more standard.

• [x] Tests updated
• [x] Passes pre-commit run --all-files

Currently list_engines returns the list of all installed backend plus the list of the internal ones. For the internal ones, there is no check on the installed dependencies. Now the registration of the internal backends is done by the backends only if the needed dependencies are installed.

• [x] Passes pre-commit run --all-files

Code Sample

```python import numpy as np import xarray as xr

arr = xr.DataArray( np.ones(5), dims=('x',), coords={ 'x': ('x', np.array([1, 1, 1, 2, 2])), 'x2': ('x', np.array([1, 2, 3, 4, 5])), } ) arr <xarray.DataArray (x: 5)> array([1., 1., 1., 1., 1.]) Coordinates: * x (x) int64 1 1 1 2 2 x2 (x) int64 1 2 3 4 5

out = arr.groupby('x').mean('x') out <xarray.DataArray (x: 2)> array([1., 1.]) Coordinates: * x (x) int64 1 2 x2 (x) int64 1 2 3 4 5 ```

Problem description

Inconsistency between: - the shape dimension x: (2,) - the shape of the coordinates x2 depending on the dimension x: (5,)

Expected Output

The coordinate x2 should be dropped. python <xarray.DataArray (x: 2)> array([1., 1.]) Coordinates: * x (x) int64 1 2

Output of xr.show_versions()

```python INSTALLED VERSIONS

commit: None python: 3.6.0.final.0 python-bits: 64 OS: Linux OS-release: 4.13.0-41-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

xarray: 0.10.4 pandas: 0.22.0 numpy: 1.14.3 scipy: 1.1.0 netCDF4: 1.3.1 h5netcdf: None h5py: 2.7.1 Nio: None zarr: None bottleneck: None cyordereddict: None dask: 0.17.4 distributed: 1.21.8 matplotlib: 2.2.2 cartopy: 0.16.0 seaborn: None setuptools: 38.4.1 pip: 10.0.1 conda: None pytest: 3.5.1 IPython: 6.2.1 sphinx: 1.7.4 ```

@jhamman, @shoyer @alexamici we discussed last time about the possibility to import directly in the xarray namespace the open function of the backends, open_dataset_${engine}. I just want to recap some pro and cons of this proposal:

Pros: - Expert users can use directly the open function of the backend (without using engine=) - They can use Tab key to autocomplete the backend kwargs. - They can easily access to the backend open function signature. (that's really useful!)

Cons: - The users they might expect in the namespace also the other corresponding functions: open_mfdataset_${engine}, open_datarray_${engine} etc ... and we are not going to do it because it is too confusing

• It's an update of @jhamman pull request https://github.com/pydata/xarray/pull/3166
• It uses entrypoints module to detect the installed engines. The detection is done at open_dataset function call and it is cached. It raises a warning in case of conflicts.
• Add a class for the backend interface BackendEtrypoint instead of a function.

Modified files: - add plugins.py containing detect_engines function and BackendEtrypoint. - dependencies file to add entrypoints. - backend.init to add detect_engines - apiv2.py and api.py do use detect_engines - zarr.py, h5netcdf_.py, cfgrib.py to instatiate the BackendEtrypoint.

• [x] Related to #3166
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

Is your feature request related to a problem? Please describe. In #4309 we proposed to separate xarray - backend tasks, more or less in this way: - Backend returns a dataset - xarray manage chunks and cache.

With the changes in open_dataset to support also zarr (#4187 ), we introduced a slightly different behavior for zarr chunking with respect the other backends.

Behavior of all the backends except zar - if chunk == {} or 'auto': it uses dask and only one chunk per variable - if the user defines chunks for not all the dimensions, along these dimensions it uses only one chunk: ```python

ds = xr.open_dataset('test.nc', chunks={'x': 4}) print(ds['foo'].chunks) ((4, 4, 4, 4, 4), (4,)) *Zarr chunking behavior* is very similar, but it has a different default when the user doesn't choose the size of the chunk along some dimensions, i.e. - if chunk == {} or 'auto': it uses in both cases the on-disk chunks - if the user defines the chunks for not all the dimensions, along these dimensions it uses no disk chunck:python ds = xr.open_dataset('test.zarr', engine='zarr', chunks={'x': 4}) print(ds['foo'].encoding['chunks']) (5, 2) print(ds['foo'].chunks) ((4, 4, 4, 4, 4), (2, 2)) ```

Describe the solution you'd like

We could extend easily zarr behavior to all the backends (which, for now, don't use the field variable.encodings['chunks']): if no chunks are defined in encoding, we use as default the dimension size, otherwise, we use the encoded chunks. So for now we are not going to change any external behavior, but if needed the other backends can use this interface. I have some additional notes:

• The key value auto is redundant because it has the same behavior as {}, we could remove one of them.
• I would separate the concepts "on disk chunk" and "preferred chunking". We can use a different key in encodings or ask the backend to expose a function to compute the preferred chunking.

One last question: - In the new interface of open_dataset there is a new key, imported from open_zarr: overwrite_encoded_chunks. Is it really needed? Why do we support to overwrite of the encoded chunks at readi time? This operation can be easily done after or at write time.

The aim is to split the PR #4595 in small PRs. The scope of this smaller PR is to modify the default of chunks in dataset.chunk to align the behaviour to xr.open_dataset. The main changes are: - Modify the default of chunk in dataset.chunk, datarray.chunk and variable.chunk from None to {}. - If the user pass chunk=None inside is set to {} - Add a future warning to advice that the usage of None will raise an error in the future.

Note that the changes currently don't modify the behaviour of dataset.chunk

• [x] Related #4496
• [x] Passes isort . && black . && mypy . && flake8

Related to PR https://github.com/pydata/xarray/pull/4595. In this smaller PR, there aren't changes functional changes, it's only a small code refactor needed to simplify pydata#4595. Changes in apiv2.dataset_from_backend_dataset: - rename ds in backend_ds and ds2 in ds. - simplify if in chunking and split code adding function _chunks_ds - add _get_mtime specific function

Make resolve_decoders_kwargs and dataset_from_backend_dataset private - [x] related to https://github.com/pydata/xarray/pull/4595 - [x] Passes isort . && black . && mypy . && flake8

This work aims to harmonize the way zarr deals with chunking to have similar behavior for all other backends and unify the code. Most of the changes involve the new API, apiv2.py, except for some changes in the code that has been added with the merge of https://github.com/pydata/xarray/pull/4187.

main changes: - refactor apiv2.dataset_from_backend_dataset function. - move _get_chunks from zarr to dataset. - modify _get_chunks to fit https://github.com/pydata/xarray/issues/4496#issuecomment-720785384 option 1 chunking behaviuor - Add warning when it is used in ds.chunk(..., chunk=None) - Add some test

nedded separate pull request for the following missing points: - standardize the key in encodings to define the on-disk chunks: chunksizes - add a specific key in encodings for preferred chunking (currently it is used chunks)

• [x] Related https://github.com/pydata/xarray/issues/4496
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This work aims to harmonize the way zarr deals with chunking to have similar behavior for all other backends and unify the code. Most of the changes involve the new API, apiv2.py, except for some changes in the code that has been added with the merge of https://github.com/pydata/xarray/pull/4187.

main changes: - refactor apiv2.dataset_from_backend_dataset function. - move get_chunks from zarr to dataset.

current status: - in apiv2.open_dataset chunks='auto' and chunks={} now has the same beahviuor - in apiv2.open_dataset for all the backends now the default chunking is provided by the backend, if it is not available it uses one big chunk.

Missing points: - standardize the key in encodings to define the on-disk chunks: chunksizes - add a specific key in encodings for preferred chunking (currently it is used chunks)

There is one open point to be discussed yet: dataset.chunks and open_dataset(..., chunks=...) have different behaviors. dataset.chunks(chunks={}) opens the dataset with only one chunk per variable, while in open_dataset(..., chunks={}) it uses encodings['chunks'], when available.

Note that also chunks=None has a different behaviour: open_dataset(..., chunks=None) (or open_dataset(...), it's the deafult) returns variables without chunks, while dataset.chunk(chunks=None) (or dataset.chunk(), it's the default) has the same behavior of dataset.chunk(chunks=None). Probably it's not worth changing it.

• [x] related to https://github.com/pydata/xarray/issues/4496
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I propose this small change with a view to unifying in open_dataset the logic of zarr chunking with that of the other backends. Currently, the function maybe_chunk is duplicated: it is defined inside the function dataset.chunks and as method of zarr.ZarrStore. This last function has been added with the recent merge of #4187 . I merged the two functions in a private function _maybe_chunk inside the module dataset.

• [x] Addresses #4309
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8

The first draft of the new backend API: - Move decoding inside the bakends. - Backends return Dataset with BackendArray. - Xarray manages chunking and caching. - Some code is duplicated, it will be simplified later. - Zarr chunking is still inside the backend for now.

cc @jhamman @shoyer

• [x] Addresses #4309
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Releated to ticket #1621. Add decode_timedelta kwargs in open_dataset, xr.open_datarray, xr.open_zarr, xr.decode_cf. If not passed explicitly the behaviour is not changed. - [x] Tests added for xr.decode_cf. - [x] Passes isort -rc . && black . && mypy . && flake8 - [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Code Sample, a copy-pastable example if possible

```python import numpy as np import pandas as pd import xarray as xr

time = np.arange('2005-02-01', '2007-03-01', dtype='datetime64') arr = xr.DataArray( np.arange(time.size), coords=[time,], dims=('time',), name='data' ) arr.resample(time='M').interpolate('linear')

ValueError Traceback (most recent call last) <ipython-input-7-6a92b6afe08e> in <module>() 7 np.arange(time.size), coords=[time,], dims=('time',), name='data' 8 ) ----> 9 arr.resample(time='M').interpolate('linear')

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/xarray/core/resample.py in interpolate(self, kind) 108 109 """ --> 110 return self._interpolate(kind=kind) 111 112 def _interpolate(self, kind='linear'):

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/xarray/core/resample.py in _interpolate(self, kind) 218 elif self._dim not in v.dims: 219 coords[k] = v --> 220 return DataArray(f(new_x), coords, dims, name=dummy.name, 221 attrs=dummy.attrs) 222

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/polyint.py in call(self, x) 77 """ 78 x, x_shape = self._prepare_x(x) ---> 79 y = self._evaluate(x) 80 return self._finish_y(y, x_shape) 81

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _evaluate(self, x_new) 632 y_new = self._call(self, x_new) 633 if not self._extrapolate: --> 634 below_bounds, above_bounds = self._check_bounds(x_new) 635 if len(y_new) > 0: 636 # Note fill_value must be broadcast up to the proper size

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _check_bounds(self, x_new) 664 "range.") 665 if self.bounds_error and above_bounds.any(): --> 666 raise ValueError("A value in x_new is above the interpolation " 667 "range.") 668

ValueError: A value in x_new is above the interpolation range. ```

Problem description

The internal format of arr.time is datetime64[D]

```python arr.time

<xarray.DataArray 'time' (time: 758)> array(['2005-02-01', '2005-02-02', '2005-02-03', ..., '2007-02-26', '2007-02-27', '2007-02-28'], dtype='datetime64[D]') Coordinates: * time (time) datetime64[D] 2005-02-01 2005-02-02 2005-02-03 ... ``` Internally there is a cast to float, for both the old time indices x and the new time indices new_x, but the new time indices are in datetime64[ns], so they don't match.

DataArrayResample._interpolate

```python x = self._obj[self._dim].astype('float') y = self._obj.data

   axis = self._obj.get_axis_num(self._dim)

   f = interp1d(x, y, kind=kind, axis=axis, bounds_error=True,
                assume_sorted=True)
   new_x = self._full_index.values.astype('float')

``` With a cast to datetime64[ns] it works:

```python import numpy as np import pandas as pd import xarray as xr

time = np.arange('2005-02-01', '2007-03-01', dtype='datetime64').astype('datetime64[ns]') arr = xr.DataArray( np.arange(time.size), coords=[time,], dims=('time',), name='data' ) arr.resample(time='M').interpolate('linear') <xarray.DataArray 'data' (time: 25)> array([ 27., 58., 88., 119., 149., 180., 211., 241., 272., 302., 333., 364., 392., 423., 453., 484., 514., 545., 576., 606., 637., 667., 698., 729., 757.]) Coordinates: * time (time) datetime64[ns] 2005-02-28 2005-03-31 2005-04-30 ... ```

Expected Output

python <xarray.DataArray 'data' (time: 25)> array([ 27., 58., 88., 119., 149., 180., 211., 241., 272., 302., 333., 364., 392., 423., 453., 484., 514., 545., 576., 606., 637., 667., 698., 729., 757.]) Coordinates: * time (time) datetime64[ns] 2005-02-28 2005-03-31 2005-04-30 ...

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python import numpy as np import pandas as pd import xarray as xr

time = np.arange('2007-02-01', '2007-03-02', dtype='datetime64').astype('datetime64[ns]') arr = xr.DataArray( np.arange(time.size), coords=[time,], dims=('time',), name='data' ) arr.resample(time='M').interpolate('linear')

ValueError Traceback (most recent call last) <ipython-input-20-ff65c4d138e7> in <module>() 7 np.arange(time.size), coords=[time,], dims=('time',), name='data' 8 ) ----> 9 arr.resample(time='M').interpolate('linear')

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/xarray/core/resample.py in interpolate(self, kind) 108 109 """ --> 110 return self._interpolate(kind=kind) 111 112 def _interpolate(self, kind='linear'):

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/xarray/core/resample.py in _interpolate(self, kind) 218 elif self._dim not in v.dims: 219 coords[k] = v --> 220 return DataArray(f(new_x), coords, dims, name=dummy.name, 221 attrs=dummy.attrs) 222

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/polyint.py in call(self, x) 77 """ 78 x, x_shape = self._prepare_x(x) ---> 79 y = self._evaluate(x) 80 return self._finish_y(y, x_shape) 81

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _evaluate(self, x_new) 632 y_new = self._call(self, x_new) 633 if not self._extrapolate: --> 634 below_bounds, above_bounds = self._check_bounds(x_new) 635 if len(y_new) > 0: 636 # Note fill_value must be broadcast up to the proper size

~/devel/c3s-cns/venv_op/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _check_bounds(self, x_new) 664 "range.") 665 if self.bounds_error and above_bounds.any(): --> 666 raise ValueError("A value in x_new is above the interpolation " 667 "range.") 668

ValueError: A value in x_new is above the interpolation range.

```

Problem description

It raise an error if I try to interpolate. If time range is exactly a month, then it works: ```python time = np.arange('2007-02-01', '2007-03-01', dtype='datetime64').astype('datetime64[ns]') arr = xr.DataArray( np.arange(time.size), coords=[time,], dims=('time',), name='data' ) arr.resample(time='M').interpolate('linear')

<xarray.DataArray 'data' (time: 1)> array([27.]) Coordinates: * time (time) datetime64[ns] 2007-02-28 ```

The problem for the interpolation seems to be that the resampler contains indices out bound ('2007-03-31'). It is ok for the aggregations, but it doesn't work with the interpolation.

```python resampler = arr.resample(time='M')

resampler._full_index DatetimeIndex(['2007-02-28', '2007-03-31'], dtype='datetime64[ns]', name='time', freq='M') ```

Expected Output

python <xarray.DataArray 'data' (time: 1)> array([27.]) Coordinates: * time (time) datetime64[ns] 2007-02-28

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python arr = xr.DataArray( np.arange(5), dims=('x',), coords={ 'x': ('x', np.array([1, 1, 1, 2, 2])), } )

gr = arr.groupby('x') gr.first()

arr

<xarray.DataArray (x: 5)> array([0, 1, 2, 3, 4]) Coordinates: * x (x) int64 1 2

```

Problem description

A side effect of the GroupBy.first method call is that it substitutes the original array coordinates with the grouped ones .

Expected Output

arr

<xarray.DataArray (x: 5)> array([0, 1, 2, 3, 4]) Coordinates: * x (x) int64 1 1 1 2 2

Output of xr.show_versions()