issue_comments

Thanks for implementing this! This is a feature, that we will be using for sure! Mostly with indices of type 1 which, in many cases, can easily be extrapolated. Having this as a default or a switch to enable extrapolation where possible would help a lot!

Thanks @mathause , this works for me. Also with a change of dtype in func, as long as "output_dtypes" is set correctly.

I am in the exact same situation. @DerWeh with the current master you can do da.to_netcdf("complex.nc", engine="h5netcdf", invalid_netcdf=True) which works for me until there is engine="hdf5" or may be a method da.to_hdf()?

I think the problem is somewhere here:

https://github.com/pydata/xarray/blob/aaeea6250b89e3605ee1d1a160ad50d6ed657c7e/xarray/core/utils.py#L85-L103

I don't think pandas.Index can hold lazy arrays. Could there be a way around exploiting dask.dataframe indexing methods?

Thanks guys!

May be this is a stupid question: What's the best way to resolve this conflict, and get the checks to run? Should I do a merge master? Or rebase (seems somewhat of a pain to get the remote up to date)? Thanks for the advice!

Guys, if you could have a look at the tests I modified (a4c3622) to check how many times things get computed. I tried to integrate it with the existing tests. The number of count check could possibly be applied to some other tests as well. May be there is a smarter way of doing this?

Thanks @max-sixty! Sure, I'll do this tomorrow then.

Thanks @shoyer. Cool, then this was easier than I expected. I added the patch and nanargmax/min to the nputils in #3244. What do you think?

Thanks @dcherian and @shoyer for the review and advice! It seems the docs build and the checks are fine now. One more thing, as this is my first PR to xarray: Should I merge/rebase master into the allow_invalid_netcdf branch, or will you guys take it from this point?

Those little changes do solve the MCVE, but break at least one test. I don't have enough of an understanding of the (nan)ops logic in xarray to get around the issue. But may be this helps:

The change

``` diff --git a/xarray/core/nanops.py b/xarray/core/nanops.py index 9ba4eae2..784a1d01 100644 --- a/xarray/core/nanops.py +++ b/xarray/core/nanops.py @@ -91,17 +91,9 @@ def nanargmin(a, axis=None): fill_value = dtypes.get_pos_infinity(a.dtype) if a.dtype.kind == "O": return _nan_argminmax_object("argmin", fill_value, a, axis=axis) - a, mask = _replace_nan(a, fill_value) - if isinstance(a, dask_array_type): - res = dask_array.argmin(a, axis=axis) - else: - res = np.argmin(a, axis=axis)

• if mask is not None:
• mask = mask.all(axis=axis)
• if mask.any():
• raise ValueError("All-NaN slice encountered")
• return res
• module = dask_array if isinstance(a, dask_array_type) else nputils
• return module.nanargmin(a, axis=axis)

def nanargmax(a, axis=None): @@ -109,17 +101,8 @@ def nanargmax(a, axis=None): if a.dtype.kind == "O": return _nan_argminmax_object("argmax", fill_value, a, axis=axis)

• a, mask = _replace_nan(a, fill_value)
• if isinstance(a, dask_array_type):
• res = dask_array.argmax(a, axis=axis)
• else:

• res = np.argmax(a, axis=axis)

• if mask is not None:
• mask = mask.all(axis=axis)
• if mask.any():
• raise ValueError("All-NaN slice encountered")
• return res
• module = dask_array if isinstance(a, dask_array_type) else nputils
• return module.nanargmax(a, axis=axis)

def nansum(a, axis=None, dtype=None, out=None, min_count=None):

```

The failing test

``` python
...
__ TestVariable.testreduce __
...

def f(values, axis=None, skipna=None, **kwargs):
    if kwargs.pop("out", None) is not None:
        raise TypeError("`out` is not valid for {}".format(name))

    values = asarray(values)

    if coerce_strings and values.dtype.kind in "SU":
        values = values.astype(object)

    func = None
    if skipna or (skipna is None and values.dtype.kind in "cfO"):
        nanname = "nan" + name
        func = getattr(nanops, nanname)
    else:
        func = _dask_or_eager_func(name)

    try:
        return func(values, axis=axis, **kwargs)
    except AttributeError:
        if isinstance(values, dask_array_type):
            try:  # dask/dask#3133 dask sometimes needs dtype argument
                # if func does not accept dtype, then raises TypeError
                return func(values, axis=axis, dtype=values.dtype, **kwargs)
            except (AttributeError, TypeError):
                msg = "%s is not yet implemented on dask arrays" % name
        else:
            msg = (
                "%s is not available with skipna=False with the "
                "installed version of numpy; upgrade to numpy 1.12 "
                "or newer to use skipna=True or skipna=None" % name
            )

      raise NotImplementedError(msg)

E NotImplementedError: argmax is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None

... ``` Note: I habe numpy 1.17 instaleed so the error msg here seems missleading.

Hey guys, I'd like to move this forward, but the doc build failed with the message below.

• Do I need to add something specific to make the docs build?
• Do I need to do anything else to get this PR merged?

Thanks!

``` python

---

Exception in /home/vsts/work/1/s/doc/io.rst at block ending on line 388 Specify :okexcept: as an option in the ipython:: block to suppress this message

ModuleNotFoundError Traceback (most recent call last) <ipython-input-17-9100cd49113c> in <module> ----> 1 da.to_netcdf("complex.nc", engine="h5netcdf", invalid_netcdf=True)

~/work/1/s/xarray/core/dataarray.py in to_netcdf(self, args, kwargs) 2214 dataset = self.to_dataset() 2215 -> 2216 return dataset.to_netcdf(args, **kwargs) 2217 2218 def to_dict(self, data: bool = True) -> dict:

~/work/1/s/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf) 1519 unlimited_dims=unlimited_dims, 1520 compute=compute, -> 1521 invalid_netcdf=invalid_netcdf, 1522 ) 1523

~/work/1/s/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf) 1052 "unrecognized option 'invalid_netcdf' for engine %s" % engine 1053 ) -> 1054 store = store_open(target, mode, format, group, **kwargs) 1055 1056 if unlimited_dims is None:

~/work/1/s/xarray/backends/h5netcdf_.py in init(self, filename, mode, format, group, lock, autoclose, invalid_netcdf) 81 invalid_netcdf=None, 82 ): ---> 83 import h5netcdf 84 85 if format not in [None, "NETCDF4"]:

ModuleNotFoundError: No module named 'h5netcdf' <<<-------------------------------------------------------------------------

/home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.integrate:12: WARNING: Unexpected indentation. /home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.interp:20: WARNING: Inline strong start-string without end-string. /home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.interpolate_na:8: WARNING: Definition list ends without a blank line; unexpected unindent.

Sphinx parallel build error: RuntimeError: Non Expected exception in /home/vsts/work/1/s/doc/io.rst line 388

[error]The operation was canceled.

```

Even though the example from above does work, sadly, the following does not: python import xarray as xr import dask.array as da import numpy as np da = xr.DataArray(np.random.rand(3*4*5).reshape((3,4,5))).chunk(dict(dim_0=1)) idcs = da.argmax('dim_2') da[dict(dim_2=idcs)] results in ``` python

TypeError Traceback (most recent call last) <ipython-input-4-3542cdd6d61c> in <module> ----> 1 da[dict(dim_2=idcs)]

~/src/xarray/xarray/core/dataarray.py in getitem(self, key) 604 else: 605 # xarray-style array indexing --> 606 return self.isel(indexers=self._item_key_to_dict(key)) 607 608 def setitem(self, key: Any, value: Any) -> None:

~/src/xarray/xarray/core/dataarray.py in isel(self, indexers, drop, **indexers_kwargs) 986 """ 987 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "isel") --> 988 ds = self._to_temp_dataset().isel(drop=drop, indexers=indexers) 989 return self._from_temp_dataset(ds) 990

~/src/xarray/xarray/core/dataset.py in isel(self, indexers, drop, **indexers_kwargs) 1901 indexes[name] = new_index 1902 else: -> 1903 new_var = var.isel(indexers=var_indexers) 1904 1905 variables[name] = new_var

~/src/xarray/xarray/core/variable.py in isel(self, indexers, drop, **indexers_kwargs) 984 if dim in indexers: 985 key[i] = indexers[dim] --> 986 return self[tuple(key)] 987 988 def squeeze(self, dim=None):

~/src/xarray/xarray/core/variable.py in getitem(self, key) 675 array x.values directly. 676 """ --> 677 dims, indexer, new_order = self._broadcast_indexes(key) 678 data = as_indexable(self._data)[indexer] 679 if new_order:

~/src/xarray/xarray/core/variable.py in _broadcast_indexes(self, key) 532 if isinstance(k, Variable): 533 if len(k.dims) > 1: --> 534 return self._broadcast_indexes_vectorized(key) 535 dims.append(k.dims[0]) 536 elif not isinstance(k, integer_types):

~/src/xarray/xarray/core/variable.py in _broadcast_indexes_vectorized(self, key) 660 new_order = None 661 --> 662 return out_dims, VectorizedIndexer(tuple(out_key)), new_order 663 664 def getitem(self, key):

~/src/xarray/xarray/core/indexing.py in init(self, key) 460 raise TypeError( 461 "unexpected indexer type for {}: {!r}".format( --> 462 type(self).name, k 463 ) 464 )

TypeError: unexpected indexer type for VectorizedIndexer: dask.array<arg_agg-aggregate, shape=(3, 4), dtype=int64, chunksize=(1, 4)> ```

Hmm, seems like I broke the docs somehow... I someone could advice me how to fix? Thanks!

This will also need a note in whats-new.rst and a note in io.rst, perhaps under "Writing encoded data"

Sure, I'll add that when we are happy with the code and tests!

As of version 0.12 indexing with dask arrays works out of the box... I think this can be closed now.

It seem's working fine with the following change but it has a lot of dublicated code... ``` diff --git a/xarray/core/indexing.py b/xarray/core/indexing.py index d51da471..9fe93581 100644 --- a/xarray/core/indexing.py +++ b/xarray/core/indexing.py @@ -7,6 +7,7 @@ from datetime import timedelta

import numpy as np import pandas as pd +import dask.array as da

from . import duck_array_ops, nputils, utils from .pycompat import ( @@ -420,6 +421,19 @@ class VectorizedIndexer(ExplicitIndexer): 'have different numbers of dimensions: {}' .format(ndims)) k = np.asarray(k, dtype=np.int64) + elif isinstance(k, dask_array_type): + if not np.issubdtype(k.dtype, np.integer): + raise TypeError('invalid indexer array, does not have ' + 'integer dtype: {!r}'.format(k)) + if ndim is None: + ndim = k.ndim + elif ndim != k.ndim: + ndims = [k.ndim for k in key + if isinstance(k, (np.ndarray) + dask_array_type)] + raise ValueError('invalid indexer key: ndarray arguments ' + 'have different numbers of dimensions: {}' + .format(ndims)) + k = da.array(k, dtype=np.int64) else: raise TypeError('unexpected indexer type for {}: {!r}' .format(type(self).name, k)) ```

@jhamman - thanks, this should be usefull...

Hi, this is still the case for version 0.10.0. arr = xr.DataArray(np.random.rand(10, 3), ...: [('time', pd.date_range('2000-01-01', periods=10)), ...: ('space', ['IA', 'IL', 'IN'])]) ...: arr.loc[arr.time[2]:arr.time[5]] fails, but doing the same thing on a pandas dataframe works just fine: dfr = arr.to_dataframe(name='dfr') dfr.loc[arr.time[2]:arr.time[5]] I'd really appreciate see this working on a DataArray.

Ok, thanks for opening the dask issue.