issue_comments

Thanks @dcherian :smile: You're very welcome!

Thanks @johnomotani . This is a significant contribution.

Thanks @keewis! I think we've addressed all the review comments now.

calling on a array with missing values raises ValueError: All-NaN slice encountered (try ds = xr.tutorial.open_dataset("rasm"); ds.argmin(dim="y"))

Some missing values should be OK. In your example though for example In [27]: ds.Tair.isel(time=0, x=0).values Out[27]: array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]) So there are no values in the array slice that argmin should be applied over, which is an error.

I guess we could add some special handling for this (not sure what though, because we can't set a variable with type int to nan or None), but I think that is a separate issue that would need a new PR.

well, we'd need to somehow be able to use sel to index every variable in a Dataset differently. While that would probably be possible it would make sel way too complicated, so I agree that an error would be good here.

I think no further changes to the pint tests should be required (except refactoring, but I can do that in a new PR after this one has been merged).

ds.argmin(dim=...) returns a single index (the same result as ds.argmin()) but ds.Tair.argmin(dim=...) returns something different from ds.Tair.argmin(). Is that intentional?

I think this is a bug. dim=... to Dataset.argmin should be an error because dim=... returns a dict of results, and it's not clear how to do that consistently for a Dataset that might have several members with different combinations of dimensions.

I noticed a few issues while debugging: - calling on a array with missing values raises ValueError: All-NaN slice encountered (try ds = xr.tutorial.open_dataset("rasm"); ds.argmin(dim="y")) - ds.argmin(dim=...) returns a single index (the same result as ds.argmin()) but ds.Tair.argmin(dim=...) returns something different from ds.Tair.argmin(). Is that intentional?

I'm going to wait a little while before merging in case anyone else has comment, but otherwise will merge in a day or two (definitely before the next release)

Merge conflicts fixed, this PR should be ready to review/merge.

you could mark the function("argmin") / function("argmax") with xfail (or skip?). We can decide on what to do with them when we close #3917

I guess this would happen if somebody calls np.argmin(xarray_object)? Does this happen from inside xarray somewhere, or in external code?

If it's the former, then we should just fix xarray not to do this. If it's only in external code, I would still consider breaking this behavior.

test_units.py has a couple of tests that apply numpy functions to xr.Variables, xr.DataArrays and xr.Datasets. Those tests break for argmin and argmax without the out argument. Should I just remove the argmin and argmax function calls? The tests would still call them as methods.

I notice that some of your new deprecation warnings get issued when running your tests. Please silence these methods, e.g., using pytest.mark.filterwarnings.

I eventually found that the cause of the errors I was getting was that the argmin and argmax methods did not have an out argument. In order for the methods to be wrapped by numpy (and I guess dask is the same), the call signature of np.argmin() has to be supported, which means axis and out arguments are needed.

I guess this would happen if somebody calls np.argmin(xarray_object)? Does this happen from inside xarray somewhere, or in external code?

If it's the former, then we should just fix xarray not to do this. If it's only in external code, I would still consider breaking this behavior.

The override mechanism that NumPy uses in argmin/argmax is really old/fragile, and it's not something we ever intended to support. There's no way we could write an argmin/argmax method with an incompatible signature that would work with these functions.

The better way to do overrides in NumPy is probably with __array_function__ now, but I'm not sure we want the trouble...

@pydata/xarray - I think this PR is ready to be merged, are there any changes I should make? Thanks!

I eventually found that the cause of the errors I was getting was that the argmin and argmax methods did not have an out argument. In order for the methods to be wrapped by numpy (and I guess dask is the same), the call signature of np.argmin() has to be supported, which means axis and out arguments are needed.

@shoyer I've kept the use of injected functions, because removing the injected argmin and argmax completely meant re-implementing handling of skipna and only within _unravel_argminmax, which seemed less nice to me. If/when method injection is refactored, it would be nice to include a mechanism to override the core operation with an explicitly implemented, extended version like argmin/argmax.

I think this PR is ready for review now :smile:

Also, feel free to rewrite to avoid using inject_all_ops_and_reduce_methods() for argmin/argmax at all. Method injection is pretty hacky, and generally not worthwhile, e.g., see this note about removing it.

These test failures seem to have uncovered a larger issue: overriding a method injected by ops.inject_all_ops_and_reduce_methods(DataArray, priority=60) is tricky. I think there may be a way around it by mangling the name of the injected operation if that method is already defined on the class; I tried this and it nearly works, but I think I need to implement an override for argmin/argmax on Variable, because DataArray.__array_wrap__ uses the Variable version of the method... Work in progress!

The rename to _argmin_base is also causing test_aggregation (which is a test of units with pint) to fail, because the numpy function and the Variable method do not have the same name. Any suggestions how to fix this? It seems tricky because I can't pass either argmin (no Variable method) or _argmin_base (no numpy method) to the test.

I've updated so the new functionality is provided by argmin() and argmax(), when they are passed a multiple dimensions. * I've added Dataset.argmin() and Dataset.argmax(), which only work for a single dimension, but give informative exceptions for unsupported cases. If anyone has some behaviour that they would like for Dataset.argmin() with multiple dimensions, I'm happy to implement it, but returning something like a dict of Datasets didn't seem nice, and as far as I understand a Dataset cannot contain something like a dict of DataArrays. * I renamed argmin and argmax in ops.py and duck_array_ops.py to _argmin_base and _argmax_base so that I could still use them inside the argmin and argmax methods. Any alternatives, or better suggestions for naming them? * Variable no longer has argmin or argmax methods. Is that a problem? I just updated test_variable.py and test_dask.py to use _argmin_base and _argmax_base.

@johnomotani FYI: For #3871 (merged) there is #3922 (yet unmerged) to fix dask-handling.

reduce over all given dimensions?

By this do you mean find the minimum as if the array were first (partially or totally) flattened along the given dims somehow? I'm not sure we provide that kind of behaviour anywhere in the current API.

@TomNicholas Probably I was bit confused by the current docstring. I think I understand now and there should be no problem at all.

how should we discern (at least for argmin/argmax), if the user wants to...

That's a good question @kmuehlbauer, and the distinction probably needs to be clearer in the docs in general.

reduce over all given dimensions?

By this do you mean find the minimum as if the array were first (partially or totally) flattened along the given dims somehow? I'm not sure we provide that kind of behaviour anywhere in the current API.

+1 for making argmin/argmax (and idxmin/idxmax) work, when given multiple dimensions.

According to the current docstring it should already work that way for argmin/'argmax`:

Reduce this DataArray’s data by applying argmin along some dimension(s).

Returns: New DataArray/Dataset object with argmin applied to it's data and the indicated dimension(s) removed

But this behaviour is broken currently (works only for one given dim).

My main concern for changing the API as suggested above is, how should we discern (at least for argmin/argmax), if the user want's to:

1. reduce over all given dimensions? (current behaviour according to docstring)
2. reduce along all given dimensions? (suggested behaviour by @johnomotani [please correct if I'm wrong])

Another option would be to overload argmin

+1 for overloading argmin (and later idxmin). IMO we should never have one function for a 1D operation and one for an N-D operation if we can avoid it, everything should be N-dimensional.

I also really like how neat this resultant property is python da.isel(da.argmin(list_of_dim)) == da.min(list_of_dim) we could even use a hypothesis test to check it...

I think returning a dict of indices would be much more useful, but it does change existing behaviour (more useful because you can then do da.isel(da.argmin())).

Although it's breaking and would require a deprecation cycle, I think this is what we should aim for.

there's a not-very-helpful exception

Yes let's take the time to make that clearer for users - this will be a commonly-used function.

@shoyer I think your last option sounds good. Questions: * What should da.argmin() with no arguments do? * Currently returns the flattened index of the global minimum. * I think returning a dict of indices would be much more useful, but it does change existing behaviour (more useful because you can then do da.isel(da.argmin())). * Could anyway do da.argmin(da.dims) to get the dict result, but that's a bit ugly * Could have something like da.argmin(...) - maybe as a temporary workaround while we deprecate the current behaviour of da.argmin()? * If we overload argmin, what's the cleanest way to get the existing argmin() method within the new one? Would something like from .duck_array_ops import argmin as argmin_1d work?

Maybe worth noting, at the moment if you try to call argmin(dim=("x", "y")) with multiple dimensions, there's a not-very-helpful exception ```

array.argmin(dim=("x", "y")) Traceback (most recent call last): File "/home/john/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 61, in _wrapfunc return bound(args, *kwds) TypeError: 'tuple' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/<...>/xarray/xarray/core/common.py", line 46, in wrapped_func return self.reduce(func, dim, axis, skipna=skipna, kwargs) File "/<...>/xarray/xarray/core/dataarray.py", line 2288, in reduce var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs) File "/<...>/xarray/xarray/core/variable.py", line 1579, in reduce data = func(input_data, axis=axis, kwargs) File "/<...>/xarray/xarray/core/duck_array_ops.py", line 304, in f return func(values, axis=axis, kwargs) File "/<...>/xarray/xarray/core/duck_array_ops.py", line 47, in f return wrapped(args, kwargs) File "<array_function internals>", line 6, in argmin File "/<...>/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 1267, in argmin return _wrapfunc(a, 'argmin', axis=axis, out=out) File "/<...>/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 70, in _wrapfunc return _wrapit(obj, method, args, kwds) File "/<...>/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 47, in _wrapit result = getattr(asarray(obj), method)(*args, kwds) TypeError: 'tuple' object cannot be interpreted as an integer ```

This looks really comprehensive, thank you!

Before doing a really careful review here, I'd like to try to work out the full API design we want. I'll write out some of my thoughts here, but your thoughts would also be very welcome!

Here's my summary of the current situation: 1. We have two pairs of methods, argmin/argmax and the new idxmin/idxmax for calculating integer positions or coordinate labels of the minimum/maximum along one dimension. 2. We would like to have a way to do a similar calculation in multiple dimensions, which should return a dictionary of DataArray objects suitable for feeding into isel/sel.

This PR implements the multidimensional equivalent of argmin/argmax. It's fine to save the multidimensional equivalent of idxmin/idxmax for later, but we'll want to copy whatever design we pick here.

My first concern is about the name: it isn't obvious to me whether indices_min refers to integer positions or coordinate labels. I think a name based on argmin/argmax could help, e.g., perhaps argmin_nd or argmin_dict.

Another option would be to overload argmin/argmax to also support return a dictionary, depending on the type of the arguments. If you pass a list of dimensions over which to calculate the min or max locations, you would get a dictionary back: - array.argmin('x') would return a single DataArray - array.argmin(['x', 'y']) would return a dict of DataArray objects with keys {'x', 'y'} - array.argmin(['x']) would also return a dict {'x': ...}

I think I like this last option best but I would be curious what others think! @pydata/xarray any thoughts on this?