issue_comments

@toddrjen thank you again! This ended up being quite an adventure, really appreciate you pushing all the way.

Thanks @toddrjen that's a great contribution!

I have gone over it one more time and made a few documentation fixes. Please take one more look before merging.

Amazing — let's merge on green!

Thank you again @toddrjen, both for the content and the iteration.

@shoyer any final thoughts? Otherwise I'll merge tomorrow?

Here is a new commit with the discussed changes.

I think I have implemented all the requested changes and all tests are passing. Please take a look.

I figured out what is going wrong. I will make a commit with a fix and include it in this pull request later today.

I am not sure why the tests are suddenly failing. The tests were all working, then I rebased on the latest master and they are failing and I can't figure out why.

Please see the newest version with the promote argument changed to fill_value.

That could work.

The corner case we would need to decide on is again promotion.

What happens if the fill value is a "higher" type in the numeric tower than the original type? What if it is lower?

1. We could try to always convert to the fill dtype (or more often the dtype equivalent to the Python native type), and raise and exception of it doesn't work.
2. We could promote the fill value or original data, whichever is "lower".

What if someone tries to use a string type for numeric data or vice versus? If we do option 1 that is easy. Otherwise we probably need to use numpy casting rules?

What about an object dtype fill value?

What about a date/time regard dtype?

On Mon, Mar 23, 2020, 23:49 Stephan Hoyer notifications@github.com wrote:

@shoyer commented on this pull request.

In xarray/core/dataset.py https://github.com/pydata/xarray/pull/3871#discussion_r396887940:

@@ -5914,5 +5921,169 @@ def pad(

     return self._replace_vars_and_dims(variables)

• def idxmin(
• self,
• dim: Hashable = None,
• axis: int = None,
• skipna: bool = None,
• promote: bool = None,

Just to throw out another API option: what about having a fill_value argument instead of promote? The default (fill_value=dtypes.NA) would do type promotion for integer dtypes and always fill with NA. Other values (e.g., fill_value=0) could be used to avoid type promotion with an integer coordinate.

Advantages:

• No special cases to keep track of.
• Consistent with other xarray methods that take a fill_value argument.

Disadvantages:

• No built-in way to raise an error instead of promotion (but users could do this themselves pretty easily)
• No built-in way to "only promote if necessary" (but this is a weird non-type stable API that doesn't work great with Dask, anyways)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/3871#discussion_r396887940, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARK43Q3GISCTFURNDUDQNDRJAUTJANCNFSM4LP7WEMA .

In general this is really nicely put together. My main asks:

1. Remove the unrelated API change to map
2. Think about if the alternative of returning an arbitrary value rather than promotion or raising an error.

For details see the comments above. If we want to support (2), then it might make sense to use a string for selecting values of promote rather than True/False/None (e.g., so we can include the option to return an arbitrary coordinate value).

LGTM! Any other thoughts before we merge?

I fixed the extra space in the docstring and moved the business logic to computation.py.

Do you mean non-dimension coordinates?

Yes, thanks for clarifying

Does that even make sense in a general way? If they are 1D and tied to just one dimension coordinate that could be done, but if they are not tied to any dimension or tied to multiple dimensions or otherwise not 1D I am not sure what it would mean to take the idxmin/idxmax of them.

Yes, it wouldn't work in all cases, fair point. There are some cases in which it would work though, I'm unsure if it would be too complicated an interface to return them depending on whether it would work.

(and completely fine to contemplate these later)

The one complication is that using DataArray.idxmax and DataArray.idxmin assumes that the Dataset would only ever contain DataArray objects

I hear you and share the impulse that baking this in seems not ideal. Though I think it's a reasonable compromise to make, and there are no plans to deviate from it.

(Ideally maybe we have a ._contained_class attribute on the dataset which is almost always DataArray)

I think having a lambda is fine too.

@max-sixty

To what extent should this support non-index coordinates?

I am not familiar with non-index coordinates, what are those?

Do you mean non-dimension coordinates? Does that even make sense in a general way? If they are 1D and tied to just one dimension coordinate that could be done, but if they are not tied to any dimension or tied to multiple dimensions or otherwise not 1D I am not sure what it would mean to take the idxmin/idxmax of them.

@keewis @max-sixty The new commit with the requested changes has been pushed to this branch (except for the map one, pending ongoing discussion). Please take a look.

@keewis @max-sixty The map thing is purely a convenience function. I know there are other ways to do it, but since I thought this would be a useful feature for users in its own right, I did it that way. But of course I can do it another way if you disagree.

The one complication is that using DataArray.idxmax and DataArray.idxmin assumes that the Dataset would only ever contain DataArray objects. That may be mostly the case now, but I didn't want to bake that into the code. I could do it using a lambda or nested function, but as I said I thought this approach had other benefits to users.

I will address the other comments inline.

This looks extremely good and thorough! Thanks @toddrjen !

I'll have a proper look through later. I see a couple of minor questions I'll add in too. Others feel free to get ahead of me!