issue_comments

FYI the apply / map change went in a couple of years ago.

We still don't have an apply_raw. I think it probably makes sense to consolidate this with https://github.com/pydata/xarray/issues/1618, so I'll close this, even though it has some good discussion.

@max-sixty I like your proposal!

I more strongly think that apply is a confusing and non-standard term for "run this function on each item in this container", even if it's pandas' name.

This is a fair point. map is definitely the standard name in the context of "map reduce" type operations.

What are your thoughts re:

• Adding map as the documented approach for run-on-each on Dataset & GroupBy
• Adding apply_raw (or similar) as a new function that runs functions on the 'raw' arrays
• Keeping apply around for backward-compat, similar to the drop case

I would support this.

OK. Does that inform your view on map vs apply?

I more strongly think that apply is a confusing and non-standard term for "run this function on each item in this container", even if it's pandas' name.

I'm keener to offer map an as option that necessarily reusing apply.

What are your thoughts re: - Adding map as the documented approach for run-on-each on Dataset & GroupBy - Adding apply_raw (or similar) as a new function that runs functions on the 'raw' arrays - Keeping apply around for backward-compat, similar to the drop case

+@dcherian

@max-sixty thanks for pushing this along!

I think I'm coming to appreciate backwards compatibility as an important consideration more and more these days. It's just really painful to reuse methods for something entirely different.

This makes me lean towards adding separate apply_raw() methods. The name is definitely less memorable than apply, but on the other hand it is also definitely easier to guess the difference between apply/apply_raw then apply/map.

I put the change for Dataset.apply -> Dataset.map in. Should we do the same for GroupBy?

I think those are probably the two easiest decisions to make (and hopefully will kick off moving this issue forwards)

Edit: the reason I hesitated for GroupBy is that it's not exactly the same: the object returned isn't the same (i.e. Dataset.map returns a Dataset, while GroupBy.map would return a Dataset)

I don't think we should consider ourselves beholden to pandas's bad names, but we should definitely try to preserve backwards compatibility and interpretability for users.

Going back to Python itself: - apply(func, args, kwargs) (from Python 2.x) is equivalent to func(*args, **kwargs) - map() maps a function over each element of an iterable - functools.reduce() applies a binary function repeatedly to convert an iterable into a single element

For xarray, we need: 1. a method for wrapping functions that work on unlabeled arrays 2. a method for mapping functions over each element of a Dataset or grouped object. 3. (possibly) a method for wrapping aggregation functions that act on unlabeled arrays

Currently, we call both (1) and (2) apply(), which is pretty confusing, and use reduce() for (3) even though it could potentially be a special case of (1) with a bit of extra magic and is quite unlike functools.reduce. In contrast, pandas calls both (1) and (2) apply() (using raw=True/raw=False to distinguish), and calls (3) aggregate or agg.

So long term, it could make sense to rename the current Dataset.apply()/GroupBy.apply() (case 2) to .map, and also rename .reduce() to the more generic .aggregate().

That said, I'm trying to imagine what the transition process for switching to new behavior for Dataset.apply looks like. We already will re-add dimensions to the output from calling functions in apply(), but at some point we have to a do a hard cut-off from passing DataArray objects to the function in apply to passing in a raw array.

I suppose we could do this by adding a raw keyword-only argument to .apply(): - If raw=False (current default), we would raise a warning about changing behavior and would pass-on DataArray objects to the applied function. Users would be encouraged to use .map() instead. - If raw=True (future default behavior), we would pass in raw numpy/dask arrays to the future function. - The dim argument might only be supported with raw=True.

We would end up with an extra extraneous raw argument, which we could remove/deprecate at our leisure.

I would like to rename rolling.reduce to rolling.apply to be consistent with pandas & groupby

+0.5 if map fails

One proposal: rename apply to map

-1 for pandas incompatibility.

I would like to rename rolling.reduce to rolling.apply to be consistent with pandas & groupby

One proposal: rename apply to map

Would we accept this? I'd be up for doing the PR to deprecate apply and introduce map. It makes a Dataset more consistent with a standard mapping interface. But it would be inconsistent with pandas and a rename of fairly widely used method

I would be +1 for apply and apply_raw.

Another option is to keep apply as-is for Dataset and GroupBy objects, but add a separate apply_raw method for applying functions that act on "raw" arrays. This would be a little more similar to pandas' apply with raw=True.

We could even do the raw=True keyword argument like pandas, but this is a little awkward because there are some additional arguments on apply_raw that don't make sense on apply (e.g., arguments that specify that some dimensions should be dropped or added).

Sounds good. It breaks the consistency with pandas' apply, but map is much more logical