issue_comments

@keewis, I agree a test would be nice, but what kind of test would you want to add? I mean, xarray just mimics the behaviour of np.pad and dask.array.pad. And the default behaviour of numpy was a bit funky, which is probably why they changed it.

My suggestion is: 1. replace the example with arr.pad(x=1, constant_values=1.23456789) and mention that the float is cast to int (or would you leave the example away?) 2. open a new issue to discuss the issue of assigning float to int

I agree, I think that would be a good solutions for now, I think replacing the example is fine. Maybe we could even open a new issue, to discuss how xarray functions handle np.nan.

Is dask.array.pad gonna handle casting the same way? It would be strange if the cast to float happens, depending on the underlying array type. But that discussion should probably happen in the newly opened issue ;-)

Hi, we had a similar discussion in de #3596, xarray makes a distinction between np.nan and xarray.dtypes.NaN. The current behaviour is consistent with that of other xarray functions such as shift. Though, I am personally not a big fan of this distinction.

Check e.g. this comment: https://github.com/pydata/xarray/pull/3596#discussion_r388612638

The example I posted in this comment: ```

da = xr.DataArray(np.arange(9).reshape(3,3), dims=("x", "y")) da.shift(x=1, fill_value=np.nan) array([[-9223372036854775808, -9223372036854775808, -9223372036854775808], [ 0, 1, 2], [ 3, 4, 5]]) Dimensions without coordinates: x, y

da.rolling(x=3).construct("new_axis", stride=3, fill_value=np.nan) <xarray.DataArray (x: 1, y: 3, new_axis: 3)> array([[[-9223372036854775808, -9223372036854775808, 0], [-9223372036854775808, -9223372036854775808, 1], [-9223372036854775808, -9223372036854775808, 2]]]) Dimensions without coordinates: x, y, new_axis ```

Hmm, so numpy changed its behaviour? Then this example, should probably also fail in numpy 1.20.

On a side note: I am not a big fan of the example in the doctest, it displays an edge case, which is not unique to pad.

I think the nicest solution would be to make the usage xarray.dtypes.NaN and np.nan equivalent. But this would require changes in all xarray functions that take some kind of fill_value.

I'll close this issue. Thank you @dcherian. I will have another look at the docs and if anything remains unclear, I'll have a look at adding a note or example.

Oh good point, after reading the docs, I thought that the min_periods was only related to the edges of the array.

Wow, didn't look for a week. Very happy 🎉! Thank you @dcherian and @max-sixty for all your input.

Must be one of the largest first contributions...

I was inspired by the hacktoberfest last year. Took a little bit over a month though 😛

:) I think we need to extrapolate indexes by default. It seems like the most sensible option.

Sorry, obviously that is a solution ;-). But I do have some concerns:

In some instances extrapolating all coords, can lead to some unwanted behaviour. Would you suggest we only interpolate the indexes?

How would we handle unsorted indexes?

How would we extrapolate all the different kind of indexes, like the MultiIndex or CategoricalIndex?

Edit: But it's more awkward for indexes than non-index coords. The index becomes less useful with non-unique values, and generally indexes don't have nulls. I'm not sure what the other options would be: to some extent it's the intersection of pad with xarray's data model.

Hmm, I don't really see a solution. What do you suggest?

The big outstanding issue is what to do about dimension coordinates or indexes. Currently this PR treats all variables in coords different from those in data_vars. I think this is confusing.

I do agree it can be confusing, but it is not unique in xarray. Dataset.shift only shifts data_vars, bfill and ffill only fill data_vars, etc.

Personally I think that extrapolating data coordinates without specifically settings a keyword or flag could also be confusing. I occasionally have data in my coords that make no sense to extrapolate. I do agree that adding an extrapolate_coord option or keyword of some sorts would be cool in the future.

Both defaults could be really useful, I'm still in a bit of doubt. Are there any other people that might want to weigh in?

Hi @dcherian

Once again an excellent code review :-)

I adressed most points in your review, except for the function signature of DataArray.pad. I think I prefer the default values of None. Now numpy/Dask will throw an error when you set invalid combinations e.g. constant_values=0, and set the mode="mean". The np.nan is unfortunate, but this is the difference between np.nan and xarray.dtypes.NaN. I could set the default to dtypes.NaN, but then you lose the automatic error when setting an explicit value with mode!="constant".

I can also see that something like coords_mode being useful, but I think it would be wise to wait and see how pad gets used and what issues we run into along the way.

I also ran into a small issue with pint (https://github.com/hgrecco/pint/issues/1026), but I'll get to that once this PR is merged.

Hi @dcherian and @fujiisoup, appart from the issues I raised in my last comment, I think this PR is close to done.

Thx in advance

Hi @fujiisoup, thx for your response.

That is exactly what I needed and what I used to do. But I mistakenly thought that there was a performance penalty to doing this. But the performance decrease turned out to be the result of the order in which I rolled and sliced.

Small update:

I was currently using: data = ( data.rolling(x=window_size ).construct("roll_x") .rolling(y=window_size ).construct("roll_y") .isel(x=slice(window_size - 1, None, stride), y=slice(window_size - 1, None, stride)) .stack(n=("x", "y")) )

but this was performing quite badly for larger arrays. However after having a look at DataArrayRolling and rewriting this piece of code to the following, performance was good.

data = ( data.rolling(x=window_size ).construct("roll_x") .isel(x=slice(window_size - 1, None, stride)) .rolling(y=window_size ).construct("roll_y") .isel(y=slice(window_size - 1, None, stride)) .stack(n=("x", "y")) )

I saw that DataArrayRolling.construct() also uses a slice (isel) to create the strided array, so there is not much of a performance penalty of first creating a rolling window with stride 1 and then slicing the data.

However, if you want to add strides to the 'Strided rolling' (#3607) it would be nice to still be able to create rolling windows that start at index 0.

By the way: feel free to close this issue, if you do not see a need for an alignment option.

Hi everyone, happy new year :tada:

I feel like this PR is slowly getting into a descent shape. But I could use your input on a couple of subjects:

How should we handle the default padding? I currently pad with dtypes.NaN this is similar behaviour as used by the replaced variable.pad_with_fill_valueand is consistent with theshift` method.

We could chose to pad with 0's as numpy does it, or we could force the user to choose a constantt_value, but I don't think I would prefer this.

How should the coordinates of a DataArray/set be padded? I chose default padding except for modes "edge", "reflect", "symmetric", "wrap". @dcherian noted that this could make sense in some situations, but that I would require some further discussion.

Personally I think it more often than not it makes sense, but padding with NaN's should also work fine.

dask_array_compat Also I created a workaround for the changing of dtype by Dask.pad mode=mean and added an additional check to validate the output shape of dask.pad. Let me know what you think about this, I'm not 100% convinced...

I used a couple of # type: ignore as any work around the mypy errors lead to uglier code. But if any of you have some suggestions, I'm happy to hear them

It seems like we have some value mismatches on dask==1.2.

The reason that mean and reflect are marked as xfail are because there too there are value mismatches. I'll see if I can find a workaround for this failing test.

I will create a WIP pull request tommorow.

@dcherian Thx! I found the Variable.pad_with_fill_value already, but I will also have a look at rolling and coarsen.

I have some questions related to this issue, I hope this is the correct place to ask those:

Dask added the pad method in version 1.7.0 according to its documentation. What is the minimum version of Dask that should be supported, I found Dask=1.2 is the continuous integration requirements.

Also, how should I handle implementation differences between numpy and Dask? E.g. in the current version of Dask and numpy I'm using: mode="mean" converts an array of integers to floats in Dask, but in numpy it keeps it an array of integers.

Has some1 started working on this? I thought I might give it a try ;-)