issue_comments

Wow, didn't look for a week. Very happy 🎉! Thank you @dcherian and @max-sixty for all your input.

Must be one of the largest first contributions...

I was inspired by the hacktoberfest last year. Took a little bit over a month though 😛

Thanks @mark-boer ! Must be one of the largest first contributions...

+1 re merge + experimental warning; maybe we should do this more often. Cheers @dcherian

Merging. I've added an experimental warning to the docstrings and we can discuss the IndexVariable situation here: https://github.com/pydata/xarray/issues/3868

Thanks @mark-boer this is a major contribution for your first PR!

Hello @mark-boer! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2020-03-18 21:58:02 UTC

Agree!

We could add an "Experimental" label and then worry less about future changes

Extremely good points @mark-boer

I propose we merge and open an issue to decide what to do with IndexVariables.

In some instances extrapolating all coords, can lead to some unwanted behaviour. Would you suggest we only interpolate the indexes?

How would we handle unsorted indexes?

How would we extrapolate all the different kind of indexes, like the MultiIndex or CategoricalIndex?

I agree; I can't see easy solutions to these. If there are evenly spaced indexes (e.g. dates, grids), then it's easy to know what to do. But there are plenty of times it's difficult, if not intractable.

One option to merge something useful without resolving these questions is to return Variables only and label this method experimental. I think this is immediately useful for the cases where these difficult questions don't need to be answered.

Of course if there are good answers to the questions, then even better!

:) I think we need to extrapolate indexes by default. It seems like the most sensible option.

Sorry, obviously that is a solution ;-). But I do have some concerns:

In some instances extrapolating all coords, can lead to some unwanted behaviour. Would you suggest we only interpolate the indexes?

How would we handle unsorted indexes?

How would we extrapolate all the different kind of indexes, like the MultiIndex or CategoricalIndex?

Hmm, I don't really see a solution. What do you suggest?

:) I think we need to extrapolate indexes by default. It seems like the most sensible option.

Edit: But it's more awkward for indexes than non-index coords. The index becomes less useful with non-unique values, and generally indexes don't have nulls. I'm not sure what the other options would be: to some extent it's the intersection of pad with xarray's data model.

Hmm, I don't really see a solution. What do you suggest?

I do agree it can be confusing, but it is not unique in xarray. Dataset.shift only shifts data_vars, bfill and ffill only fill data_vars, etc.

I agree, I wouldn't have expected coords to be included given existing behavior. People can switch coords <> data_vars as needed, so there's an escape hatch

Edit: But it's more awkward for indexes than non-index coords. The index becomes less useful with non-unique values, and generally indexes don't have nulls. I'm not sure what the other options would be: to some extent it's the intersection of pad with xarray's data model.

The big outstanding issue is what to do about dimension coordinates or indexes. Currently this PR treats all variables in coords different from those in data_vars. I think this is confusing.

I do agree it can be confusing, but it is not unique in xarray. Dataset.shift only shifts data_vars, bfill and ffill only fill data_vars, etc.

Personally I think that extrapolating data coordinates without specifically settings a keyword or flag could also be confusing. I occasionally have data in my coords that make no sense to extrapolate. I do agree that adding an extrapolate_coord option or keyword of some sorts would be cool in the future.

Both defaults could be really useful, I'm still in a bit of doubt. Are there any other people that might want to weigh in?

This looks excellent @mark-boer , thank you!

I will try and have a proper look through today (but don't wait for me)

I pushed some minor changes.

I think this is ready to go in.

The big outstanding issue is what to do about dimension coordinates or indexes. Currently this PR treats all variables in coords different from those in data_vars. I think this is confusing.

I am thinking that we want to use linear extrapolation for IndexVariables by default and apply the same padding mode to all other variables. The reasoning being that IndexVariables with NaNs are hard to deal with and it's hard to fill them in: padded["x"] = padded.x.drop_vars("x").interpolate_na("x", fill_value="extrapolate")

Hi @dcherian

Once again an excellent code review :-)

I adressed most points in your review, except for the function signature of DataArray.pad. I think I prefer the default values of None. Now numpy/Dask will throw an error when you set invalid combinations e.g. constant_values=0, and set the mode="mean". The np.nan is unfortunate, but this is the difference between np.nan and xarray.dtypes.NaN. I could set the default to dtypes.NaN, but then you lose the automatic error when setting an explicit value with mode!="constant".

I can also see that something like coords_mode being useful, but I think it would be wise to wait and see how pad gets used and what issues we run into along the way.

I also ran into a small issue with pint (https://github.com/hgrecco/pint/issues/1026), but I'll get to that once this PR is merged.

Hi @dcherian and @fujiisoup, appart from the issues I raised in my last comment, I think this PR is close to done.

Thx in advance

Hi everyone, happy new year :tada:

I feel like this PR is slowly getting into a descent shape. But I could use your input on a couple of subjects:

How should we handle the default padding? I currently pad with dtypes.NaN this is similar behaviour as used by the replaced variable.pad_with_fill_valueand is consistent with theshift` method.

We could chose to pad with 0's as numpy does it, or we could force the user to choose a constantt_value, but I don't think I would prefer this.

How should the coordinates of a DataArray/set be padded? I chose default padding except for modes "edge", "reflect", "symmetric", "wrap". @dcherian noted that this could make sense in some situations, but that I would require some further discussion.

Personally I think it more often than not it makes sense, but padding with NaN's should also work fine.

dask_array_compat Also I created a workaround for the changing of dtype by Dask.pad mode=mean and added an additional check to validate the output shape of dask.pad. Let me know what you think about this, I'm not 100% convinced...

I used a couple of # type: ignore as any work around the mypy errors lead to uglier code. But if any of you have some suggestions, I'm happy to hear them

It seems like we have some value mismatches on dask==1.2.

The reason that mean and reflect are marked as xfail are because there too there are value mismatches. I'll see if I can find a workaround for this failing test.

It seems like we have some value mismatches on dask==1.2. @fujiisoup is this the error you found?

``` =================================== FAILURES =================================== _ TestVariableWithDask.testpad[xr_arg0-np_arg0-linear_ramp] __

self = <xarray.tests.test_variable.TestVariableWithDask object at 0x7fb69b629b00> mode = 'linear_ramp', xr_arg = {'x': (2, 1)}, np_arg = ((2, 1), (0, 0), (0, 0))

@pytest.mark.parametrize(
    "mode",
    [
        pytest.param("mean", marks=pytest.mark.xfail),
        pytest.param("median", marks=pytest.mark.xfail),
        pytest.param("reflect", marks=pytest.mark.xfail),
        "edge",
        "linear_ramp",
        "maximum",
        "minimum",
        "symmetric",
        "wrap",
    ],
)
@pytest.mark.parametrize(
    "xr_arg, np_arg",
    [
        [{"x": (2, 1)}, ((2, 1), (0, 0), (0, 0))],
        [{"y": (0, 3)}, ((0, 0), (0, 3), (0, 0))],
        [{"x": (3, 1), "z": (2, 0)}, ((3, 1), (0, 0), (2, 0))],
    ],
)
def test_pad(self, mode, xr_arg, np_arg):
    data = np.arange(4 * 3 * 2).reshape(4, 3, 2)
    v = self.cls(["x", "y", "z"], data)

    actual = v.pad(mode=mode, **xr_arg)
    expected = np.pad(data, np_arg, mode=mode,)

  assert_array_equal(actual, expected)

E AssertionError: E Arrays are not equal E
E (mismatch 2.3809523809523796%) E x: array([[[ 0, 0], E [ 0, 0], E [ 0, 0]],... E y: array([[[ 0, 0], E [ 0, 0], E [ 0, 0]],...

xarray/tests/test_variable.py:821: AssertionError _ TestVariableWithDask.testpad[xr_arg1-np_arg1-linear_ramp] __

self = <xarray.tests.test_variable.TestVariableWithDask object at 0x7fb69df9eb00> mode = 'linear_ramp', xr_arg = {'y': (0, 3)}, np_arg = ((0, 0), (0, 3), (0, 0))

@pytest.mark.parametrize(
    "mode",
    [
        pytest.param("mean", marks=pytest.mark.xfail),
        pytest.param("median", marks=pytest.mark.xfail),
        pytest.param("reflect", marks=pytest.mark.xfail),
        "edge",
        "linear_ramp",
        "maximum",
        "minimum",
        "symmetric",
        "wrap",
    ],
)
@pytest.mark.parametrize(
    "xr_arg, np_arg",
    [
        [{"x": (2, 1)}, ((2, 1), (0, 0), (0, 0))],
        [{"y": (0, 3)}, ((0, 0), (0, 3), (0, 0))],
        [{"x": (3, 1), "z": (2, 0)}, ((3, 1), (0, 0), (2, 0))],
    ],
)
def test_pad(self, mode, xr_arg, np_arg):
    data = np.arange(4 * 3 * 2).reshape(4, 3, 2)
    v = self.cls(["x", "y", "z"], data)

    actual = v.pad(mode=mode, **xr_arg)
    expected = np.pad(data, np_arg, mode=mode,)

  assert_array_equal(actual, expected)

E AssertionError: E Arrays are not equal E
E (mismatch 16.66666666666667%) E x: array([[[ 0, 1], E [ 2, 3], E [ 4, 5],... E y: array([[[ 0, 1], E [ 2, 3], E [ 4, 5],...

xarray/tests/test_variable.py:821: AssertionError _ TestVariableWithDask.testpad[xr_arg2-np_arg2-linear_ramp] __

self = <xarray.tests.test_variable.TestVariableWithDask object at 0x7fb69c609860> mode = 'linear_ramp', xr_arg = {'x': (3, 1), 'z': (2, 0)} np_arg = ((3, 1), (0, 0), (2, 0))

@pytest.mark.parametrize(
    "mode",
    [
        pytest.param("mean", marks=pytest.mark.xfail),
        pytest.param("median", marks=pytest.mark.xfail),
        pytest.param("reflect", marks=pytest.mark.xfail),
        "edge",
        "linear_ramp",
        "maximum",
        "minimum",
        "symmetric",
        "wrap",
    ],
)
@pytest.mark.parametrize(
    "xr_arg, np_arg",
    [
        [{"x": (2, 1)}, ((2, 1), (0, 0), (0, 0))],
        [{"y": (0, 3)}, ((0, 0), (0, 3), (0, 0))],
        [{"x": (3, 1), "z": (2, 0)}, ((3, 1), (0, 0), (2, 0))],
    ],
)
def test_pad(self, mode, xr_arg, np_arg):
    data = np.arange(4 * 3 * 2).reshape(4, 3, 2)
    v = self.cls(["x", "y", "z"], data)

    actual = v.pad(mode=mode, **xr_arg)
    expected = np.pad(data, np_arg, mode=mode,)

  assert_array_equal(actual, expected)

E AssertionError: E Arrays are not equal E
E (mismatch 5.208333333333329%) E x: array([[[ 0, 0, 0, 0], E [ 0, 0, 0, 0], E [ 0, 0, 0, 0]],... E y: array([[[ 0, 0, 0, 0], E [ 0, 0, 0, 0], E [ 0, 0, 0, 0]],... ```

Hi, @mark-boer. In #3587, I tried using dask's pad method but noticed a few bugs in older (but newer than 1.2) dask. For me, it would be very welcome if you add this method to dask_array_compat. Then, I would wait for merging #3587 until this PR is completed.

Thanks for your contribution :)