Hi. Sorry for my late reply. Well, I've just left this PR untouched.

More specifically, the approach here (i.e., passing indexes via the pad_width argument) may be tricky in the context of flexible indexes where multiple indexes/coordinates are allowed for one dimension.

I think we can just discard this PR if this does not fit with the index refactoring. This PR is not big anyway and maybe rewriting this functionality is faster.

Thanks @mathause I also think the current behavior is not perfect but the best.

I would expect both to return np.nan

I expect that np.nansum(ds) is equivalent to np.sum(not nan values) and thus should be 0, while np.mean should be NaN as @dcherian pointed out.

To me, the future average function would also return np.nan for all nan slices.

Maybe a Jupyter issue and not related to libraries in use?

I see. Indeed, I didn't see any significant difference among branches.

I may be able later today when I am back to my main computer

I tried but I think maybe better to wait for your update.

Maybe can we measure the first-loading time? I observe the first-loading time is very long... (movie)

The only way I was able to see it was to use the Web Dev tools that come as part of Firefox or Chrome.

Can you tell me more about this? I'll try to reproduce and measure the performance.

https://user-images.githubusercontent.com/6815844/124090964-4e601e80-da90-11eb-8333-7c2a25a8f33d.mp4

did you try if there are differences when running an individual cell, not just when loading the page the first time?

I tried to measure the performance by running all the cells as shown in the image but I could not find any significant difference.

However, I'm not very confident if this actually measures the css performance.

@SimonHeybrock, do you have any suggestions how to measure the peformance?

I am trying to measure the performance of master, this PR and mine (which fixes this PR to be compatible with dark mode) but couldn't see any big difference in my environment.

What I did in this experiment is to make a notebook with hundreds of empty cells with xarray under these branches. Refreshed the browser to render the htmls. Number of cells are the same in all these experiments, but only the xarray branches (and produced html) are different.

Maybe we may need more cells? Any advice would be appreciated.

https://user-images.githubusercontent.com/6815844/124035536-9ef75d80-da37-11eb-9c78-a9c76d16da1a.mp4

movie top left: this branch top right: mine bottom left: master

Closed as the discussions can be continued in #5361 .

This

https://github.com/fujiisoup/xarray/blob/6225f158626e75977a0a944fbc09c50769884e35/xarray/static/css/style.css#L5-L29

looks working with a darkmode, but I'm not sure if this solves the original problem.

It looks to me that defining custom properties in html[theme=dark] may cause the same problem.

Confirmed that this also breaks the darkmode also in google colab.

@fujiisoup added the vscode dark mode support, maybe he has ideas.

I did it in #4036 but this was actually a workaround and should be improved by an expert. I'll take a look, but with little hope to fix.

There's a cumtrapz in https://github.com/fujiisoup/xr-scipy/blob/master/xrscipy/integrate.py. Does that help?

Now most of xr-scipy functionalities are already implemented in xarray and also I couldn't take time to maintain this package.

I think basic functionalities would be better to be integrated into xarray itself and cumulative_trapezoid would be a good candidate, as integrate is already there.

The implementation looks good to me. I didn't find any edge cases where duck_array_ops.cumulative_trapezoid behaves differently from scipy.integrate.cumtrapz.

Thank you @mathause for your suggestion. This looks all the tests are passing now.

Not sure why the doctest is failing. The same tests in test_dataset.py do not fail...

I was thinking to wait for the pad method implemented but forgot until now. I am not sure this is easily merginable, as the rolling.py has been updated for a while...

The first motivation was to implement the rolling operation for the periodic coordinate, but it is not yet implemented.

@mathause Oh, I missed this issue. Yes, this is implemented only for count.

the thing is that rolling itself is already quite complicated

Agreed. We need to clean this up.

One possible option would be to drop support of bottleneck. This does not work for nd-rolling and if we implement the nd-nanreduce, the speed should be comparable with bottleneck.

Hi @aulemahal

I think you want to interpolate along tas well as x, and y. If so, you can do

python In [8]: da.interp(t=dx['t'], y=dy, x=dx, method='linear') Out[8]: <xarray.DataArray (t: 2, u: 2)> array([[2., 3.], [2., 3.]]) Coordinates: * t (t) int64 10 12 y (u) float64 1.5 2.5 x (t, u) float64 1.5 1.5 1.5 1.5 * u (u) int64 45 55

If not, this fails as dx['t'] and da['t'] do not match each other. The error message can be improved. A contribution is welcome ;)

My last post was wrong.

I think this part overwrites the attrs,

https://github.com/pydata/xarray/blob/43a2a4bdf3a492d89aae9f2c5b0867932ff51cef/xarray/core/variable.py#L2028 https://github.com/pydata/xarray/blob/43a2a4bdf3a492d89aae9f2c5b0867932ff51cef/xarray/core/variable.py#L2073-L2076

The first line should be replaced by variable = self.copy(deep=False)

~~These lines are suspicious. Maybe we should copy attrs here not geting its reference.~~

https://github.com/pydata/xarray/blob/43a2a4bdf3a492d89aae9f2c5b0867932ff51cef/xarray/core/rolling.py#L498

https://github.com/pydata/xarray/blob/43a2a4bdf3a492d89aae9f2c5b0867932ff51cef/xarray/core/rolling.py#L498

@cyhsu Yes, because it is not yet released. (I'm not sure when the next release will be, but maybe a few months later) If you do pip install git+https://github.com/pydata/xarray, the current master will be installed in your system and interpolation over the chunks can be used. But note that this means you will install (a kind of) beta version.

@cyhsu Yes, in the current master.

Thanks @pums974 :)

Thanks, @keewis , for the clarification.

It was a bug in the documentation page but not in rolling.construct.

It should raise an error in this case, because for 2d rolling we need 2 dimension names, python rolling_da = r.construct(x="x_win", y='y_win' , stride=2) I corrected the documentation and error message.

Thanks @keewis for checking. I'm not sure what causes the error in rolling_da.mean("window_dim", skipna=False)

self._mapping_to_list should handle this problem. How can I get the details of this error? I just saw the time-out error...

Ah, this attrs = [ --> 102 "{k}->{v}".format(k=k, v=getattr(self, k)) 103 for k in list(self.dim) + self.window + self.center + [self.min_periods] 104 ] should be "{k}->{v}".format(k=k, v=getattr(self.dims, k)) not "{k}->{v}".format(k=k, v=getattr(self, k)) I'll send a fix.

Or maybe we can convolve over the shared dimensions. python da = xr.DataArray(np.random.randn(15, 30), dims=['x', 'y']) kernel = xr.DataArray(np.random.randn(3, 3), dims=['x', 'y']) da.convolve(kernel, mode='same') Other dimensions maybe broadcasted.

Maybe we can keep this issue open.

python da.convolve(kernel, x='kx', y='ky', mode='same') would be a possible API?

The contribution will be very much appreciated ;)

Maybe we can have a simpler API for convolution operation, though.

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.random.randn(15, 30), dims=['x', 'y']) ...: kernel = xr.DataArray(np.random.randn(3, 3), dims=['kx', 'ky']) ...: ...: da.rolling(x=3, y=3).construct(x='kx', y='ky').dot(kernel) Out[1]: <xarray.DataArray (x: 15, y: 30)> array([[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], ... [ nan, nan, -2.30319699e-01, 3.98542408e-01, 7.65734275e+00, -3.78602564e-01, -3.79670552e+00, -4.63870114e+00, 3.34264622e-02, -3.12097772e+00, -5.76697267e+00, 1.19804861e+00, -8.94696248e-01, 2.29308845e+00, -6.39524525e-01, 4.63574750e+00, 9.72065650e-01, -2.79080617e-01, -4.08284408e-01, 4.09498738e+00, 2.21513156e+00, 2.46188185e-01, -1.30140822e+00, -4.70525588e+00, -4.60012056e+00, 2.33333189e-01, -2.86204413e-01, -5.63190762e-01, 9.31915537e-01, 7.84188609e-01], [ nan, nan, 1.04286238e+00, -1.51693719e+00, 2.49199283e+00, 1.74931359e-01, -4.26361392e+00, -1.85066273e-01, -2.45780660e+00, -3.20920459e+00, -4.13765502e+00, -3.64119127e+00, 1.13819179e-01, -2.10588083e-01, -2.58307399e-02, -6.73602885e-01, 1.51186293e+00, 2.22395020e+00, 3.59169613e+00, 4.44203028e+00, 3.15528384e-01, -2.30913656e+00, 3.07864240e+00, -9.21743416e-01, -2.87995499e+00, -1.92025700e+00, -3.95047208e-01, 4.60378793e+00, 1.11828099e+00, 4.29419626e-01]]) Dimensions without coordinates: x, y

@max-sixty thanks for the review. merged

Thanks @max-sixty .

You are completely correct. As the test pass, I was fooling myself.

The reason was that the dataset I was using for the test does not have time and x simultaneously. So I was not testing the 2d-rolling but just 1d-rolling.

Fixed. Now it correctly fails for mean and std, but passes with max and sum

This PR looks good for me. Maybe we can wait for a few days in case anyone has some comments on it. If no comments, I'll merge this then.

Thanks @max-sixty for the review ;) I'll work for the update in a few days.

Could anyone kindly review this?

OK. If you have additional time, it would be nicer if you could add more comments on tests, like what is being tested there ;)

Thanks @pums974 for this update and sorry for my late response. It looks good but I'll take a deeper look in the next week.

Thanks, @jenssss for sending a PR. This looks good to me. Could you add a line for this contribution to our whatsnew?

I got an error for typechecking, only in CI but not in local, from the code that I didn't change.

A possible improvement will be nan-reduction methods for nd-rolling. Currently, we just use numpy nan-reductions, which is memory consuming for strided arrays.

This issue can be solved by replacing nan by appropriate values and applying nonnan-reduction methods, e.g., python da.rolling(x=2, y=3).construct(x='xw', y='yw').sum(['xw', 'yw']) should be the same with python da.rolling(x=2, y=3).construct(x='xw', y='yw', fill_value=0).sum(['xw', 'yw'], skipna=False) and the latter is much more memory efficient.

I'd like to leave this improvement to future PR.

I think now it is ready for review, though I'm sure tests miss a lot of edge cases. Maybe we can fix them if pointed out.

I think the preferred option for dealing with accidentally pushed changes is to push a "revert" commit generated from git revert

OK, understood.

but as long as we keep the master branch protected, it's always possible always possible to move forward by reverting changes -- there is no way to lose work.

Then, probably the most dangarous part was when I unprotected the master branch. I was afraid of messing up the commit history, but it is much better than losing entire commit history...

Another API concern. We now use min_periods, in which we implicitly assume one-dimension cases.

With nd-dimension, I think min_counts argument is more appropriate like bottleneck, which will limit the lower bound of the number of missing entries in the n-dimensional window.

Even if we leave it, we may disallow nd-argument of min_periods but keep it a scalar.

Hi @max-sixty

One alternative is to allow fluent args, like: ...but does that then seem like the second rolling is operating on the result of the first?

I couldn't think of it until just now. But yes, it sounds to me like a repeated rolling operation.

I'm being slow, but where is the nd-rolling algo? I had thought bottleneck didn't support more than one dimension?

No. With nd-rolling, we need to use numpy reductions. Its skipna=True operation is currently slow, but it can be improved replacing nan before the stride-trick.

OK, thanks.

So I think either a pre-push hook or git config branch.master.pushRemote no_push (but then you also can't push to your own master anymore) are the best way forward

Agreed. I'll use your pre-push hook. Thanks @keewis .

BTW, is it possible to disallow direct push to master on github? Maybe we only need to merge PRs and but not push.

OK. Done. Thanks. I'll use your script. Thanks.

And sorry again for my mistake.

Maybe I can unprotect the master, but I'm hesitating this action...

Thanks. but it looks the master is protected and I cannot force push. Total 0 (delta 0), reused 0 (delta 0) remote: error: GH006: Protected branch update failed for refs/heads/master. remote: error: Cannot force-push to this protected branch To https://github.com/pydata/xarray.git ! [remote rejected] master -> master (protected branch hook declined) error: failed to push some refs to 'https://github.com/pydata/xarray.git'

One thing I would like to implement in somday is multi-dimensional rolling operation. The 1-dimensional convolution can be done with rolling -> construct -> dot, as can be seen in the doc page (see the last paragraph of http://xarray.pydata.org/en/stable/computation.html#rolling-window-operations)

This is can be extended to multiple dimensions, but it may not be straightforward.

@cwerner ```python In [40]: idx = (da.count('y').cumsum() != 0) * (da.count('y')[::-1].cumsum()[::- ...: 1] != 0)

In [42]: da.isel(x=idx)
Out[42]: <xarray.DataArray (x: 3, y: 4)> array([[nan, 0., 2., nan], [nan, nan, nan, nan], [nan, 2., 0., nan]]) Dimensions without coordinates: x, y ``` Maybe this works, but I have no cleaner solution.

@keewis I think it is close to da.dropna(how='all') python In [12]: da.dropna('x', how='all').dropna('y', how='all') Out[12]: <xarray.DataArray (x: 2, y: 2)> array([[0., 2.], [2., 0.]]) Dimensions without coordinates: x, y I think supporting multiple dimensions for dropna is totally in our scope. Currently, dropna only works with a single dimension and da.dropna(how='all') does not work.

@cwerner Is it close to your example? If you don't want to drop all nans but only those located at the edges, the above example does not work.

Hum, ok, but I don't see how it would work if all points are between chunks (see my second example)

Maybe we can support sequential interpolation only at this moment. In this case, python res = data.interp(x=np.linspace(0, 1), y=0.5) can be interpreted as python res = data.interp(x=np.linspace(0, 1)).interp(y=0.5) which might not be too difficult.

I agree that it's better not to sort.

I think the #3953 fixes the case where the multiindex has unused levels. I had no better idea than #3953, but if it works without #3953, it would be better ;)

As for implementing this in dask, you may be right, it probably belong there, But I am even less use to their code base, and have no clue where to put it.

OK. Even so, I would suggest restructuring the code base; maybe we can add an interp1d equivalence into core.dask_array_ops.interp1d which works with dask-arrays (non-xarray object). It'll be easier to test. The API should be the as same with scipy.interp.interp1d as possible.

In missing.py, we can call this function.

Also in my local environment, it gives AttributeError: 'memoryview' object has no attribute 'dtype'

The full stack trace is ``` _________ test_interpolate_1d[1-y-cubic] ____________

method = 'cubic', dim = 'y', case = 1

@pytest.mark.parametrize("method", ["linear", "cubic"])
@pytest.mark.parametrize("dim", ["x", "y"])
@pytest.mark.parametrize("case", [0, 1])
def test_interpolate_1d(method, dim, case):
    if not has_scipy:
        pytest.skip("scipy is not installed.")

    if not has_dask and case in [1]:
        pytest.skip("dask is not installed in the environment.")

    da = get_example_data(case)
    xdest = np.linspace(0.0, 0.9, 80)

    actual = da.interp(method=method, **{dim: xdest})

    # scipy interpolation for the reference
    def func(obj, new_x):
        return scipy.interpolate.interp1d(
            da[dim],
            obj.data,
            axis=obj.get_axis_num(dim),
            bounds_error=False,
            fill_value=np.nan,
            kind=method,
        )(new_x)

    if dim == "x":
        coords = {"x": xdest, "y": da["y"], "x2": ("x", func(da["x2"], xdest))}
    else:  # y
        coords = {"x": da["x"], "y": xdest, "x2": da["x2"]}

    expected = xr.DataArray(func(da, xdest), dims=["x", "y"], coords=coords)

  assert_allclose(actual, expected)

xarray/tests/test_interp.py:86:

xarray/testing.py:132: in compat_variable return a.dims == b.dims and (a._data is b._data or equiv(a.data, b.data)) xarray/testing.py:31: in _data_allclose_or_equiv return duck_array_ops.allclose_or_equiv(arr1, arr2, rtol=rtol, atol=atol) xarray/core/duck_array_ops.py:221: in allclose_or_equiv arr1 = np.array(arr1) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/array/core.py:1314: in array x = self.compute() ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/base.py:165: in compute (result,) = compute(self, traverse=False, kwargs) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/base.py:436: in compute results = schedule(dsk, keys, kwargs) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:527: in get_sync return get_async(apply_sync, 1, dsk, keys, kwargs) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:494: in get_async fire_task() ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:466: in fire_task callback=queue.put, ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:516: in apply_sync res = func(*args, kwds) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:227: in execute_task result = pack_exception(e, dumps) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/local.py:222: in execute_task result = _execute_task(task, data) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/core.py:119: in _execute_task return func(args2) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/optimization.py:982: in call return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args))) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/core.py:149: in get result = _execute_task(task, cache) ../../../anaconda3/envs/xarray/lib/python3.7/site-packages/dask/core.py:119: in _execute_task return func(args2) xarray/core/missing.py:830: in _dask_aware_interpnd return _interpnd(var, old_x, new_x, func, kwargs) xarray/core/missing.py:793: in _interpnd x, new_x = _floatize_x(x, new_x) xarray/core/missing.py:577: in _floatize_x if _contains_datetime_like_objects(x[i]): xarray/core/common.py:1595: in _contains_datetime_like_objects return is_np_datetime_like(var.dtype) or contains_cftime_datetimes(var) xarray/core/common.py:1588: in contains_cftime_datetimes return _contains_cftime_datetimes(var.data)

array = <memory at 0x7f771d6daef0>

def _contains_cftime_datetimes(array) -> bool:
    """Check if an array contains cftime.datetime objects
    """
    try:
        from cftime import datetime as cftime_datetime
    except ImportError:
        return False
    else:

      if array.dtype == np.dtype("O") and array.size > 0:

E AttributeError: 'memoryview' object has no attribute 'dtype'

xarray/core/common.py:1574: AttributeError ```

Hi @pums974

Thanks for sending the PR. I'm working to review it, but it may take more time.

A few comments; Does it work with an unsorted destination? e.g., python da.interp(y=[0, -1, 2])

I'm feeling that the basic algorithm, such as np.interp-equivalence, should be interpreted in upstream. I'm sure Dask community welcomes this addition. Do you have an interest on it?

@dcherian. Now I understood. Your working examples were really nice for me to understand the idea. Thank you for this clarification.

I think the use of this convention is the best idea to save MultiIndex in netCDF. Maybe we can start implementing this?

@dcherian I think the problem is how to serialize MultiIndex objects rather than the array itself. In your encoded, how can we tell the MultiIndex is [('a', 1), ('b', 1), ('a', 2), ('b', 2)] or [('a', 1), ('a', 2), ('b', 1), ('b', 2)]? Maybe we need to store similar objects to landpoint for level variables, such as latpoint and lonpoint.

I think just using reset_index is simpler and easier to restore.

Do we already have something similar encoding (and decoding) scheme to write (and read) data? (does CFTime use a similar scheme?) I think we don't have a scheme to save multiindex yet but need to manually convert by reset_index.

1077

Maybe we can decide this encoding-decoding API before #1603.

@dcherian Though I have no experience with this gather compression, it looks that python-netcdf4 does not have this function impremented.

One thing we can do is sparse -> multiindex -> reset_index > netCDF or maybe we can even add a function to skip constructing a multiindex but just make flattened index arrays from a sparse array.

Reading with chunks load the memory more than reading without chunks, but not loading an amount of memory equals to the size of the array (300MB for a 800MB array in the example below). And by the way, also loading up the memory a bit more when stacking.

I think it depends on the chunk size. If I use the chunks chunks=dict(x=128, y=128), the memory usage is RAM: 118.14 MB da: 800.0 MB RAM: 119.14 MB RAM: 125.59 MB RAM: 943.79 MB

When stacking a chunked array, only chunks alongside the first stacking dimension are conserved, and chunks along the second stacking dimension seem to be merged.

I am not sure where 512 comes from in your example (maybe dask does something). If I work with chunks=dict(x=128, y=128), the chunksize after the stacking was (100, 16384), which is reasonable (z=100, px=(128, 128)).

A workaround could have been to save the data already stacked, but "MultiIndex cannot yet be serialized to netCDF".

You can do reset_index before saving it into the netCDF, but it requires another computation when creating the MultiIndex after loading.

Thank you for raising an issue. I confirmed this problem is reproduced.

Since our Lazyarray does not support the reshaping, it loads the data automatically. This automatic loading happens in many other operations.

For example, if you multiply your array by a scalar, python mda = da *2 It also loads the data into memory. Maybe we should improve the documentation.

FYI, using dask arrays may solve this problem. To open the file with dask, you could add chunks keywords, python da = xr.open_dataarray("da.nc", chunks={'x': 16, 'y': 16}) Then, the reshape will be a lazy operation too.

Thanks @DWesl Maybe better to continue discussion either in #3297. I'll close this issue. Thanks for pointing it out.

@dcherian

Personally, I think the h5netcdf workaround is good enough until there is a CF standard for writing complex numbers.

Agreed. Thanks for your thoughts.

I'll merge this tomorrow.

Maybe I'll merge this in a few days.

Added a style for colab darkmode according to googlecolab/colabtools/issues/1214 and now it works also in colab dark theme :)

If no further comments, I'll merge this in a day.

Thanks, @shoyer and @DocOtak for the suggestions.

It looks like there may be some standard ways to detect dark vs light mode in CSS? https://medium.com/js-dojo/how-to-enable-dark-mode-on-your-website-with-pure-css-32640335474

It looks not working in vscode...

VS Code will tell you if it is in "dark" "light" or "high contrast" modes https://code.visualstudio.com/api/extension-guides/webview#theming-webview-content

In #4036 I used css body.vscode-dark { } code block, but maybe more general solution would be better if available...

It looks that Pandas is taking a very different approach and codebase and I don't think it is easy to adapt their approach...

I am not familiar with the css staff in jupyter but the simplest approach may be just to disable the text- and background-coloring but use the default color only. Then, our html repr becomes less pretty but maybe more robust.

pandas has a good style. We may be able to take it.

It is how it looks like in Light mode

Here is the css definition https://github.com/pydata/xarray/blob/59b470f5d1464366dc55b082618ea87da8fbc9af/xarray/static/css/style.css#L5-L14

It looks like that --jp-content-font-color0 and --jp-layout-color0 come from the theme but the others come from our default values. I have no idea yet how we can manage this...

Ah, OK. Makes sense. Thanks.

Hi, @zxdawn

Thank you for raising the issue. I think you need an actual value of z as your b.expand_dims('z') does not have a value for z but it only knows the z is the dimension name.

You can do like python b['z'] = 3 # add a scalar coordinate named 'z' to add a value (we call it coordinate) for z Then, your script will work, b = b.expand_dims('z') # expand 2d to 3d comb = xr.concat([a, b], dim='z')

Thanks, @delgadom, for reporting this issue. Reproduced.

I'll take a look.

Agreed with @max-sixty. I also like sel and isel as they are clearly distinguishable. It is not clear to me if parenthesis corresponds to sel or isel.

For me, the largest drawback of sel and isel is the fact that autocompleters can not suggest the dimension names (it is another issue though)

An inspection of the dask dashboard indicates that the computation is not distributed among workers though. How could I make sure this happens?

Ah, I have no idea... Are you able to distribute the function some_exp without wrapping by xarray?

Within my limited knowledge, it may be better to prepare another function that distributes some_exp over the workers and put this another function into apply_ufunc, but I am not 100% sure. Probably there is a better way...

Hi @johnomotani . Probably I have no time to finish this up and this is already too old. It would be nice if someone can update this PR.

Is python xr.apply_ufunc(some_exp, ds.x, ds.y, dask='parallelized', output_dtypes=[float], output_sizes={'stats': Nstats}, output_core_dims=[['stats']], vectorize=True) what you want? This gives ```python

<xarray.DataArray (x: 10, y: 20, stats: 5)> array([[[1., 1., 1., 1., 1.], [1., 1., 1., 1., 1.], [1., 1., 1., 1., 1.], ... [1., 1., 1., 1., 1.], [1., 1., 1., 1., 1.]]]) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 * y (y) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Dimensions without coordinates: stats

In [26]: ```

How about passing an Index instead of just a simple integer to the pad method? ```python In [4]: da = xr.DataArray([0.5, 1.5, 2.5], dims=['x'], coords={'x': [0, 1, 2]})

In [5]: da
Out[5]: <xarray.DataArray (x: 3)> array([0.5, 1.5, 2.5]) Coordinates: * x (x) int64 0 1 2

In [8]: da.pad(x=([-1, -2], 0))
Out[8]: <xarray.DataArray (x: 5)> array([nan, nan, 0.5, 1.5, 2.5]) Coordinates: * x (x) int64 -1 -2 0 1 2 ```

Thank you, @mancellin, for sending the fix. And thank you @max-sixty for the review. It looks all great to me.

Merging. Have a good weekend:)

@mancellin Sorry for my no response. Yes, there may be some possible workarounds, but nowadays I have less spare time... Do you have the interest to send a PR?

Thanks, @dcherian and @keewis , for keeping this updated. Merging.

Thanks, @mathause :)

I'll merge this after the conflict in whats-new.rst is solved.

I'll merge this tomorrow if no more commens.

I have no strong opinion, but if most of the arithmetic in xarray uses join='inner', then it would be nicer to do so here too.

I think this PR is ready for review.

But I mistakenly thought that there was a performance penalty to doing this.

Yes, construct(stride=2) does exactly the same thing before returning an array. https://github.com/pydata/xarray/blob/ff75081304eb2e2784dcb229cc48a532da557896/xarray/core/rolling.py#L242

Hi @mark-boer for raising an issue. I am not sure if I got the point exactly, but the following is similar to what you want? python In [81]: arr = xr.DataArray(np.arange(4), dims=("x",)) ...: arr.rolling(x=2).construct("roll_x").isel(x=slice(1, None, 2)) Out[81]: <xarray.DataArray (x: 2, roll_x: 2)> array([[0., 1.], [2., 3.]]) Dimensions without coordinates: x, roll_x

xref: #3670

Hi @sfinkens. Thank you for raising an issue.

I think what you actually want would be ```python In [16]: ds = xr.Dataset({'data': ('x', [1, 2]), ...: 'x': ('x', [1, 2]) ...: }, coords={'x_bnds': (('x', 'bnds'), [[0.5, 1.5], [1.5, ...: 2.5]])}) ...: ds['x'].attrs['bounds'] = 'x_bnds' ...: ds = ds.expand_dims({'time': [0]})

In [17]: ds
Out[17]: <xarray.Dataset> Dimensions: (bnds: 2, time: 1, x: 2) Coordinates: * time (time) int64 0 * x (x) int64 1 2 x_bnds (x, bnds) float64 0.5 1.5 1.5 2.5 Dimensions without coordinates: bnds Data variables: data (time, x) int64 1 2 `` wherex_bnds` would be a coordinate rather than a data variable.

Thanks for reporting again. OK. It looks there are several places to be fixed.

Please add comments here if you find another not-working case.

Let's close this after #3670 is merged.

I don't think the check failure is related.

Thanks, @mancellin

I sent a quick fix. Please feel free to comment there.

Thanks, @yohai !

thanks, @hazbottles Merged

What is the best way to save sparse array into a disc?

One naive way would be to use stack -> reset_index, but it flattened coordinates and if there is another variable that depends on these coordinates, they will be also flattened and may consume a lot of space.

@niowniow Thank you for your contribution!

I think stride option is a good idea. One thing is how to implement this efficient nan-reduction method.

Currently, we use 'bottleneck' if it is installed for speeding up nan-ops, but bottleneck does not support stride option. Another problem is inefficiency of nan-ops of numpy for strided arrays; he copies the strided array into full array and replace np.nan by zero before the reduction.

One way we could do is 1. skip using 'bottleneck' if stride is other than 1 2. implement our nan-ops for rolling. For example, for nansum, we can replace np.nan by 0 before creating the strided arrays and apply usual sum for the strided array.

In rolling.count, we did a similar thing.

Hi, @mark-boer. In #3587, I tried using dask's pad method but noticed a few bugs in older (but newer than 1.2) dask. For me, it would be very welcome if you add this method to dask_array_compat. Then, I would wait for merging #3587 until this PR is completed.

Thanks for your contribution :)

This behaviour, however, seems to be slightly different from the .loc API of pandas.DataFrame which can take boolean arrays for selection. Is there a reason for the discrepancy?

Hi, @roxyboy

This is just because that multidimensional boolean indexing is not yet implemented in xarray (#1887). The one-dimensional indexing would work with .loc, ```python In [2]: da = xr.DataArray([0, 1, 2], dims=['x'])

In [3]: da.loc[da < 1]
Out[3]: <xarray.DataArray (x: 1)> array([0]) Dimensions without coordinates: x ```

FYI, in xarray, probably .sel and .isel methods are more convenient than .loc, as we don't need to remember the dimension order. For the above (my) example, I would write python da.isel(x=da < 1) instead of da.loc[da < 1].

Do we arrive at the consensus here for API to change the sparse or numpy array? xref #3542

To make it sparse array, to_sparse() would be better? How about or as_sparse()? + to_sparse() is probably consistent to todense() method + as_sparse() sounds similar to sparse's function, e.g. sparse.as_coo

To change the backend back from sparse array, to_dense() would be better? FYI, sparse uses todense().

I personally like as_sparse or as_numpy (or as_dense?), which sounds similar to as_type, which gives xarray object not dtype itself.

Thanks, @max-sixty, for the review :)