What happened?

I'm getting task graphs > 1GB, I think possibly because the full indexes are being included in every task?

What did you expect to happen?

Only the relevant sections of the index would be included

Minimal Complete Verifiable Example

```Python da = xr.tutorial.load_dataset('air_temperature')

Dropping the index doesn't generally matter that much...

len(cloudpickle.dumps(da.chunk(lat=1, lon=1)))

15569320

len(cloudpickle.dumps(da.chunk().drop_vars(da.indexes)))

15477313

But with .map_blocks, it really matters — it's really big with the indexes, and the same size without:

len(cloudpickle.dumps(da.chunk(lat=1, lon=1).map_blocks(lambda x: x)))

79307120

len(cloudpickle.dumps(da.chunk(lat=1, lon=1).drop_vars(da.indexes).map_blocks(lambda x: x)))

16016173

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem? Please describe. Currently ds.nbytes returns the size of the data.

But sys.getsizeof(ds) returns a very small number.

Describe the solution you'd like If we implement __sizeof__ on DataArrays & Datasets, this would work.

I think that would be something like ds.nbytes + the size of the ds container, + maybe attrs if those aren't handled by .nbytes?

Is your feature request related to a problem? Please describe.

Pandas has a .expanding aggregation, which is basically rolling with a full lookback. I often end up supplying rolling with the length of the dimension, and this is some nice sugar for that.

Describe the solution you'd like Basically the same as pandas — a .expanding method that returns an Expanding class, which implements the same methods as a Rolling class.

Describe alternatives you've considered Some options: – This – Don't add anything, the sugar isn't worth the additional API. – Go full out and write specialized expanding algos — which will be faster since they don't have to keep track of the window. But not that much faster, likely not worth the effort.

Pandas no longer has a fill_value parameter for interpolate.

Weirdly I wasn't getting this locally, on pandas 2.1.0, only in CI on https://github.com/pydata/xarray/actions/runs/6054400455/job/16431747966?pr=8138.

Removing it passes locally, let's see whether this works in CI

Would close #8125

• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

As discussed https://github.com/pydata/xarray/pull/5690#issuecomment-897280353

Is your feature request related to a problem?

Currently we call .rolling_exp like:

da.rolling_exp(date=20).mean()

20 refers to a "standard" window type — broadly "the same average distance as a simple rolling window. That works well, and matches the .rolling(date=20).mean() format.

But we also have different window types, and this makes it a bit incongruent:

da.rolling_exp(date=0.5, window_type="alpha").mean()

...since the window_type is completely changing the meaning of the value we pass to the dimension argument. A bit like someone asking "how many apples would you like to buy", and replying "5", and then separately saying "when I said 5, I meant 5 tonnes".

Describe the solution you'd like

One option would be:

.rolling_exp(dptr={"alpha": 0.5})

We pass a dict if we want a non-standard window type — so the value is attached to its type.

We could still have the original form for da.rolling_exp(date=20).mean().

Describe alternatives you've considered

No response

Additional context

(I realize I wrote this originally, all criticism directed at me! This is based on feedback from a colleague, which on reflection I agree with.)

Unless anyone disagrees, I'll try and do this soon-ish™

• [x] Closes https://github.com/pydata/xarray/issues/4473
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Very early effort — I found this harder than I expected — I was trying to use the existing groupby infra, but think I maybe should start afresh. The result of the numpy_groupies operation is a fully formed array, whereas we're used to handling an iterable of results which need to be concat.

I also added some type signature / notes and I was going through the existing code; mostly for my own understanding

If anyone has any thoughts, feel free to comment — otherwise I'll resume this soon

ref https://github.com/pydata/xarray/pull/2650#issuecomment-454164295

This seems an ugly way of accomplishing the goal; any ideas for a better way of doing this?

And stepping back, do others think a) it's helpful to skip variables in a dataset, and b) apply_ufunc should do this?

I'm happy to debug this further but before I do, is this an issue people have seen before? I'm running tests on master and hit an issue very early on.

FWIW I don't use netCDF, and don't think I've got that installed

Code Sample, a copy-pastable example if possible

```python ========================================================================== FAILURES ========================================================================== _________ ScipyInMemoryDataTest.test_bytesio_pickle __________

self = <xarray.tests.test_backends.ScipyInMemoryDataTest testMethod=test_bytesio_pickle>

@pytest.mark.skipif(PY2, reason='cannot pickle BytesIO on Python 2')
def test_bytesio_pickle(self):
    data = Dataset({'foo': ('x', [1, 2, 3])})
    fobj = BytesIO(data.to_netcdf())
    with open_dataset(fobj, autoclose=self.autoclose) as ds:

      unpickled = pickle.loads(pickle.dumps(ds))

E TypeError: can't pickle _thread.lock objects

xarray/tests/test_backends.py:1384: TypeError ```

Problem description

[this should explain why the current behavior is a problem and why the expected output is a better solution.]

Expected Output

Skip or pass backends tests

Output of xr.show_versions()

• [x] Closes https://github.com/pydata/xarray/issues/3814
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [ ] Closes #xxxx
• [ ] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Starting to get some fixes in.

It's going to be a long journey though. I think maybe we whitelist some files and move gradually through before whitelisting the whole library.

https://dev.azure.com/xarray/xarray/_build/results?buildId=695

```python

=================================== FAILURES =================================== ___ TestSparseVariable.test_unary_op ___

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f0b21b70>

def test_unary_op(self):

  sparse.utils.assert_eq(-self.var.data, -self.data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:285: AttributeError ___ TestSparseVariable.test_univariate_ufunc _____

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24ebc2bb38>

def test_univariate_ufunc(self):

  sparse.utils.assert_eq(np.sin(self.data), xu.sin(self.var).data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:290: AttributeError ___ TestSparseVariable.test_bivariate_ufunc ______

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f02a7e10>

def test_bivariate_ufunc(self):

  sparse.utils.assert_eq(np.maximum(self.data, 0), xu.maximum(self.var, 0).data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:293: AttributeError ___ TestSparseVariable.testpickle ____

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f04f2c50>

def test_pickle(self):
    v1 = self.var
    v2 = pickle.loads(pickle.dumps(v1))

  sparse.utils.assert_eq(v1.data, v2.data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:307: AttributeError ```

Any ideas?

Currently pandas-dev tests get 'stuck' on the conda install. The last instruction to run is the standard install:

$ if [[ "$CONDA_ENV" == "docs" ]]; then conda env create -n test_env --file doc/environment.yml; elif [[ "$CONDA_ENV" == "lint" ]]; then conda env create -n test_env --file ci/requirements-py37.yml; else conda env create -n test_env --file ci/requirements-$CONDA_ENV.yml; fi

And after installing the libraries, it prints this and then stops:

Preparing transaction: - - done Verifying transaction: | / \ | / - \ | / / done Executing transaction: \ | / - \ | / - \ | / - \ | / - \ | / - \ | / / - \ | / - \ done No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

I'm not that familiar with conda. Anyone have any ideas as to why this would fail while the other builds would succeed?

``` python In [1]: arr = xr.DataArray(np.random.rand(4, 3), ...: ...: [('time', pd.date_range('2000-01-01', periods=4)), ...: ...: ('space', ['IA', 'IL', 'IN'])]) ...: ...:

In [17]: arr.coords['space2'] = ('space', ['A','B','C'])

In [18]: arr Out[18]: <xarray.DataArray (time: 4, space: 3)> array([[ 0.05187049, 0.04743067, 0.90329666], [ 0.59482538, 0.71014366, 0.86588207], [ 0.51893157, 0.49442107, 0.10697737], [ 0.16068189, 0.60756757, 0.31935279]]) Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04 * space (space) |S2 'IA' 'IL' 'IN' space2 (space) |S1 'A' 'B' 'C' ```

Now try to select on the space2 coord:

``` python In [19]: arr.sel(space2='A')

ValueError Traceback (most recent call last) <ipython-input-19-eae5e4b64758> in <module>() ----> 1 arr.sel(space2='A')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataarray.pyc in sel(self, method, tolerance, indexers) 601 """ 602 return self.isel(indexing.remap_label_indexers( --> 603 self, indexers, method=method, tolerance=tolerance)) 604 605 def isel_points(self, dim='points', **indexers):

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataarray.pyc in isel(self, indexers) 588 DataArray.sel 589 """ --> 590 ds = self._to_temp_dataset().isel(indexers) 591 return self._from_temp_dataset(ds) 592

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataset.pyc in isel(self, **indexers) 908 invalid = [k for k in indexers if k not in self.dims] 909 if invalid: --> 910 raise ValueError("dimensions %r do not exist" % invalid) 911 912 # all indexers should be int, slice or np.ndarrays

ValueError: dimensions ['space2'] do not exist ```

Is there an easier way to do this? I couldn't think of anything...

CC @justinkuosixty