This came up as part of my clean-up of serializing unicode strings in https://github.com/pydata/xarray/pull/1648.

There are two ways to represent strings in netCDF files.

• As character arrays (NC_CHAR), supported by both netCDF3 and netCDF4
• As variable length unicode strings (NC_STRING), only supported by netCDF4/HDF5.

Currently, by default (if no _FillValue is set) we replace missing values (NaN) with an empty string when writing data to disk.

For character arrays, we could use the normal _FillValue mechanism to set a fill value and decode when data is read back from disk. In fact, this already currently works for dtype=bytes (though it isn't documented): ``` In [10]: ds = xr.Dataset({'foo': ('x', np.array([b'bar', np.nan], dtype=object), {}, {'_FillValue': b''})})

In [11]: ds Out[11]: <xarray.Dataset> Dimensions: (x: 2) Dimensions without coordinates: x Data variables: foo (x) object b'bar' nan

In [12]: ds.to_netcdf('foobar.nc')

In [13]: xr.open_dataset('foobar.nc').load() Out[13]: <xarray.Dataset> Dimensions: (x: 2) Dimensions without coordinates: x Data variables: foo (x) object b'bar' nan ```

For variable length strings, it currently isn't possible to set a fill-value. So there's no good way to indicate missing values, though this may change if the future depending on the resolution of the netCDF-python issue.

It would obviously be nice to always automatically round-trip missing values, both for strings and bytes. I see two possible ways to do this: 1. Require setting an explicit _FillValue when a string contains missing values, by raising an error if this isn't done. We need an explicit choice because there aren't any extra unused characters left over, at least for character arrays. (NetCDF explicitly allows arbitrary bytes to be stored in NC_CHAR, even though this maps to an HDF5 fixed-width string with ASCII encoding.) For variable length strings, we could potentially set a non-character unicode symbol like U+FFFF, but again that isn't supported yet. 2. Treat empty strings as equivalent to a missing value (NaN). This has the advantage of not requiring an explicit choice of _FillValue, so we don't need to wait for any netCDF4 issues to be resolved. However, this does mean that empty strings would not round-trip. Still, given the relative prevalence of missing values vs empty strings in xarray/pandas, it's probably the lesser evil to not preserve empty string.

The default option is to adopt neither of these, and keep the current behavior where missing values are written as empty strings and not decoded at all.

Any opinions? I am leaning towards option (2).

This is a nice option for working with in-file HFD5/netCDF4 compression: https://github.com/pydata/xarray/pull/1128#issuecomment-261936849

Mixed multi-threading/multi-processing could also be interesting, if anyone wants to revive that: https://github.com/dask/dask/pull/457 (I think it would work now that xarray data stores are pickle-able)

CC @mrocklin

I'm not sure why, but when I run the integration tests with dask-distributed locally (on my MacBook pro), they fail: ``` $ pytest xarray/tests/test_distributed.py --maxfail 1 ================================================ test session starts ================================================= platform darwin -- Python 3.7.2, pytest-4.0.1, py-1.7.0, pluggy-0.8.0 rootdir: /Users/shoyer/dev/xarray, inifile: setup.cfg plugins: repeat-0.7.0 collected 19 items

xarray/tests/test_distributed.py F

====================================================== FAILURES ====================================================== ____ test_dask_distributed_netcdf_roundtrip[netcdf4-NETCDF3_CLASSIC] _______

loop = <tornado.platform.asyncio.AsyncIOLoop object at 0x1c182da1d0> tmp_netcdf_filename = '/private/var/folders/15/qdcz0wqj1t9dg40m_ld0fjkh00b4kd/T/pytest-of-shoyer/pytest-3/test_dask_distributed_netcdf_r0/testfile.nc' engine = 'netcdf4', nc_format = 'NETCDF3_CLASSIC'

@pytest.mark.parametrize('engine,nc_format', ENGINES_AND_FORMATS)  # noqa
def test_dask_distributed_netcdf_roundtrip(
        loop, tmp_netcdf_filename, engine, nc_format):

    if engine not in ENGINES:
        pytest.skip('engine not available')

    chunks = {'dim1': 4, 'dim2': 3, 'dim3': 6}

    with cluster() as (s, [a, b]):
        with Client(s['address'], loop=loop):

            original = create_test_data().chunk(chunks)

            if engine == 'scipy':
                with pytest.raises(NotImplementedError):
                    original.to_netcdf(tmp_netcdf_filename,
                                       engine=engine, format=nc_format)
                return

            original.to_netcdf(tmp_netcdf_filename,
                               engine=engine, format=nc_format)

            with xr.open_dataset(tmp_netcdf_filename,
                                 chunks=chunks, engine=engine) as restored:
                assert isinstance(restored.var1.data, da.Array)
                computed = restored.compute()

              assert_allclose(original, computed)

xarray/tests/test_distributed.py:87:

../../miniconda3/envs/xarray-py37/lib/python3.7/contextlib.py:119: in exit next(self.gen)

nworkers = 2, nanny = False, worker_kwargs = {}, active_rpc_timeout = 1, scheduler_kwargs = {}

@contextmanager
def cluster(nworkers=2, nanny=False, worker_kwargs={}, active_rpc_timeout=1,
            scheduler_kwargs={}):
    ...  # trimmed
    start = time()
    while list(ws):
        sleep(0.01)

      assert time() < start + 1, 'Workers still around after one second'

E AssertionError: Workers still around after one second

../../miniconda3/envs/xarray-py37/lib/python3.7/site-packages/distributed/utils_test.py:721: AssertionError ------------------------------------------------ Captured stderr call ------------------------------------------------ distributed.scheduler - INFO - Clear task state distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:51715 distributed.worker - INFO - Start worker at: tcp://127.0.0.1:51718 distributed.worker - INFO - Listening to: tcp://127.0.0.1:51718 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Memory: 17.18 GB distributed.worker - INFO - Local Directory: /Users/shoyer/dev/xarray/_test_worker-5cabd1b7-4d9c-49eb-a79e-205c588f5dae/worker-n8uv72yx distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Start worker at: tcp://127.0.0.1:51720 distributed.worker - INFO - Listening to: tcp://127.0.0.1:51720 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:51715 distributed.scheduler - INFO - Register tcp://127.0.0.1:51718 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Memory: 17.18 GB distributed.worker - INFO - Local Directory: /Users/shoyer/dev/xarray/_test_worker-71a426d4-bd34-4808-9d33-79cac2bb4801/worker-a70rlf4r distributed.worker - INFO - ------------------------------------------------- distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:51718 distributed.core - INFO - Starting established connection distributed.worker - INFO - Registered to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Register tcp://127.0.0.1:51720 distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:51720 distributed.core - INFO - Starting established connection distributed.worker - INFO - Registered to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Receive client connection: Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.core - INFO - Starting established connection distributed.core - INFO - Event loop was unresponsive in Worker for 1.05s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. distributed.scheduler - INFO - Receive client connection: Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.core - INFO - Starting established connection distributed.core - INFO - Event loop was unresponsive in Worker for 1.33s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. distributed.scheduler - INFO - Receive client connection: Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Remove client Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.scheduler - INFO - Remove client Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:51720 distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:51718 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:51720 distributed.core - INFO - Removing comms to tcp://127.0.0.1:51720 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:51718 distributed.core - INFO - Removing comms to tcp://127.0.0.1:51718 distributed.scheduler - INFO - Lost all workers distributed.scheduler - INFO - Remove client Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.scheduler - INFO - Remove client Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.scheduler - INFO - Scheduler closing... distributed.scheduler - INFO - Scheduler closing all comms ```

Version info: ``` In [2]: xarray.show_versions()

INSTALLED VERSIONS

commit: 2ce0639ee2ba9c7b1503356965f77d847d6cfcdf python: 3.7.2 (default, Dec 29 2018, 00:00:04) [Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.2.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2

xarray: 0.12.1+4.g2ce0639e pandas: 0.24.0 numpy: 1.15.4 scipy: 1.1.0 netCDF4: 1.4.3.2 pydap: None h5netcdf: 0.7.0 h5py: 2.9.0 Nio: None zarr: 2.2.0 cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 dask: 1.1.5 distributed: 1.26.1 matplotlib: 3.0.2 cartopy: 0.17.0 seaborn: 0.9.0 setuptools: 40.0.0 pip: 18.0 conda: None pytest: 4.0.1 IPython: 6.5.0 sphinx: 1.8.2 ```

@mrocklin does this sort of error look familiar to you?

Intended to fix https://github.com/pydata/xarray/issues/4449

The goal is to restore behavior to match what we had prior to https://github.com/pydata/xarray/pull/4379 for all types of data other than np.ndarray objects

Needs tests!

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

The old calendar was on @jhamman's UCAR account, which he no longer has access to!

xref https://github.com/pydata/xarray/issues/4001

as reported by @crusaderky in #1522

Several users (@raj-kesavan, @richardotis, now myself) have wondered about how to concatenate xray Datasets with different variables.

With the current xray.concat, you need to awkwardly create dummy variables filled with NaN in datasets that don't have them (or drop mismatched variables entirely). Neither of these are great options -- concat should have an option (the default?) to take care of this for the user.

This would also be more consistent with pd.concat, which takes a more relaxed approach to matching dataframes with different variables (it does an outer join).

This means that explicitly opening a file multiple times with open_dataset (e.g., after modifying it on disk) now reopens the file from scratch, rather than reusing a cached version.

If users want to reuse the cached file, they can reuse the same xarray object. We don't need this for handling many files in Dask (the original motivation for caching), because in those cases only a single CachingFileManager is created.

I think this should some long-standing usability issues: #4240, #4862

Conveniently, this also obviates the need for some messy reference counting logic.

• [x] Closes #4240, #4862
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

``` filslp = "ChonghuaYinData/prmsl.mon.mean.nc" filtmp = "ChonghuaYinData/air.sig995.mon.mean.nc" filprc = "ChonghuaYinData/precip.mon.mean.nc"

ds_slp = xr.open_dataset(filslp).sel(time=slice(str(yrStrt)+'-01-01', str(yrLast)+'-12-31'))

ds_slp outputs: <xarray.Dataset> Dimensions: (nbnds: 2, time: 480, x: 349, y: 277) Coordinates: * time (time) datetime64[ns] 1979-01-01 ... 2018-12-01 lat (y, x) float32 ... lon (y, x) float32 ... * y (y) float32 0.0 32463.0 64926.0 ... 8927325.0 8959788.0 * x (x) float32 0.0 32463.0 64926.0 ... 11264660.0 11297120.0 Dimensions without coordinates: nbnds Data variables: Lambert_Conformal int32 ... prmsl (time, y, x) float32 ... time_bnds (time, nbnds) float64 ... Attributes: Conventions: CF-1.2 centerlat: 50.0 centerlon: -107.0 comments:
institution: National Centers for Environmental Prediction latcorners: [ 1.000001 0.897945 46.3544 46.63433 ] loncorners: [-145.5 -68.32005 -2.569891 148.6418 ] platform: Model standardpar1: 50.0 standardpar2: 50.000001 title: NARR Monthly Means dataset_title: NCEP North American Regional Reanalysis (NARR) history: created 2016/04/12 by NOAA/ESRL/PSD references: https://www.esrl.noaa.gov/psd/data/gridded/data.narr.html source: http://www.emc.ncep.noaa.gov/mmb/rreanl/index.html References:
```

``` yrStrt = 1950 # manually specify for convenience yrLast = 2018 # 20th century ends 2018

clStrt = 1950 # reference climatology for SOI clLast = 1979

yrStrtP = 1979 # 1st year GPCP yrLastP = yrLast # match 20th century

latT = -17.6 # Tahiti lonT = 210.75
latD = -12.5 # Darwin lonD = 130.83

select grids of T and D

T = ds_slp.sel(lat=latT, lon=lonT, method='nearest') D = ds_slp.sel(lat=latD, lon=lonD, method='nearest') outputs:

ValueError Traceback (most recent call last) <ipython-input-27-6702b30f473f> in <module> 1 # select grids of T and D ----> 2 T = ds_slp.sel(lat=latT, lon=lonT, method='nearest') 3 D = ds_slp.sel(lat=latD, lon=lonD, method='nearest')

~\Anaconda3\lib\site-packages\xarray\core\dataset.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs) 2004 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel") 2005 pos_indexers, new_indexes = remap_label_indexers( -> 2006 self, indexers=indexers, method=method, tolerance=tolerance 2007 ) 2008 result = self.isel(indexers=pos_indexers, drop=drop)

~\Anaconda3\lib\site-packages\xarray\core\coordinates.py in remap_label_indexers(obj, indexers, method, tolerance, **indexers_kwargs) 378 379 pos_indexers, new_indexes = indexing.remap_label_indexers( --> 380 obj, v_indexers, method=method, tolerance=tolerance 381 ) 382 # attach indexer's coordinate to pos_indexers

~\Anaconda3\lib\site-packages\xarray\core\indexing.py in remap_label_indexers(data_obj, indexers, method, tolerance) 257 new_indexes = {} 258 --> 259 dim_indexers = get_dim_indexers(data_obj, indexers) 260 for dim, label in dim_indexers.items(): 261 try:

~\Anaconda3\lib\site-packages\xarray\core\indexing.py in get_dim_indexers(data_obj, indexers) 223 ] 224 if invalid: --> 225 raise ValueError("dimensions or multi-index levels %r do not exist" % invalid) 226 227 level_indexers = defaultdict(dict)

ValueError: dimensions or multi-index levels ['lat', 'lon'] do not exist ```

Does any know how fix to this problem?Thank you very much.

Originally posted by @JimmyGao0204 in https://github.com/pydata/xarray/issues/475#issuecomment-633172787

What happened?

In testing an upgrade to the latest version of xarray in our systems, I noticed this test failing: ``` def test_weighted_operations_nonequal_coords(): # There are no weights for a == 4, so that data point is ignored. weights = DataArray(np.random.randn(4), dims=("a",), coords=dict(a=[0, 1, 2, 3])) data = DataArray(np.random.randn(4), dims=("a",), coords=dict(a=[1, 2, 3, 4])) check_weighted_operations(data, weights, dim="a", skipna=None)

    q = 0.5
    result = data.weighted(weights).quantile(q, dim="a")
    # Expected value computed using code from [https://aakinshin.net/posts/weighted-quantiles/](https://www.google.com/url?q=https://aakinshin.net/posts/weighted-quantiles/&sa=D) with values at a=1,2,3
    expected = DataArray([0.9308707], coords={"quantile": [q]}).squeeze()

  assert_allclose(result, expected)

E AssertionError: Left and right DataArray objects are not close E
E Differing values: E L E array(0.919569) E R E array(0.930871) ```

It appears that this test is hard-coded to match a particular random number seed, which in turn would fix the resutls of np.random.randn().

What did you expect to happen?

Whenever possible, Xarray's own tests should avoid relying on particular random number generators, e.g., in this case we could specify random numbers instead.

A back-up option would be to explicitly set random seed locally inside the tests, e.g., by creating a np.random.RandomState() with a fixed seed and using that. The global random state used by np.random.randn() is sensitive to implementation details like the order in which tests are run.

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

...

What happened?

With the latest released version of Xarray, it is possible to delete all variables corresponding to a MultiIndex by simply deleting the name of the MultiIndex.

After the explicit indexes refactor (i.e,. using the "main" development branch) this now raises error about how this would "corrupt" index state. This comes up when using drop() and assign_coords() and possibly some other methods.

This is not hard to work around, but we may want to consider this bug a blocker for the next Xarray release. I found the issue surfaced in several projects when attempting to use the new version of Xarray inside Google's codebase.

CC @benbovy in case you have any thoughts to share.

What did you expect to happen?

For now, we should preserve the behavior of deleting the variables corresponding to MultiIndex levels, but should issue a deprecation warning encouraging users to explicitly delete everything.

Minimal Complete Verifiable Example

```Python import xarray

array = xarray.DataArray( [[1, 2], [3, 4]], dims=['x', 'y'], coords={'x': ['a', 'b']}, ) stacked = array.stack(z=['x', 'y']) print(stacked.drop('z')) print() print(stacked.assign_coords(z=[1, 2, 3, 4])) ```

Relevant log output

```Python ValueError Traceback (most recent call last) Input In [1], in <cell line: 9>() 3 array = xarray.DataArray( 4 [[1, 2], [3, 4]], 5 dims=['x', 'y'], 6 coords={'x': ['a', 'b']}, 7 ) 8 stacked = array.stack(z=['x', 'y']) ----> 9 print(stacked.drop('z')) 10 print() 11 print(stacked.assign_coords(z=[1, 2, 3, 4]))

File ~/dev/xarray/xarray/core/dataarray.py:2425, in DataArray.drop(self, labels, dim, errors, labels_kwargs) 2408 def drop( 2409 self, 2410 labels: Mapping = None, (...) 2414 labels_kwargs, 2415 ) -> DataArray: 2416 """Backward compatible method based on drop_vars and drop_sel 2417 2418 Using either drop_vars or drop_sel is encouraged (...) 2423 DataArray.drop_sel 2424 """ -> 2425 ds = self._to_temp_dataset().drop(labels, dim, errors=errors) 2426 return self._from_temp_dataset(ds)

File ~/dev/xarray/xarray/core/dataset.py:4590, in Dataset.drop(self, labels, dim, errors, **labels_kwargs) 4584 if dim is None and (is_scalar(labels) or isinstance(labels, Iterable)): 4585 warnings.warn( 4586 "dropping variables using drop will be deprecated; using drop_vars is encouraged.", 4587 PendingDeprecationWarning, 4588 stacklevel=2, 4589 ) -> 4590 return self.drop_vars(labels, errors=errors) 4591 if dim is not None: 4592 warnings.warn( 4593 "dropping labels using list-like labels is deprecated; using " 4594 "dict-like arguments with drop_sel, e.g. `ds.drop_sel(dim=[labels]).", 4595 DeprecationWarning, 4596 stacklevel=2, 4597 )

File ~/dev/xarray/xarray/core/dataset.py:4549, in Dataset.drop_vars(self, names, errors) 4546 if errors == "raise": 4547 self._assert_all_in_dataset(names) -> 4549 assert_no_index_corrupted(self.xindexes, names) 4551 variables = {k: v for k, v in self._variables.items() if k not in names} 4552 coord_names = {k for k in self._coord_names if k in variables}

File ~/dev/xarray/xarray/core/indexes.py:1394, in assert_no_index_corrupted(indexes, coord_names) 1392 common_names_str = ", ".join(f"{k!r}" for k in common_names) 1393 index_names_str = ", ".join(f"{k!r}" for k in index_coords) -> 1394 raise ValueError( 1395 f"cannot remove coordinate(s) {common_names_str}, which would corrupt " 1396 f"the following index built from coordinates {index_names_str}:\n" 1397 f"{index}" 1398 )

ValueError: cannot remove coordinate(s) 'z', which would corrupt the following index built from coordinates 'z', 'x', 'y': <xarray.core.indexes.PandasMultiIndex object at 0x148c95150> ```

Anything else we need to know?

No response

Environment

Is your feature request related to a problem? Please describe.

Xarray's groupby aggregations (e.g., groupby(..).sum()) are very slow compared to pandas, as described in https://github.com/pydata/xarray/issues/659.

Describe the solution you'd like

We could speed things up considerably (easily 100x) by wrapping the numpy-groupies package.

Additional context

One challenge is how to handle dask arrays (and other duck arrays). In some cases it might make sense to apply the numpy-groupies function (using apply_ufunc), but in other cases it might be better to stick with the current indexing + concatenate solution. We could either pick some simple heuristics for choosing the algorithm to use on dask arrays, or could just stick with the current algorithm for now.

In particular, it might make sense to stick with the current algorithm if there are a many chunks in the arrays to aggregated along the "grouped" dimension (depending on the size of the unique group values).

Here's an example dataset of a large dataset from @alimanfoo: https://nbviewer.jupyter.org/gist/alimanfoo/b74b08465727894538d5b161b3ced764 <xarray.Dataset> Dimensions: (__variants/BaseCounts_dim1: 4, __variants/MLEAC_dim1: 3, __variants/MLEAF_dim1: 3, alt_alleles: 3, ploidy: 2, samples: 1142, variants: 21442865) Coordinates: samples/ID (samples) object dask.array<chunksize=(1142,), meta=np.ndarray> variants/CHROM (variants) object dask.array<chunksize=(21442865,), meta=np.ndarray> variants/POS (variants) int32 dask.array<chunksize=(4194304,), meta=np.ndarray> Dimensions without coordinates: __variants/BaseCounts_dim1, __variants/MLEAC_dim1, __variants/MLEAF_dim1, alt_alleles, ploidy, samples, variants Data variables: variants/ABHet (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/ABHom (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/AC (variants, alt_alleles) int32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> variants/AF (variants, alt_alleles) float32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> ...

I know similarly large datasets with lots of dimensions come up in other contexts as well, e.g., with geophysical model output.

That's a very long first line! This would be easier to read as: <xarray.Dataset> Dimensions: (__variants/BaseCounts_dim1: 4, __variants/MLEAC_dim1: 3, __variants/MLEAF_dim1: 3, alt_alleles: 3, ploidy: 2, samples: 1142, variants: 21442865) Coordinates: samples/ID (samples) object dask.array<chunksize=(1142,), meta=np.ndarray> variants/CHROM (variants) object dask.array<chunksize=(21442865,), meta=np.ndarray> variants/POS (variants) int32 dask.array<chunksize=(4194304,), meta=np.ndarray> Dimensions without coordinates: __variants/BaseCounts_dim1, __variants/MLEAC_dim1, __variants/MLEAF_dim1, alt_alleles, ploidy, samples, variants Data variables: variants/ABHet (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/ABHom (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/AC (variants, alt_alleles) int32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> variants/AF (variants, alt_alleles) float32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> ...

or maybe: <xarray.Dataset> Dimensions: __variants/BaseCounts_dim1: 4 __variants/MLEAC_dim1: 3 __variants/MLEAF_dim1: 3 alt_alleles: 3 ploidy: 2 samples: 1142 variants: 21442865 Coordinates: samples/ID (samples) object dask.array<chunksize=(1142,), meta=np.ndarray> variants/CHROM (variants) object dask.array<chunksize=(21442865,), meta=np.ndarray> variants/POS (variants) int32 dask.array<chunksize=(4194304,), meta=np.ndarray> Dimensions without coordinates: __variants/BaseCounts_dim1, __variants/MLEAC_dim1, __variants/MLEAF_dim1, alt_alleles, ploidy, samples, variants Data variables: variants/ABHet (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/ABHom (variants) float32 dask.array<chunksize=(4194304,), meta=np.ndarray> variants/AC (variants, alt_alleles) int32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> variants/AF (variants, alt_alleles) float32 dask.array<chunksize=(4194304, 3), meta=np.ndarray> ...

Dimensions without coordinates could probably use some wrapping, too.

We currently have two types of methods that take a function to apply to xarray objects: - pipe (on DataArray and Dataset): apply a function to this entire object (array.pipe(func) -> func(array)) - apply (on Dataset and GroupBy): apply a function to each labeled object in this object (e.g., ds.apply(func) -> ds({k: func(v) for k, v in ds.data_vars.items()})).

And one more method that we want to add but isn't finalized yet -- currently named apply_ufunc: - Apply a function that acts on unlabeled (i.e., numpy) arrays to each array in the object

I'd like to have three distinct names that makes it clear what these methods do and how they are different. This has come up a few times recently, e.g., https://github.com/pydata/xarray/issues/1130

One proposal: rename apply to map, and then use apply only for methods that act on unlabeled arrays. This would require a deprecation cycle, but eventually it would let us add .apply methods for handling raw arrays to both Dataset and DataArray. (We could use a separate apply method from apply_ufunc to convert dim arguments to axis and not do automatic broadcasting.)

Some utility functions that should be useful for https://github.com/pydata/xarray/issues/1603

Still very much a work in progress -- it would be great if someone has time to finish writing any of these in another PR!

As @Hoeze notes in https://github.com/pydata/xarray/issues/5179, calling stack() can be "incredibly slow and memory-demanding, since it creates a MultiIndex of every possible coordinate in the array."

This is true with how stack() works currently, but I'm not sure this is necessary. I suspect it's a vestigial design choice from copying pandas, back from before Xarray had optional indexes. One benefit is that it's convenient for making unstack() the inverse of stack(), but isn't always required.

Regardless of how we define the semantics for boolean indexing (https://github.com/pydata/xarray/issues/1887), it seems like it could be a good idea to allow stack to skip creating a MultiIndex for the new dimension, via a new keyword argument such as ds.stack(index=False). This would be equivalent to calling reset_index() after stack() but would be cheaper because the MultiIndex is never created in the first place.

This PR simplifies the fix from https://github.com/pydata/xarray/pull/5731, specifically for the benefit of xarray.corr. There is no need to use map_blocks instead of using where directly.

It is a basically an alternative version of https://github.com/pydata/xarray/pull/5284. It is potentially slightly less efficient to do this masking step when unnecessary, but I doubt this makes a noticeable performance difference in practice (and I doubt this optimization is useful insdie map_blocks, anyways).

One of the PTSA authors told me about the new URL by email.

Consolidated metadata was a new feature in Zarr v2.3, which was released over two year ago (March 22, 2019).

Since then, I have used consolidated=True every time I've written or opened a Zarr store. As far as I can tell, this is almost always a good idea: - With local storage, it usually doesn't really matter. You spend a bit of time writing the consolidated metadata and have one extra file on disk, but the overhead is typically negligible. - With Cloud object stores or network filesystems, it can matter quite a large amount. Without consolidated metadata, these systems can be unusably slow for opening datasets. Cloud storage is of course the main use-case for Zarr. If you're using a local disk, you might as well stick with single files such as netCDF.

I wonder if consolidated metadata is mature enough now that we could consider switching the default behavior in Xarray. From my perspective, this is a big "gotcha" for getting good performance with Zarr. More than one of my colleagues has been unimpressed with the performance of Zarr until they learned to set consolidated=True.

I would suggest doing this in way is almost entirely backwards compatible, with only a minor performance costs for reading non-consolidated datasets: - to_zarr() switches the default to consolidated=True. The consolidate_metadata() will thus happen by default. - open_zarr() switches the default to consolidated=None, which means "Try reading consolidated metadata, and fall-back to non-consolidated if that fails." This will be slightly slower for non-consolidated metadata due to the extra file-lookup, but given that opening with non-consolidated metadata already requires a moderately large number of file look-ups, I doubt anyone will notice the difference.

CC @rabernat

This is a best practice for inclusive projects.

See https://github.com/github/renaming for guidance.

Type checking may be a little worse if typing-extensions are not installed, but I don't think it's worth the trouble of adding another hard dependency just for one use for TypeGuard.

Note: sadly this doesn't work yet. Mypy (and pylance) don't like the type alias defined with try/except. Any ideas? In the worst case, we could revert the TypeGuard entirely, but that would be a shame...

• [x] Closes #5495
• [x] Passes pre-commit run --all-files

This works with xarray=0.17.0: python import xarray xarray.Dataset({'x': [1, 2, 3]}).to_netcdf('tmp') xarray.open_dataset('tmp')

On xarray 0.18.0, it fails: ```

ValueError Traceback (most recent call last) <ipython-input-1-20e128a730aa> in <module>() 2 3 xarray.Dataset({'x': [1, 2, 3]}).to_netcdf('tmp') ----> 4 xarray.open_dataset('tmp')

/usr/local/lib/python3.7/dist-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, args, *kwargs) 483 484 if engine is None: --> 485 engine = plugins.guess_engine(filename_or_obj) 486 487 backend = plugins.get_backend(engine)

/usr/local/lib/python3.7/dist-packages/xarray/backends/plugins.py in guess_engine(store_spec) 110 warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning) 111 --> 112 raise ValueError("cannot guess the engine, try passing one explicitly") 113 114

ValueError: cannot guess the engine, try passing one explicitly ```

I'm not entirely sure what changed. My guess is that we used to fall-back to trying to use SciPy, but don't do that anymore. A potential fix would be reading strings as filenames in xarray.backends.utils.read_magic_number.

Related: https://github.com/pydata/xarray/issues/5291

Currently, Xarray's internal backends don't get registered unless the necessary dependencies are installed: https://github.com/pydata/xarray/blob/1305d9b624723b86050ca5b2d854e5326bbaa8e6/xarray/backends/netCDF4_.py#L567-L568

In order to facilitating suggesting a specific backend to install (e.g., to improve error messages from opening tutorial datasets https://github.com/pydata/xarray/issues/5291), I would suggest that Xarray always registers its own backend entrypoints. Then we make the following changes to the plugin protocol:

• guess_can_open() should work regardless of whether the underlying backend is installed
• installed() returns a boolean reporting whether backend is installed. The default method in the base class would return True, for backwards compatibility.
• open_dataset() of course should error if the backend is not installed.

This will let us leverage the existing guess_can_open() functionality to suggest specific optional dependencies to install. E.g., if you supply a netCDF3 file: Xarray cannot find a matching installed backend for this file in the installed backends ["h5netcdf"]. Consider installing one of the following backends which reports a match: ["scipy", "netcdf4"]

Does this reasonable and worthwhile?

CC @aurghs @alexamici

mode="r+" only allows for modifying pre-existing array values in a Zarr store. This makes it a safer default mode when doing a limited region write. It also offers a nice performance bonus when using consolidated metadata, because the store to modify can be opened in "consolidated" mode -- rather than painfully slow non-consolidated mode.

This PR includes several related changes to to_zarr():

1. It adds support for the new mode="r+".
2. consolidated=True in to_zarr() now means "open in consolidated mode" if using using mode="r+", instead of "write in consolidated mode" (which would not make sense for r+).
3. It allows setting consolidated=True when using region, mostly for the sake of fast store opening with r+.
4. Validation in to_zarr() has been reorganized to always use the existing Zarr group, rather than re-opening zar stores from scratch, which could require additional network requests.
5. Incidentally, I've renamed the ZarrStore.ds attribute to ZarrStore.zarr_group, which is a much more descriptive name.

These changes gave me a ~5x boost in write performance in a large parallel job making use of to_zarr with region.

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

E.g., see https://github.com/pydata/xarray/issues/1151 where stalebot closed an issue even after another comment.

Is this something we need to reconfigure or just a bug?

cc @pydata/xarray

Also includes a better error message when loading a tutorial dataset but an underlying IO dependency is not found.

• [x] Fixes #5291
• [x] Tests added
• [x] Passes pre-commit run --all-files

The new version checks magic numbers in files on disk, not just already open file objects.

I've also added a bunch of unit-tests.

Fixes GH5295

• [x] Closes #5295
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This would be the complement of the DataArray constructor, converting an xray.DataArray into a 1D series, 2D DataFrame or 3D panel, whichever is appropriate.

to_pandas would also makes sense for Dataset, if it could convert 0d datasets to series, e.g., pd.Series({k: v.item() for k, v in ds.items()}) (there is currently no direct way to do this), and revert to to_dataframe for higher dimensional input. - [x] DataArray method - [ ] Dataset method

This files in cases where the current working directory does not support writes, e.g., as seen here ``` def test_88_character_filename_segmentation_fault(self): # should be fixed in netcdf4 v1.3.1 with mock.patch('netCDF4.version', '1.2.4'): with warnings.catch_warnings(): message = ('A segmentation fault may occur when the ' 'file path has exactly 88 characters') warnings.filterwarnings('error', message) with pytest.raises(Warning): # Need to construct 88 character filepath

              xr.Dataset().to_netcdf('a' * (88 - len(os.getcwd()) - 1))

tests/test_backends.py:1234:

core/dataset.py:1150: in to_netcdf compute=compute) backends/api.py:715: in to_netcdf autoclose=autoclose, lock=lock) backends/netCDF4_.py:332: in open ds = opener() backends/netCDF4_.py:231: in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, **kwargs) third_party/py/netCDF4/_netCDF4.pyx:2111: in netCDF4._netCDF4.Dataset.init ???

??? E IOError: [Errno 13] Permission denied ```

open_dataset has a long list of arguments: xarray.open_dataset(filename_or_obj, group=None, decode_cf=True, mask_and_scale=None, decode_times=True, autoclose=None, concat_characters=True, decode_coords=True, engine=None, chunks=None, lock=None, cache=None, drop_variables=None, backend_kwargs=None, use_cftime=None)

Similarly to the case for pandas (https://github.com/pandas-dev/pandas/issues/27544), it would be nice to make most of these arguments keyword-only, e.g., def open_dataset(filename_or_obj, group, *, ...). For consistency, this would also apply to open_dataarray, decode_cf, open_mfdataset, etc.

This would encourage writing readable code when calling open_dataset() and would allow us to use better organization when adding new arguments (e.g., decode_timedelta in https://github.com/pydata/xarray/pull/4071).

To make this change, we could make use of the deprecate_nonkeyword_arguments decorator from https://github.com/pandas-dev/pandas/pull/27573

Fixes the issue noticed in: https://github.com/pydata/xarray/pull/4175#issuecomment-649135372

Let's see if this passes CI...

• [x] Passes isort -rc . && black . && mypy . && flake8

Our current policy is that xarray supports "the minor version (X.Y) initially published no more than N months ago" where N is:

• Python: 42 months (NEP 29)
• numpy: 24 months (NEP 29)
• pandas: 12 months
• scipy: 12 months
• sparse, pint and other libraries that rely on NEP-18 for integration: very latest available versions only,
• all other libraries: 6 months

I think this policy is too aggressive, particularly for pandas, SciPy and other libraries. Some of these projects can go 6+ months between minor releases. For example, version 2.3 of zarr is currently more than 6 months old. So if zarr released 2.4 today and xarray issued a new release tomorrow, and then our policy would dictate that we could ask users to upgrade to the new version.

In https://github.com/pydata/xarray/pull/4178, I misinterpreted our policy as supporting "the most recent minor version (X.Y) initially published more than N months ago". This version makes a bit more sense to me: users only need to upgrade dependencies at least every N months to use the latest xarray release.

I understand that NEP-29 chose its language intentionally, so that distributors know ahead of time when they can drop support for a Python or NumPy version. But this seems like a (very) poor fit for projects without regular releases. At the very least we should adjust the specific time windows.

I'll see if I can gain some understanding of the motivation for this particular language over on the NumPy tracker...

82 warnings are currently issued in the process of running our test suite: https://gist.github.com/shoyer/db0b2c82efd76b254453216e957c4345

Some of can probably be safely ignored, but others are likely noticed by users, e.g., https://stackoverflow.com/questions/41130138/why-is-invalid-value-encountered-in-greater-warning-thrown-in-python-xarray-fo/41147570#41147570

It would be nice to clean up all of these, either by catching the appropriate upstream warning (if irrelevant) or changing our usage to avoid the warning. There may very well be a lurking FutureWarning in there somewhere that could cause issues when another library updates.

Probably the easiest way to get started here is to get the test suite running locally, and use py.test -W error to turn all warnings into errors.

merge() currently supports four options for merging attrs: combine_attrs : {"drop", "identical", "no_conflicts", "override"}, \ default: "drop" String indicating how to combine attrs of the objects being merged: - "drop": empty attrs on returned Dataset. - "identical": all attrs must be the same on every object. - "no_conflicts": attrs from all objects are combined, any that have the same name must also have the same value. - "override": skip comparing and copy attrs from the first dataset to the result.

It would be nice to have an option to combine attrs from all objects like "no_conflicts", but that drops attributes with conflicting values rather than raising an error. We might call this combine_attrs="drop_conflicts" or combine_attrs="matching".

This is similar to how xarray currently handles conflicting values for DataArray.name and would be more suitable to consider for the default behavior of merge and other functions/methods that merge coordinates (e.g., apply_ufunc, concat, where, binary arithmetic).

cc @keewis

This PR adds support for a region keyword argument to to_zarr(), to support parallel writes to different parts of arrays in a zarr stores, e.g., ds.to_zarr(..., region={'x': slice(1000, 2000)}) to write a dataset over the range 1000:2000 along the x dimension.

This is useful for creating large Zarr datasets without requiring dask. For example, the separate workers in a simulation job might each write a single non-overlapping chunk of a Zarr file. The standard way to handle such datasets today is to first write netCDF files in each process, and then consolidate them afterwards with dask (see #3096).

Creating empty Zarr stores

In order to do so, the Zarr file must be pre-existing with desired variables in the right shapes/chunks. It is desirable to be able to create such stores without actually writing data, because datasets that we want to write in parallel may be very large.

In the example below, I achieve this filling a Dataset with dask arrays, and passing compute=False to to_zarr(). This works, but it relies on an undocumented implementation detail of the compute argument. We should either:

1. Officially document that the compute argument only controls writing array values, not metadata (at least for zarr).
2. Add a new keyword argument or entire new method for creating an unfilled Zarr store, e.g., write_values=False.

I think (1) is maybe the cleanest option (no extra API endpoints).

Unchunked variables

One potential gotcha concerns coordinate arrays that are not chunked, e.g., consider parallel writing of a dataset divided along time with 2D latitude and longitude arrays that are fixed over all chunks. With the current PR, such coordinate arrays would get rewritten by each separate writer.

If a Zarr store does not have atomic writes, then conceivably this could result in corrupted data. The default DirectoryStore has atomic writes and cloud based object stores should also be atomic, so perhaps this doesn't matter in practice, but at the very least it's inefficient and could cause issues for large-scale jobs due to resource contention.

Options include:

1. Current behavior. Variables whose dimensions do not overlap with region are written by to_zarr(). This is likely the most intuitive behavior for writing from a single process at a time.
2. Exclude variables whose dimensions do not overlap with region from being written. This is likely the most convenient behavior for writing from multiple processes at once.
3. Like (2), but issue a warning if any such variables exist instead of silently dropping them.
4. Like (2), but raise an error instead of a warning. Require the user to explicitly drop them with .drop(). This is probably the safest behavior.

I think (4) would be my preferred option. Some users would undoubtedly find this annoying, but the power-users for whom we are adding this feature would likely appreciate it.

Usage example

```python import xarray import dask.array as da

ds = xarray.Dataset({'u': (('x',), da.arange(1000, chunks=100))})

create the new zarr store, but don't write data

path = 'my-data.zarr' ds.to_zarr(path, compute=False)

look at the unwritten data

ds_opened = xarray.open_zarr(path) print('Data before writing:', ds_opened.u.data[::100].compute())

Data before writing: [ 1 100 1 100 100 1 1 1 1 1]

write out each slice (could be in separate processes)

for start in range(0, 1000, 100): selection = {'x': slice(start, start + 100)} ds.isel(selection).to_zarr(path, region=selection)

print('Data after writing:', ds_opened.u.data[::100].compute())

Data after writing: [ 0 100 200 300 400 500 600 700 800 900]

```

• [x] Closes https://github.com/pydata/xarray/issues/3096
• [x] Integration test
• [x] Unit tests
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It would be useful to document the internal Variable class and the internal structure of Dataset and DataArray. This would be helpful for both new contributors and expert users, who might find Variable helpful as an advanced API.

I had some notes in an earlier version of the docs that could be adapted. Note, however, that the internal structure of DataArray changed in #648: http://xray.readthedocs.org/en/v0.2/tutorial.html#notes-on-xray-s-internals

This should result in a better user experience, e.g., specifically pointing out the attribute with an invalid value.

• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

There are already lots of keyword arguments, and users want even more! (#843)

Maybe we should use some sort of object to encapsulate desired options?

Proposal

Given that a major purpose for plotting an array is to look at data or attributes, I wonder if we should expand these sections by default? - I worry that clicking on icons to expand sections may not be easy to discover - This would also be consistent with the text repr, which shows these sections by default (the Dataset repr is already consistent by default between text and HTML already)

Context

Currently the HTML repr for DataArray/Variable looks like this:

To see array data, you have to click on the icon:

(thanks to @max-sixty for making this a little bit more manageably sized in https://github.com/pydata/xarray/pull/3905!)

There's also a really nice repr for nested dask arrays:

• [x] Closes #4425
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

Fixes #4283

The underlying issue is that calling .item() on a NumPy array with dtype=datetime64[ns] returns an integer, rather than an np.datetime64 scalar. This is somewhat baffling but works this way because .item() returns native Python types, but datetime.datetime doesn't support nanosecond precision.

pandas.Index.get_loc used to support these integers, but now is more strict. Hence we get errors.

We can fix this by using array[()] to convert 0d arrays into NumPy scalars instead of calling array.item().

I've added a crude regression test. There may well be a better way to test this but I haven't figured it out yet.

• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

This is due to setting multi-dimensional attributes being an error, as of the latest netCDF4-Python release: https://github.com/Unidata/netcdf4-python/blob/master/Changelog

E.g., as seen on Appveyor: https://ci.appveyor.com/project/shoyer/xray/builds/22834250/job/9q0ip6i3cchlbkw2 ``` ================================== FAILURES =================================== ___ TestValidateAttrs.test_validating_attrs _____ self = <xarray.tests.test_backends.TestValidateAttrs object at 0x00000096BE5FAFD0> def test_validating_attrs(self): def new_dataset(): return Dataset({'data': ('y', np.arange(10.0))}, {'y': np.arange(10)})

    def new_dataset_and_dataset_attrs():
        ds = new_dataset()
        return ds, ds.attrs

    def new_dataset_and_data_attrs():
        ds = new_dataset()
        return ds, ds.data.attrs

    def new_dataset_and_coord_attrs():
        ds = new_dataset()
        return ds, ds.coords['y'].attrs

    for new_dataset_and_attrs in [new_dataset_and_dataset_attrs,
                                  new_dataset_and_data_attrs,
                                  new_dataset_and_coord_attrs]:
        ds, attrs = new_dataset_and_attrs()

        attrs[123] = 'test'
        with raises_regex(TypeError, 'Invalid name for attr'):
            ds.to_netcdf('test.nc')

        ds, attrs = new_dataset_and_attrs()
        attrs[MiscObject()] = 'test'
        with raises_regex(TypeError, 'Invalid name for attr'):
            ds.to_netcdf('test.nc')

        ds, attrs = new_dataset_and_attrs()
        attrs[''] = 'test'
        with raises_regex(ValueError, 'Invalid name for attr'):
            ds.to_netcdf('test.nc')

        # This one should work
        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = 'test'
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = {'a': 5}
        with raises_regex(TypeError, 'Invalid value for attr'):
            ds.to_netcdf('test.nc')

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = MiscObject()
        with raises_regex(TypeError, 'Invalid value for attr'):
            ds.to_netcdf('test.nc')

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = 5
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = 3.14
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = [1, 2, 3, 4]
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = (1.9, 2.5)
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = np.arange(5)
        with create_tmp_file() as tmp_file:
            ds.to_netcdf(tmp_file)

        ds, attrs = new_dataset_and_attrs()
        attrs['test'] = np.arange(12).reshape(3, 4)
        with create_tmp_file() as tmp_file:

          ds.to_netcdf(tmp_file)

xarray\tests\test_backends.py:3450:

xarray\core\dataset.py:1323: in to_netcdf compute=compute) xarray\backends\api.py:767: in to_netcdf unlimited_dims=unlimited_dims) xarray\backends\api.py:810: in dump_to_store unlimited_dims=unlimited_dims) xarray\backends\common.py:262: in store self.set_attributes(attributes) xarray\backends\common.py:278: in set_attributes self.set_attribute(k, v) xarray\backends\netCDF4_.py:418: in set_attribute set_nc_attribute(self.ds, key, value) xarray\backends\netCDF4.py:294: in _set_nc_attribute obj.setncattr(key, value) netCDF4_netCDF4.pyx:2781: in netCDF4._netCDF4.Dataset.setncattr ???

??? E ValueError: multi-dimensional array attributes not supported netCDF4_netCDF4.pyx:1514: ValueError ```

As discussed in https://github.com/pydata/xarray/issues/2346, it would be nice to support explicitly setting the desired order of dimensions when calling Dataset.to_dataframe() or DataArray.to_dataframe().

There is nice precedent for this in the to_dask_dataframe method: http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_dask_dataframe.html

I imagine we could copy the exact same API for `to_dataframe.

@canol reports on our mailing that our setuptools 41.2 (released 21 August 2019) install requirement is making it hard to install recent versions of xarray at his company: https://groups.google.com/g/xarray/c/HS_xcZDEEtA/m/GGmW-3eMCAAJ

Hello, this is just a feedback about an issue we experienced which caused our internal tools stack to stay with xarray 0.15 version instead of a newer versions.

We are a company using xarray in our internal frameworks and at the beginning we didn't have any restrictions on xarray version in our requirements file, so that new installations of our framework were using the latest version of xarray. But a few months ago we started to hear complaints from users who were having problems with installing our framework and the installation was failing because of xarray's requirement to use at least setuptools 41.2 which is released on 21th of August last year. So it hasn't been a year since it got released which might be considered relatively new.

During the installation of our framework, pip was failing to update setuptools by saying that some other process is already using setuptools files so it cannot update setuptools. The people who are using our framework are not software developers so they didn't know how to solve this problem and it became so overwhelming for us maintainers that we set the xarray requirement to version >=0.15 <0.16. We also share our internal framework with customers of our company so we didn't want to bother the customers with any potential problems.

You can see some other people having having similar problem when trying to update setuptools here (although not related to xarray): https://stackoverflow.com/questions/49338652/pip-install-u-setuptools-fail-windows-10

It is not a big deal but I just wanted to give this as a feedback. I don't know how much xarray depends on setuptools' 41.2 version.

I was surprised to see this in our setup.cfg file, added by @crusaderky in #3628. The version requirement is not documented in our docs.

Given that setuptools may be challenging to upgrade, would it be possible to relax this version requirement?

See https://github.com/dask/dask/issues/new/choose for an approximate example of what this looks like.

Over in #1288 @nbren12 wrote:

Overall, I think the xarray community could really benefit from some kind of centralized contrib package which has a low barrier to entry for these kinds of functions.

Yes, I agree that we should explore this. There are a lot of interesting projects building on xarray now but not great ways to discover them.

Are there other open source projects with a good model we should copy here? - Scikit-Learn has a separate GitHub org/repositories for contrib projects: https://github.com/scikit-learn-contrib. - TensorFlow has a contrib module within the TensorFlow namespace: tensorflow.contrib

This gives us two different models to consider. The first "separate repository" model might be easier/flexible from a maintenance perspective. Any preferences/thoughts?

There's also some nice overlap with the Pangeo project.

Before:

pandas.MultiIndexSeries.time_to_xarray
======= ========= ==========
--             subset
------- --------------------
dtype     True     False
======= ========= ==========
  int    505±0ms   37.1±0ms
 float   485±0ms   38.3±0ms
======= ========= ==========

After:

pandas.MultiIndexSeries.time_to_xarray
======= ============ ==========
--               subset
------- -----------------------
dtype      True       False
======= ============ ==========
  int    10.7±0.4ms   22.6±1ms
 float   10.0±0.8ms   21.1±1ms
======= ============ ==========

~~There are still some cases where we have to fall back to the existing slow implementation, but hopefully they should now be relatively rare.~~ Edit: now we always use the new implementation

• [x] Closes #2459, closes #4186
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] Passes isort -rc . && black . && mypy . && flake8

• [x] Closes #4176
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Using <pre> messes up the display of nested HTML reprs, e.g., from dask. Now we only use the <pre> tag when displaying raw text reprs.

Before (Jupyter notebook):

After:

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8

See below for the new language, to clarify that documentation is only necessary for "user visible changes."

I added "including notable bug fixes" to indicate that minor bug fixes may not be worth noting (I was thinking of test-suite only fixes in this category) but perhaps that is too confusing.

cc @pydata/xarray for opinions!

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for user visible changes (including notable bug fixes) and api.rst for new API

https://docs.readthedocs.io/en/latest/guides/autobuild-docs-for-pull-requests.html

Don't merge this!

Instead, we'll use RTD's new doc builder instead. For an example, click on "docs/readthedocs.org:xray" below or look at GH4159

• [x] Closes https://github.com/pydata/xarray/issues/4146

We don't support creating a Panel with to_pandas() with any version of pandas at present, so this test was previous broken if pandas < 0.25 was installed.

I don't know what's going on, but it currently times out after 1 hour: https://dev.azure.com/xarray/xarray/_build/results?buildId=2767&view=logs&j=7e620c85-24a8-5ffa-8b1f-642bc9b1fc36&t=68484831-0a19-5145-bfe9-6309e5f7691d

Is it possible to login to Azure to debug this stuff?

Fixes https://github.com/pydata/xarray/issues/4030

It looks like this is triggered by the new cartopy version now being installed on RTD (version 0.17.0 -> 0.18.0).

Long term we should fix this, but for now it's better just to disable the warning.

Here's the message from RTD: `` Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/latest/lib/python3.8/site-packages/IPython/sphinxext/ipython_directive.py", line 586, in process_input raise RuntimeError('Non Expected warning in{}line {}'.format(filename, lineno)) RuntimeError: Non Expected warning in/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/plotting.rst` line 732 The full traceback has been saved in /tmp/sphinx-err-qav6jjmm.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!

---

Warning in /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/plotting.rst at block ending on line 732 Specify :okwarning: as an option in the ipython:: block to suppress this message

/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/xarray/plot/facetgrid.py:373: UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations. self.fig.tight_layout() <<<------------------------------------------------------------------------- ``` https://readthedocs.org/projects/xray/builds/10969146/

This is nice for consistency with to_netcdf. It should be useful for cases where users want to update values in existing Zarr datasets.

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

The goal here is to make something extensible that we can live with for quite some time, and to clean up the internals of xarray's backend interface.

Most of these are analogues of existing xarray classes with a cleaned up interface. I have not yet worried about backwards compatibility or tests -- I would appreciate feedback on the approach here.

Several parts of the logic exist for the sake of dask. I've included the word "dask" in comments to facilitate inspection by mrocklin.

CC @rabernat, @pwolfram, @jhamman, @mrocklin -- for review

CC @mcgibbon, @JoyMonteiro -- this is relevant to our discussion today about adding support for appending to netCDF files. Don't let this stop you from getting started on that with the existing interface, though.

The "data" toggles are pre-expanded and don't work.

See https://github.com/googlecolab/colabtools/issues/1145 for a full description.

Now that we have preliminary support for sparse arrays in xarray, one really cool feature we could explore is creating sparse arrays from MultiIndexed pandas DataFrames.

Right now, xarray's methods for creating objects from pandas always create dense arrays, but the size of these dense arrays can get big really quickly if the MultiIndex is sparsely populated, e.g., python import pandas as pd import numpy as np import xarray df = pd.DataFrame({ 'w': range(10), 'x': list('abcdefghij'), 'y': np.arange(0, 100, 10), 'z': np.ones(10), }).set_index(['w', 'x', 'y']) print(xarray.Dataset.from_dataframe(df)) This length 10 DataFrame turned into a dense array with 1000 elements (only 10 of which are not NaN): <xarray.Dataset> Dimensions: (w: 10, x: 10, y: 10) Coordinates: * w (w) int64 0 1 2 3 4 5 6 7 8 9 * x (x) object 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' * y (y) int64 0 10 20 30 40 50 60 70 80 90 Data variables: z (w, x, y) float64 1.0 nan nan nan nan nan ... nan nan nan nan 1.0

We can imagine xarray.Dataset.from_dataframe(df, sparse=True) would make the same Dataset, but with sparse array (with a NaN fill value) instead of dense arrays.

Once sparse arrays work pretty well, this could actually obviate most of the use cases for MultiIndex in arrays. Arguably the model is quite a bit cleaner.

Currently, variable attributes are checked for equality before allowing for a merge via a call to xarray_equal. It should be possible to merge datasets even if some of the variable metadata disagrees (conflicting attributes should be dropped). This is already the behavior for global attributes.

The right design of this feature should probably include some optional argument to Dataset.merge indicating how strict we want the merge to be. I can see at least three versions that could be useful: 1. Drop conflicting metadata silently. 2. Don't allow for conflicting values, but drop non-matching keys. 3. Require all keys and values to match.

We can argue about which of these should be the default option. My inclination is to be as flexible as possible by using 1 or 2 in most cases.

Should match the pandas function: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.idxmax.html

``` In [60]: da = xarray.DataArray([1, 2, 3], dims='x', coords={'y': ('x', [1, 1, 1])})

In [61]: da.groupby('y').apply(lambda x: x[:2])

IndexError Traceback (most recent call last) <ipython-input-61-4c28a4712c34> in <module>() ----> 1 da.groupby('y').apply(lambda x: x[:2])

~/dev/xarray/xarray/core/groupby.py in apply(self, func, shortcut, kwargs) 516 applied = (maybe_wrap_array(arr, func(arr, kwargs)) 517 for arr in grouped) --> 518 return self._combine(applied, shortcut=shortcut) 519 520 def _combine(self, applied, shortcut=False):

~/dev/xarray/xarray/core/groupby.py in _combine(self, applied, shortcut) 526 else: 527 combined = concat(applied, dim) --> 528 combined = _maybe_reorder(combined, dim, positions) 529 530 if isinstance(combined, type(self._obj)):

~/dev/xarray/xarray/core/groupby.py in _maybe_reorder(xarray_obj, dim, positions) 436 return xarray_obj 437 else: --> 438 return xarray_obj[{dim: order}] 439 440

~/dev/xarray/xarray/core/dataarray.py in getitem(self, key) 476 else: 477 # orthogonal array indexing --> 478 return self.isel(**self._item_key_to_dict(key)) 479 480 def setitem(self, key, value):

~/dev/xarray/xarray/core/dataarray.py in isel(self, drop, indexers) 710 DataArray.sel 711 """ --> 712 ds = self._to_temp_dataset().isel(drop=drop, indexers) 713 return self._from_temp_dataset(ds) 714

~/dev/xarray/xarray/core/dataset.py in isel(self, drop, indexers) 1172 for name, var in iteritems(self._variables): 1173 var_indexers = dict((k, v) for k, v in indexers if k in var.dims) -> 1174 new_var = var.isel(var_indexers) 1175 if not (drop and name in var_indexers): 1176 variables[name] = new_var

~/dev/xarray/xarray/core/variable.py in isel(self, **indexers) 596 if dim in indexers: 597 key[i] = indexers[dim] --> 598 return self[tuple(key)] 599 600 def squeeze(self, dim=None):

~/dev/xarray/xarray/core/variable.py in getitem(self, key) 426 dims = tuple(dim for k, dim in zip(key, self.dims) 427 if not isinstance(k, integer_types)) --> 428 values = self._indexable_data[key] 429 # orthogonal indexing should ensure the dimensionality is consistent 430 if hasattr(values, 'ndim'):

~/dev/xarray/xarray/core/indexing.py in getitem(self, key) 476 def getitem(self, key): 477 key = self._convert_key(key) --> 478 return self._ensure_ndarray(self.array[key]) 479 480 def setitem(self, key, value):

IndexError: index 2 is out of bounds for axis 1 with size 2 ```

This would be useful, for example, for grouped sampling: https://stackoverflow.com/questions/46498247/how-to-downsample-xarray-dataset-using-groupby

To fix this, we will need to update our heuristics that decide if a groupby operation is a "transform" type operation that should have the output reordered to the original order: https://github.com/pydata/xarray/blob/24643ecee2eab04d0f84c41715d753e829f448e6/xarray/core/groupby.py#L293-L299

It would be nice to have a generic .pad() method to xarray objects based on numpy.pad and dask.array.pad.

In particular,pad with mode='wrap' could solve several use-cases related to periodic boundary conditions: https://github.com/pydata/xarray/issues/1005 , https://github.com/pydata/xarray/issues/2007. For example, ds.pad(longitude=(0, 1), mode='wrap') to add an extra point with periodic boundary conditions along the longitude dimension.

It probably makes sense to linearly extrapolate coordinates along padded dimensions, as long as they are regularly spaced. This might use heuristics and/or a keyword argument.

I don't have a plans to work on this in the near term. It could be a good project of moderate complexity for a new contributor.

This works with isel and sel (see below). There's no particular reason why it can't work with reindex and interp, too, though we would need to implement our own vectorized version of linear interpolation (should not be too bad, it's mostly a matter of indexing twice and calculating weights from the difference in coordinate values).

Apparently this is quite important for vertical regridding in weather/climate science (cc @rabernat, @nbren12 ) ``` In [35]: import xarray as xr

In [36]: import numpy as np

In [37]: data = xr.DataArray(np.arange(12).reshape((3, 4)), [('x', np.arange(3)), ('y', np.arange(4))])

In [38]: ind = xr.DataArray([[0, 2], [1, 0], [1, 2]], dims=['x', 'z'], coords={'x': [0, 1, 2]})

In [39]: data Out[39]: <xarray.DataArray (x: 3, y: 4)> array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) Coordinates: * x (x) int64 0 1 2 * y (y) int64 0 1 2 3

In [40]: ind Out[40]: <xarray.DataArray (x: 3, z: 2)> array([[0, 2], [1, 0], [1, 2]]) Coordinates: * x (x) int64 0 1 2 Dimensions without coordinates: z

In [41]: data.isel(y=ind) Out[41]: <xarray.DataArray (x: 3, z: 2)> array([[ 0, 2], [ 5, 4], [ 9, 10]]) Coordinates: * x (x) int64 0 1 2 y (x, z) int64 0 2 1 0 1 2 Dimensions without coordinates: z

In [42]: data.sel(y=ind) Out[42]: <xarray.DataArray (x: 3, z: 2)> array([[ 0, 2], [ 5, 4], [ 9, 10]]) Coordinates: * x (x) int64 0 1 2 y (x, z) int64 0 2 1 0 1 2 Dimensions without coordinates: z

In [43]: data.interp(y=ind)

ValueError Traceback (most recent call last) <ipython-input-43-e6eb7e39fd31> in <module> ----> 1 data.interp(y=ind)

~/dev/xarray/xarray/core/dataarray.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 1303 kwargs=kwargs, 1304 assume_sorted=assume_sorted, -> 1305 coords_kwargs 1306 ) 1307 return self._from_temp_dataset(ds)

~/dev/xarray/xarray/core/dataset.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 2455 } 2456 variables[name] = missing.interp( -> 2457 var, var_indexers, method, kwargs 2458 ) 2459 elif all(d not in indexers for d in var.dims):

~/dev/xarray/xarray/core/missing.py in interp(var, indexes_coords, method, *kwargs) 533 else: 534 out_dims.add(d) --> 535 return result.transpose(tuple(out_dims)) 536 537

~/dev/xarray/xarray/core/variable.py in transpose(self, *dims) 1219 return self.copy(deep=False) 1220 -> 1221 data = as_indexable(self._data).transpose(axes) 1222 return type(self)(dims, data, self._attrs, self._encoding, fastpath=True) 1223

~/dev/xarray/xarray/core/indexing.py in transpose(self, order) 1218 1219 def transpose(self, order): -> 1220 return self.array.transpose(order) 1221 1222 def getitem(self, key):

ValueError: axes don't match array

In [44]: data.reindex(y=ind) /Users/shoyer/dev/xarray/xarray/core/dataarray.py:1240: FutureWarning: Indexer has dimensions ('x', 'z') that are different from that to be indexed along y. This will behave differently in the future. fill_value=fill_value,

ValueError Traceback (most recent call last) <ipython-input-44-1277ead996ae> in <module> ----> 1 data.reindex(y=ind)

~/dev/xarray/xarray/core/dataarray.py in reindex(self, indexers, method, tolerance, copy, fill_value, **indexers_kwargs) 1238 tolerance=tolerance, 1239 copy=copy, -> 1240 fill_value=fill_value, 1241 ) 1242 return self._from_temp_dataset(ds)

~/dev/xarray/xarray/core/dataset.py in reindex(self, indexers, method, tolerance, copy, fill_value, **indexers_kwargs) 2360 tolerance, 2361 copy=copy, -> 2362 fill_value=fill_value, 2363 ) 2364 coord_names = set(self._coord_names)

~/dev/xarray/xarray/core/alignment.py in reindex_variables(variables, sizes, indexes, indexers, method, tolerance, copy, fill_value) 398 ) 399 --> 400 target = new_indexes[dim] = utils.safe_cast_to_index(indexers[dim]) 401 402 if dim in indexes:

~/dev/xarray/xarray/core/utils.py in safe_cast_to_index(array) 104 index = array 105 elif hasattr(array, "to_index"): --> 106 index = array.to_index() 107 else: 108 kwargs = {}

~/dev/xarray/xarray/core/dataarray.py in to_index(self) 545 arrays. 546 """ --> 547 return self.variable.to_index() 548 549 @property

~/dev/xarray/xarray/core/variable.py in to_index(self) 445 def to_index(self): 446 """Convert this variable to a pandas.Index""" --> 447 return self.to_index_variable().to_index() 448 449 def to_dict(self, data=True):

~/dev/xarray/xarray/core/variable.py in to_index_variable(self) 438 """Return this variable as an xarray.IndexVariable""" 439 return IndexVariable( --> 440 self.dims, self._data, self._attrs, encoding=self._encoding, fastpath=True 441 ) 442

~/dev/xarray/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 1941 super().init(dims, data, attrs, encoding, fastpath) 1942 if self.ndim != 1: -> 1943 raise ValueError("%s objects must be 1-dimensional" % type(self).name) 1944 1945 # Unlike in Variable, always eagerly load values into memory

ValueError: IndexVariable objects must be 1-dimensional ```

https://travis-ci.org/pydata/xarray/builds/359093748

Example failure: ``` ___ ScipyFilePathTest.test_append_write ____ self = <xarray.tests.test_backends.ScipyFilePathTest testMethod=test_append_write> def test_append_write(self): # regression for GH1215 data = create_test_data()

  with self.roundtrip_append(data) as actual:

xarray/tests/test_backends.py:786:

../../../miniconda/envs/test_env/lib/python3.6/contextlib.py:81: in enter return next(self.gen) xarray/tests/test_backends.py:155: in roundtrip_append self.save(data[[key]], path, mode=mode, save_kwargs) xarray/tests/test_backends.py:162: in save kwargs) xarray/core/dataset.py:1131: in to_netcdf unlimited_dims=unlimited_dims) xarray/backends/api.py:657: in to_netcdf unlimited_dims=unlimited_dims) xarray/core/dataset.py:1068: in dump_to_store unlimited_dims=unlimited_dims) xarray/backends/common.py:363: in store unlimited_dims=unlimited_dims) xarray/backends/common.py:402: in set_variables self.writer.add(source, target) xarray/backends/common.py:265: in add target[...] = source xarray/backends/scipy_.py:61: in setitem data[key] = value

self = <scipy.io.netcdf.netcdf_variable object at 0x7fe3eb3ec6a0> index = Ellipsis, data = array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. ]) def setitem(self, index, data): if self.maskandscale: missing_value = ( self._get_missing_value() or getattr(data, 'fill_value', 999999)) self._attributes.setdefault('missing_value', missing_value) self._attributes.setdefault('_FillValue', missing_value) data = ((data - self._attributes.get('add_offset', 0.0)) / self._attributes.get('scale_factor', 1.0)) data = np.ma.asarray(data).filled(missing_value) if self._typecode not in 'fd' and data.dtype.kind == 'f': data = np.round(data)

    # Expand data for record vars?
    if self.isrec:
        if isinstance(index, tuple):
            rec_index = index[0]
        else:
            rec_index = index
        if isinstance(rec_index, slice):
            recs = (rec_index.start or 0) + len(data)
        else:
            recs = rec_index + 1
        if recs > len(self.data):
            shape = (recs,) + self._shape[1:]
            # Resize in-place does not always work since
            # the array might not be single-segment
            try:
                self.data.resize(shape)
            except ValueError:
                self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)

  self.data[index] = data

E ValueError: assignment destination is read-only ```

I'm not sure how this got here, but it was probably my fault at one point :)

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

We don't want to release docs in a broken state.

Ideally, we would simply fail the build when Sphinx encounters a warning (e.g., by adding the -W flag). I attempted to do this in https://github.com/pydata/xarray/pull/1984. However, there are two issues with this: 1. We currently issue a very long list of warnings as part of a sphinx-build (see below), most of these due to failed references or a formatting issue with docstrings for numpydoc. Fixing this will be non-trivial. 2. IPython's sphinx directive currently does not even issue warnings/errors, due a bug in recent versions of IPython. This has been fixed on master, but not in a released version yet. We should be able to fix this when pandas removes their versioned version of the IPython directive (https://github.com/pandas-dev/pandas/pull/19657).

Expand for warnings from sphinx:

Code Sample, a copy-pastable example if possible

python input_ds = xarray.Dataset({'foo': ('x', [1, 2])}, {'x': [0, 1]}) input_ds.swap_dims({'x': 'z'})

Problem description

Currently this results in the error KeyError: 'z'

Expected Output

We now support dimensions without associated coordinate variables. So swap_dims() should be able to create new dimensions (e.g., z in this example) even if there isn't already a coordinate variable.

Some of our tests for warnings currently fail when run using an old version of pytest. The problem appears to be that we rely on pytest.warns() accepting subclasses rather exact matches.

This was fixed upstream in pytest (https://github.com/pytest-dev/pytest/pull/2166), but we still should specify the more specific warning types in xarray.

``` =================================== FAILURES =================================== __ TestEncodeCFVariable.testmissing_fillvalue ____

self = <xarray.tests.test_conventions.TestEncodeCFVariable testMethod=test_missing_fillvalue>

def test_missing_fillvalue(self):
    v = Variable(['x'], np.array([np.nan, 1, 2, 3]))
    v.encoding = {'dtype': 'int16'}
    with pytest.warns(Warning, match='floating point data as an integer'):

      conventions.encode_cf_variable(v)

E Failed: DID NOT WARN

tests/test_conventions.py:89: Failed

_____ TestAlias.test _____

self = <xarray.tests.test_utils.TestAlias testMethod=test>

def test(self):
    def new_method():
        pass
    old_method = utils.alias(new_method, 'old_method')
    assert 'deprecated' in old_method.__doc__
    with pytest.warns(Warning, match='deprecated'):

      old_method()

E Failed: DID NOT WARN

tests/test_utils.py:28: Failed

___ TestIndexVariable.test_coordinate_alias ______

self = <xarray.tests.test_variable.TestIndexVariable testMethod=test_coordinate_alias>

def test_coordinate_alias(self):
    with pytest.warns(Warning, match='deprecated'):

      x = Coordinate('x', [1, 2, 3])

E Failed: DID NOT WARN

tests/test_variable.py:1752: Failed ____ TestAccessor.test_register ______

self = <xarray.tests.test_extensions.TestAccessor testMethod=test_register>

def test_register(self):

    @xr.register_dataset_accessor('demo')
    @xr.register_dataarray_accessor('demo')
    class DemoAccessor(object):
        """Demo accessor."""

        def __init__(self, xarray_obj):
            self._obj = xarray_obj

        @property
        def foo(self):
            return 'bar'

    ds = xr.Dataset()
    assert ds.demo.foo == 'bar'

    da = xr.DataArray(0)
    assert da.demo.foo == 'bar'

    # accessor is cached
    assert ds.demo is ds.demo

    # check descriptor
    assert ds.demo.__doc__ == "Demo accessor."
    assert xr.Dataset.demo.__doc__ == "Demo accessor."
    assert isinstance(ds.demo, DemoAccessor)
    assert xr.Dataset.demo is DemoAccessor

    # ensure we can remove it
    del xr.Dataset.demo
    assert not hasattr(xr.Dataset, 'demo')

    with pytest.warns(Warning, match='overriding a preexisting attribute'):
        @xr.register_dataarray_accessor('demo')

      class Foo(object):

E Failed: DID NOT WARN

tests/test_extensions.py:60: Failed

```

See https://dev.azure.com/xarray/d5e7a686-a114-4b8c-a2d8-4b5b11efd896/_build/results?buildId=1218&view=logs&jobId=41d90575-019f-5cfd-d78e-c2adebf9a30b for a log.

Appears to be due to https://github.com/pandas-dev/pandas/pull/29062, which adds a .attrs attribute to pandas objects. We copy this attribute in the DataArray constructor.

Follow-up on #3436, needed now that https://github.com/Unidata/cftime/pull/127 has been merged.

I really should have known better than to remove this check -- there was a whole comment I had written explaining why it was there! I guess this is a lesson in why it's always important to write regression tests.

• [x] Closes #3377
• [x] Tests added
• [x] Passes black . && mypy . && flake8

Part of the explicit indexes refactor (https://github.com/pydata/xarray/issues/1603)

No user facing changes.

• [x] Passes black . && mypy . && flake8

xref #2227

Before: indexing.BooleanIndexing.time_indexing 898±0ms

After indexing.BooleanIndexing.time_indexing 401±0ms

• [x] Passes black . && mypy . && flake8

When I read this, I thought it was referring to scattering data somehow :).

Clean-ups we should definitely do: - [x] remove deprecated options from xarray.concat (deprecated back in July 2015!): https://github.com/pydata/xarray/blob/79dc7dc461c7540cc0b84a98543c6f7796c05268/xarray/core/concat.py#L114-L144 (edit by @max-sixty ) - [x] argument order in DataArray.to_dataset (also from July 2015) https://github.com/pydata/xarray/blob/41fecd8658ba50ddda0a52e04c21cec5e53415ac/xarray/core/dataarray.py#L491 (edit by @max-sixty ) - [x] remove the warning back reindex with variables with different dimensions (from 2017). This could either be replaced by replacing dimensions like sel or by simply raising an error for now and leaving replacing dimensions for later: https://github.com/pydata/xarray/pull/1639): https://github.com/pydata/xarray/blob/79dc7dc461c7540cc0b84a98543c6f7796c05268/xarray/core/alignment.py#L389-L398 (edit by @max-sixty ) - [x] remove xarray.broadcast_array, deprecated back in 2016 in https://github.com/pydata/xarray/commit/52ee95f8ae6b9631ac381b5b889de47e41f2440e (edit by @max-sixty ) - [x] remove Variable.expand_dims (deprecated back in August 2017), whose implementation actually looks like it's already broken: https://github.com/pydata/xarray/blob/41fecd8658ba50ddda0a52e04c21cec5e53415ac/xarray/core/variable.py#L1232-L1237 (edit by @max-sixty ) - [x] stop supporting a list of colors in the cmap argument (dating back to at least v0.7.0): https://github.com/pydata/xarray/blob/d089df385e737f71067309ff7abae15994d581ec/xarray/plot/utils.py#L737-L745 (edit by @max-sixty ) - [x] push the removal of the compat and encoding arguments from Dataset/DataArray back to 0.14. These were only deprecated 7 months ago in https://github.com/pydata/xarray/pull/2703. (edit by @max-sixty )

Clean-ups to consider: - [x] switch the default reduction dimension of groupby and resample? (https://github.com/pydata/xarray/pull/2366) This has been giving a FutureWarning since v0.11.0, released back in November 2018. We could also potentially push this back to 0.14, but these warnings are a little annoying... - [x] deprecate auto_combine (#2616) only since 29 June 2019, so that should be pushed.

These would be shortcuts for isel/slice syntax: - ds.head(time=5) -> ds.isel(time=slice(5)): select the first five time values - ds.tail(time=5) -> ds.isel(time=slice(-5, None)): select the last five time values - ds.thin(time=5) -> ds.isel(time=slice(None, None, 5)): select every 5th time value

Fixes https://github.com/pydata/xarray/issues/3206

Example usage:

In [3]: import pandas as pd
   ...: import numpy as np
   ...: import xarray
   ...: df = pd.DataFrame({
   ...:     'w': range(10),
   ...:     'x': list('abcdefghij'),
   ...:     'y': np.arange(0, 100, 10),
   ...:     'z': np.ones(10),
   ...: }).set_index(['w', 'x', 'y'])
   ...:

In [4]: ds = xarray.Dataset.from_dataframe(df, sparse=True)

In [5]: ds.z.data
Out[5]: <COO: shape=(10, 10, 10), dtype=float64, nnz=10, fill_value=nan>

• [x] Closes #3206, Closes #2139
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

min and max are now working.

notnull was already fixed by one of my earlier PRs.

std/var/median are still broken, but only because sparse hasn't implemented the corresponding NumPy functions yet (nanstd, nanvar and nanmedian).

rank needs pure NumPy implementation (not via bottleneck) if we want it to work on sparse or dask arrays.

• [x] Tests added
• [x] Passes black . && mypy . && flake8

We shouldn't be checking the details of dask's repr.

• [x] Tests added
• [x] Passes black . && mypy . && flake8

Currently, to drop labels along an existing dimension, you need to write something like: ds.drop(['a', 'b'], dim='x).

It would be nice if keyword arguments were supported, e.g., ds.drop(x=['a', 'b']). This would make drop() more symmetric with sel().

I also did a bit of cleanup (e.g., renaming methods) in xarray.core.formatting.

Sparse arrays were previously not shown in the Dataset repr:

<xarray.Dataset>
Dimensions:  (x: 4)
Coordinates:
    y        (x) int64 ...
Dimensions without coordinates: x
Data variables:
    a        (x) float64 ..."""

Now they are:

<xarray.Dataset>
Dimensions:  (x: 4)
Coordinates:
    y        (x) int64 <COO: shape=(4,), nnz=3, fill_value=0>
Dimensions without coordinates: x
Data variables:
    a        (x) float64 <COO: shape=(4,), nnz=4, fill_value=0.0>"""

• [x] Tests added
• [x] Passes black . && mypy . && flake8

It has some questionable coercion logic that no longer seems to be necessary.

Not a user facing change.

• [x] Passes black . && mypy . && flake8

This version should be much more compatible out of the box with duck typing.

No user facing changes.

``` @network def test_rasterio_vrt_network(self): import rasterio

    url = 'https://storage.googleapis.com/\
    gcp-public-data-landsat/LC08/01/047/027/\
    LC08_L1TP_047027_20130421_20170310_01_T1/\
    LC08_L1TP_047027_20130421_20170310_01_T1_B4.TIF'
    env = rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',
                       CPL_VSIL_CURL_USE_HEAD=False,
                       CPL_VSIL_CURL_ALLOWED_EXTENSIONS='TIF')
    with env:

      with rasterio.open(url) as src:

xarray/tests/test_backends.py:3734:

/usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/env.py:430: in wrapper return f(args, kwds) /usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/init.py:216: in open s = DatasetReader(path, driver=driver, sharing=sharing, *kwargs)

??? E rasterio.errors.RasterioIOError: HTTP response code: 400 - Failed writing header ``` https://dev.azure.com/xarray/xarray/_build/results?buildId=150&view=ms.vss-test-web.build-test-results-tab&runId=2358&resultId=101228&paneView=debug

I'm not sure what's going on here -- the tiff file is still available at the given URL.

@scottyhq any idea?

This should get things building again.

xref https://github.com/pydata/xarray/issues/3154

The upstream issue that required the constraint seems to have been fixed: https://github.com/barronh/pseudonetcdf/issues/69

In theory, this should make codecov wait until reports from Azure are back (and require that all Azure checks pass) before posting its comment.

It would be nice to have documentation recipe showing how to combine datasets with different latitude/longitude arrays, as often occurs due to numerical precision issues. It's a little more complicated than just using xarray.open_mfdataset(), and comes up with some regularity.

NetCDF-C and HDF5 are not threads-safe, so it's likely that closing the file object associated with one group could invalidate other open groups from the same file.

Now, we cache a single object corresponding to the root group for each file, and access sub-groups on the fly as needed.

• [x] Closes https://github.com/pydata/xarray/issues/2954
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• Suppressed various warnings.
• PsuedoNetCDF was not getting tested. Now it is.
• Added test coverage for print_versions.py

Using pre-built wheels for NumPy and pandas, so hopefully things will install with reasonable speed.

Adapted from https://github.com/dask/dask/blob/2.0.0/continuous_integration/travis/install.sh

The location of the XML output was being changed by pytest-azurepipelines.

More progress towards https://github.com/pydata/xarray/issues/1603 without any user facing changes.

The only part that's left is explicit index handling in merge.

Azure seems to be working pretty well, e.g., we now use it for test coverage with codecov. I don't see any particular reason to keep Travis around at this point.

The old setup didn't seem to work on CI, even when we explicitly passed the relevant flags.

xref https://github.com/Unidata/MetPy/issues/1077

No user facing changes

See https://github.com/tonybaloney/pytest-azurepipelines

No user facing changes