issues

At what level of support should Xarray acknowledge sponsors on our website?

I would like to surface this for open discussion because there are potential sponsoring organizations with conflicts of interest with members of Xarray's leadership team (e.g., Earthmover, which employs @jhamman, @rabernat and @dcherian).

My suggestion is to use NumPy's guidelines, with an adjustment down to 1/3 of the thresholds to account for the smaller size of the project:

• $10,000/yr for unrestricted financial contributions (e.g., donations)
• $20,000/yr for financial contributions for a particular purpose (e.g., grants)
• $30,000/yr for in-kind contributions (e.g., time for employees to contribute)
• 2 person-months/yr of paid work time for one or more Xarray maintainers or regular contributors to any Xarray team or activity

The NumPy guidelines also include a grace period of a minimum of 6 months for acknowledging support. I would suggest increasing this to a minimum of 1 year for Xarray.

I would greatly appreciate any feedback from members of the community, either in this issue or on the next team meeting.

This came up as part of my clean-up of serializing unicode strings in https://github.com/pydata/xarray/pull/1648.

There are two ways to represent strings in netCDF files.

• As character arrays (NC_CHAR), supported by both netCDF3 and netCDF4
• As variable length unicode strings (NC_STRING), only supported by netCDF4/HDF5.

Currently, by default (if no _FillValue is set) we replace missing values (NaN) with an empty string when writing data to disk.

For character arrays, we could use the normal _FillValue mechanism to set a fill value and decode when data is read back from disk. In fact, this already currently works for dtype=bytes (though it isn't documented): ``` In [10]: ds = xr.Dataset({'foo': ('x', np.array([b'bar', np.nan], dtype=object), {}, {'_FillValue': b''})})

In [11]: ds Out[11]: <xarray.Dataset> Dimensions: (x: 2) Dimensions without coordinates: x Data variables: foo (x) object b'bar' nan

In [12]: ds.to_netcdf('foobar.nc')

In [13]: xr.open_dataset('foobar.nc').load() Out[13]: <xarray.Dataset> Dimensions: (x: 2) Dimensions without coordinates: x Data variables: foo (x) object b'bar' nan ```

For variable length strings, it currently isn't possible to set a fill-value. So there's no good way to indicate missing values, though this may change if the future depending on the resolution of the netCDF-python issue.

It would obviously be nice to always automatically round-trip missing values, both for strings and bytes. I see two possible ways to do this: 1. Require setting an explicit _FillValue when a string contains missing values, by raising an error if this isn't done. We need an explicit choice because there aren't any extra unused characters left over, at least for character arrays. (NetCDF explicitly allows arbitrary bytes to be stored in NC_CHAR, even though this maps to an HDF5 fixed-width string with ASCII encoding.) For variable length strings, we could potentially set a non-character unicode symbol like U+FFFF, but again that isn't supported yet. 2. Treat empty strings as equivalent to a missing value (NaN). This has the advantage of not requiring an explicit choice of _FillValue, so we don't need to wait for any netCDF4 issues to be resolved. However, this does mean that empty strings would not round-trip. Still, given the relative prevalence of missing values vs empty strings in xarray/pandas, it's probably the lesser evil to not preserve empty string.

The default option is to adopt neither of these, and keep the current behavior where missing values are written as empty strings and not decoded at all.

Any opinions? I am leaning towards option (2).

This is a nice option for working with in-file HFD5/netCDF4 compression: https://github.com/pydata/xarray/pull/1128#issuecomment-261936849

Mixed multi-threading/multi-processing could also be interesting, if anyone wants to revive that: https://github.com/dask/dask/pull/457 (I think it would work now that xarray data stores are pickle-able)

CC @mrocklin

I'm not sure why, but when I run the integration tests with dask-distributed locally (on my MacBook pro), they fail: ``` $ pytest xarray/tests/test_distributed.py --maxfail 1 ================================================ test session starts ================================================= platform darwin -- Python 3.7.2, pytest-4.0.1, py-1.7.0, pluggy-0.8.0 rootdir: /Users/shoyer/dev/xarray, inifile: setup.cfg plugins: repeat-0.7.0 collected 19 items

xarray/tests/test_distributed.py F

====================================================== FAILURES ====================================================== ____ test_dask_distributed_netcdf_roundtrip[netcdf4-NETCDF3_CLASSIC] _______

loop = <tornado.platform.asyncio.AsyncIOLoop object at 0x1c182da1d0> tmp_netcdf_filename = '/private/var/folders/15/qdcz0wqj1t9dg40m_ld0fjkh00b4kd/T/pytest-of-shoyer/pytest-3/test_dask_distributed_netcdf_r0/testfile.nc' engine = 'netcdf4', nc_format = 'NETCDF3_CLASSIC'

@pytest.mark.parametrize('engine,nc_format', ENGINES_AND_FORMATS)  # noqa
def test_dask_distributed_netcdf_roundtrip(
        loop, tmp_netcdf_filename, engine, nc_format):

    if engine not in ENGINES:
        pytest.skip('engine not available')

    chunks = {'dim1': 4, 'dim2': 3, 'dim3': 6}

    with cluster() as (s, [a, b]):
        with Client(s['address'], loop=loop):

            original = create_test_data().chunk(chunks)

            if engine == 'scipy':
                with pytest.raises(NotImplementedError):
                    original.to_netcdf(tmp_netcdf_filename,
                                       engine=engine, format=nc_format)
                return

            original.to_netcdf(tmp_netcdf_filename,
                               engine=engine, format=nc_format)

            with xr.open_dataset(tmp_netcdf_filename,
                                 chunks=chunks, engine=engine) as restored:
                assert isinstance(restored.var1.data, da.Array)
                computed = restored.compute()

              assert_allclose(original, computed)

xarray/tests/test_distributed.py:87:

../../miniconda3/envs/xarray-py37/lib/python3.7/contextlib.py:119: in exit next(self.gen)

nworkers = 2, nanny = False, worker_kwargs = {}, active_rpc_timeout = 1, scheduler_kwargs = {}

@contextmanager
def cluster(nworkers=2, nanny=False, worker_kwargs={}, active_rpc_timeout=1,
            scheduler_kwargs={}):
    ...  # trimmed
    start = time()
    while list(ws):
        sleep(0.01)

      assert time() < start + 1, 'Workers still around after one second'

E AssertionError: Workers still around after one second

../../miniconda3/envs/xarray-py37/lib/python3.7/site-packages/distributed/utils_test.py:721: AssertionError ------------------------------------------------ Captured stderr call ------------------------------------------------ distributed.scheduler - INFO - Clear task state distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:51715 distributed.worker - INFO - Start worker at: tcp://127.0.0.1:51718 distributed.worker - INFO - Listening to: tcp://127.0.0.1:51718 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Memory: 17.18 GB distributed.worker - INFO - Local Directory: /Users/shoyer/dev/xarray/_test_worker-5cabd1b7-4d9c-49eb-a79e-205c588f5dae/worker-n8uv72yx distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Start worker at: tcp://127.0.0.1:51720 distributed.worker - INFO - Listening to: tcp://127.0.0.1:51720 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:51715 distributed.scheduler - INFO - Register tcp://127.0.0.1:51718 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Memory: 17.18 GB distributed.worker - INFO - Local Directory: /Users/shoyer/dev/xarray/_test_worker-71a426d4-bd34-4808-9d33-79cac2bb4801/worker-a70rlf4r distributed.worker - INFO - ------------------------------------------------- distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:51718 distributed.core - INFO - Starting established connection distributed.worker - INFO - Registered to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Register tcp://127.0.0.1:51720 distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:51720 distributed.core - INFO - Starting established connection distributed.worker - INFO - Registered to: tcp://127.0.0.1:51715 distributed.worker - INFO - ------------------------------------------------- distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Receive client connection: Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.core - INFO - Starting established connection distributed.core - INFO - Event loop was unresponsive in Worker for 1.05s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. distributed.scheduler - INFO - Receive client connection: Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.core - INFO - Starting established connection distributed.core - INFO - Event loop was unresponsive in Worker for 1.33s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. distributed.scheduler - INFO - Receive client connection: Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.core - INFO - Starting established connection distributed.scheduler - INFO - Remove client Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.scheduler - INFO - Remove client Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-59a7918c-5972-11e9-912a-8c85907bce57 distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:51720 distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:51718 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:51720 distributed.core - INFO - Removing comms to tcp://127.0.0.1:51720 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:51718 distributed.core - INFO - Removing comms to tcp://127.0.0.1:51718 distributed.scheduler - INFO - Lost all workers distributed.scheduler - INFO - Remove client Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.scheduler - INFO - Remove client Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-worker-5b2496d8-5972-11e9-9137-8c85907bce57 distributed.scheduler - INFO - Close client connection: Client-worker-5a5c81de-5972-11e9-9136-8c85907bce57 distributed.scheduler - INFO - Scheduler closing... distributed.scheduler - INFO - Scheduler closing all comms ```

Version info: ``` In [2]: xarray.show_versions()

INSTALLED VERSIONS

commit: 2ce0639ee2ba9c7b1503356965f77d847d6cfcdf python: 3.7.2 (default, Dec 29 2018, 00:00:04) [Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.2.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2

xarray: 0.12.1+4.g2ce0639e pandas: 0.24.0 numpy: 1.15.4 scipy: 1.1.0 netCDF4: 1.4.3.2 pydap: None h5netcdf: 0.7.0 h5py: 2.9.0 Nio: None zarr: 2.2.0 cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 dask: 1.1.5 distributed: 1.26.1 matplotlib: 3.0.2 cartopy: 0.17.0 seaborn: 0.9.0 setuptools: 40.0.0 pip: 18.0 conda: None pytest: 4.0.1 IPython: 6.5.0 sphinx: 1.8.2 ```

@mrocklin does this sort of error look familiar to you?

I noticed two minor issues in our online docs, now that we've switched to the hip new HTML repr by default.

1. Most doc pages still show text, not HTML. I suspect this is a limitation of the IPython sphinx derictive we use for our snippets. We might be able to fix that by switching to jupyter-sphinx?

2. The "attributes" part of the HTML repr in our notebook examples looks a little funny, with strange blue formatting around each attribute name. It looks like part of the outer style of our docs is leaking into the HTML repr:

As discussed in #947, we should remove the IndexVariable.name attribute. It should be fine to use an IndexVariable anywhere, regardless of whether or not it labels ticks along a dimension.

What happened?

With the latest released version of Xarray, it is possible to delete all variables corresponding to a MultiIndex by simply deleting the name of the MultiIndex.

After the explicit indexes refactor (i.e,. using the "main" development branch) this now raises error about how this would "corrupt" index state. This comes up when using drop() and assign_coords() and possibly some other methods.

This is not hard to work around, but we may want to consider this bug a blocker for the next Xarray release. I found the issue surfaced in several projects when attempting to use the new version of Xarray inside Google's codebase.

CC @benbovy in case you have any thoughts to share.

What did you expect to happen?

For now, we should preserve the behavior of deleting the variables corresponding to MultiIndex levels, but should issue a deprecation warning encouraging users to explicitly delete everything.

Minimal Complete Verifiable Example

```Python import xarray

array = xarray.DataArray( [[1, 2], [3, 4]], dims=['x', 'y'], coords={'x': ['a', 'b']}, ) stacked = array.stack(z=['x', 'y']) print(stacked.drop('z')) print() print(stacked.assign_coords(z=[1, 2, 3, 4])) ```

Relevant log output

```Python ValueError Traceback (most recent call last) Input In [1], in <cell line: 9>() 3 array = xarray.DataArray( 4 [[1, 2], [3, 4]], 5 dims=['x', 'y'], 6 coords={'x': ['a', 'b']}, 7 ) 8 stacked = array.stack(z=['x', 'y']) ----> 9 print(stacked.drop('z')) 10 print() 11 print(stacked.assign_coords(z=[1, 2, 3, 4]))

File ~/dev/xarray/xarray/core/dataarray.py:2425, in DataArray.drop(self, labels, dim, errors, labels_kwargs) 2408 def drop( 2409 self, 2410 labels: Mapping = None, (...) 2414 labels_kwargs, 2415 ) -> DataArray: 2416 """Backward compatible method based on drop_vars and drop_sel 2417 2418 Using either drop_vars or drop_sel is encouraged (...) 2423 DataArray.drop_sel 2424 """ -> 2425 ds = self._to_temp_dataset().drop(labels, dim, errors=errors) 2426 return self._from_temp_dataset(ds)

File ~/dev/xarray/xarray/core/dataset.py:4590, in Dataset.drop(self, labels, dim, errors, **labels_kwargs) 4584 if dim is None and (is_scalar(labels) or isinstance(labels, Iterable)): 4585 warnings.warn( 4586 "dropping variables using drop will be deprecated; using drop_vars is encouraged.", 4587 PendingDeprecationWarning, 4588 stacklevel=2, 4589 ) -> 4590 return self.drop_vars(labels, errors=errors) 4591 if dim is not None: 4592 warnings.warn( 4593 "dropping labels using list-like labels is deprecated; using " 4594 "dict-like arguments with drop_sel, e.g. `ds.drop_sel(dim=[labels]).", 4595 DeprecationWarning, 4596 stacklevel=2, 4597 )

File ~/dev/xarray/xarray/core/dataset.py:4549, in Dataset.drop_vars(self, names, errors) 4546 if errors == "raise": 4547 self._assert_all_in_dataset(names) -> 4549 assert_no_index_corrupted(self.xindexes, names) 4551 variables = {k: v for k, v in self._variables.items() if k not in names} 4552 coord_names = {k for k in self._coord_names if k in variables}

File ~/dev/xarray/xarray/core/indexes.py:1394, in assert_no_index_corrupted(indexes, coord_names) 1392 common_names_str = ", ".join(f"{k!r}" for k in common_names) 1393 index_names_str = ", ".join(f"{k!r}" for k in index_coords) -> 1394 raise ValueError( 1395 f"cannot remove coordinate(s) {common_names_str}, which would corrupt " 1396 f"the following index built from coordinates {index_names_str}:\n" 1397 f"{index}" 1398 )

ValueError: cannot remove coordinate(s) 'z', which would corrupt the following index built from coordinates 'z', 'x', 'y': <xarray.core.indexes.PandasMultiIndex object at 0x148c95150> ```

Anything else we need to know?

No response

Environment

Currently, Xarray's internal backends don't get registered unless the necessary dependencies are installed: https://github.com/pydata/xarray/blob/1305d9b624723b86050ca5b2d854e5326bbaa8e6/xarray/backends/netCDF4_.py#L567-L568

In order to facilitating suggesting a specific backend to install (e.g., to improve error messages from opening tutorial datasets https://github.com/pydata/xarray/issues/5291), I would suggest that Xarray always registers its own backend entrypoints. Then we make the following changes to the plugin protocol:

• guess_can_open() should work regardless of whether the underlying backend is installed
• installed() returns a boolean reporting whether backend is installed. The default method in the base class would return True, for backwards compatibility.
• open_dataset() of course should error if the backend is not installed.

This will let us leverage the existing guess_can_open() functionality to suggest specific optional dependencies to install. E.g., if you supply a netCDF3 file: Xarray cannot find a matching installed backend for this file in the installed backends ["h5netcdf"]. Consider installing one of the following backends which reports a match: ["scipy", "netcdf4"]

Does this reasonable and worthwhile?

CC @aurghs @alexamici

Code Sample, a copy-pastable example if possible

python input_ds = xarray.Dataset({'foo': ('x', [1, 2])}, {'x': [0, 1]}) input_ds.swap_dims({'x': 'z'})

Problem description

Currently this results in the error KeyError: 'z'

Expected Output

We now support dimensions without associated coordinate variables. So swap_dims() should be able to create new dimensions (e.g., z in this example) even if there isn't already a coordinate variable.

``` @network def test_rasterio_vrt_network(self): import rasterio

    url = 'https://storage.googleapis.com/\
    gcp-public-data-landsat/LC08/01/047/027/\
    LC08_L1TP_047027_20130421_20170310_01_T1/\
    LC08_L1TP_047027_20130421_20170310_01_T1_B4.TIF'
    env = rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',
                       CPL_VSIL_CURL_USE_HEAD=False,
                       CPL_VSIL_CURL_ALLOWED_EXTENSIONS='TIF')
    with env:

      with rasterio.open(url) as src:

xarray/tests/test_backends.py:3734:

/usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/env.py:430: in wrapper return f(args, kwds) /usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/init.py:216: in open s = DatasetReader(path, driver=driver, sharing=sharing, *kwargs)

??? E rasterio.errors.RasterioIOError: HTTP response code: 400 - Failed writing header ``` https://dev.azure.com/xarray/xarray/_build/results?buildId=150&view=ms.vss-test-web.build-test-results-tab&runId=2358&resultId=101228&paneView=debug

I'm not sure what's going on here -- the tiff file is still available at the given URL.

@scottyhq any idea?

For operations where arrays are aggregated but then combined, the keepdims=True option for NumPy aggregations is convenient.

We should consider supporting this in xarray as well. Aggregating a DataArray/Dataset with keepdims=True (or maybe keep_dims=True) would remove all original coordinates along aggregated dimensions and return a result with a dimension of size 1 without any coordinates, e.g., ```

array = xr.DataArray([1, 2, 3], dims='x', coords={'x': ['a', 'b', 'c']}) array.mean(keepdims=True) <xarray.DataArray (x: 1)> array([2.]) Dimensions without coordinates: x ```

In case, array.mean(keepdims=True() is equivalent to array.mean().expand_dims('x') but in general this equivalent does not hold, because the location of the original dimension is lost.

Implementation-wise, we have two options: 1. Pass on keepdims=True to NumPy functions like numpy.mean(), or 2. Implement keepdims=True ourselves, in Variable.reduce().

I think I like option 2 a little better, because it places fewer requirements on aggregation functions. For example, functions like bottleneck.nanmean() don't accept a keepdims argument.

Only on Python 3.5 and 3.6 for now: ``` =================================== FAILURES =================================== _ GenericNetCDFDataTest.testcross_engine_read_write_netcdf3 __ self = <xarray.tests.test_backends.GenericNetCDFDataTest testMethod=test_cross_engine_read_write_netcdf3> def test_cross_engine_read_write_netcdf3(self): data = create_test_data() valid_engines = set() if has_netCDF4: valid_engines.add('netcdf4') if has_scipy: valid_engines.add('scipy')

    for write_engine in valid_engines:
        for format in ['NETCDF3_CLASSIC', 'NETCDF3_64BIT']:
            with create_tmp_file() as tmp_file:
                data.to_netcdf(tmp_file, format=format,
                               engine=write_engine)
                for read_engine in valid_engines:
                    with open_dataset(tmp_file,

                                    engine=read_engine) as actual:

xarray/tests/test_backends.py:1596:

xarray/backends/api.py:299: in open_dataset autoclose=autoclose) xarray/backends/netCDF4_.py:280: in open ds = opener() xarray/backends/netCDF4_.py:204: in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, **kwargs) netCDF4/_netCDF4.pyx:2015: in netCDF4._netCDF4.Dataset.init ???

??? E OSError: [Errno -36] NetCDF: Invalid argument: b'/tmp/tmpu5no_wbf/temp-1157.nc' netCDF4/_netCDF4.pyx:1636: OSError ___ GenericNetCDFDataTestAutocloseTrue.test_cross_engine_read_write_netcdf3 ____ self = <xarray.tests.test_backends.GenericNetCDFDataTestAutocloseTrue testMethod=test_cross_engine_read_write_netcdf3> def test_cross_engine_read_write_netcdf3(self): data = create_test_data() valid_engines = set() if has_netCDF4: valid_engines.add('netcdf4') if has_scipy: valid_engines.add('scipy')

    for write_engine in valid_engines:
        for format in ['NETCDF3_CLASSIC', 'NETCDF3_64BIT']:
            with create_tmp_file() as tmp_file:
                data.to_netcdf(tmp_file, format=format,
                               engine=write_engine)
                for read_engine in valid_engines:
                    with open_dataset(tmp_file,

                                    engine=read_engine) as actual:

xarray/tests/test_backends.py:1596:

xarray/backends/api.py:299: in open_dataset autoclose=autoclose) xarray/backends/netCDF4_.py:280: in open ds = opener() xarray/backends/netCDF4_.py:204: in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, **kwargs) netCDF4/_netCDF4.pyx:2015: in netCDF4._netCDF4.Dataset.init ???

??? E OSError: [Errno -36] NetCDF: Invalid argument: b'/tmp/tmp9ak1v4wj/temp-1238.nc' netCDF4/_netCDF4.pyx:1636: OSError ```

Here's the diff of conda packages: diff --- broken.txt 2018-04-10 11:22:39.400835307 -0700 +++ works.txt 2018-04-10 11:23:12.840755416 -0700 @@ -9,2 +9,2 @@ -boto3 1.7.2 py_0 conda-forge -botocore 1.10.2 py_0 conda-forge +boto3 1.7.0 py_0 conda-forge +botocore 1.10.1 py_0 conda-forge @@ -23 +23 @@ -curl 7.59.0 1 conda-forge +curl 7.59.0 0 conda-forge @@ -29 +29 @@ -distributed 1.21.6 py36_0 conda-forge +distributed 1.21.5 py36_0 conda-forge @@ -62 +62 @@ -libgdal 2.2.4 1 conda-forge +libgdal 2.2.4 0 conda-forge @@ -66 +66 @@ -libnetcdf 4.5.0 3 conda-forge +libnetcdf 4.4.1.1 10 conda-forge @@ -83 +83 @@ -netcdf4 1.3.1 py36_2 conda-forge +netcdf4 1.3.1 py36_1 conda-forge @@ -85 +85 @@ -numcodecs 0.5.5 py36_0 conda-forge +numcodecs 0.5.4 py36_0 conda-forge @@ -131 +131 @@ -tornado 5.0.2 py36_0 conda-forge +tornado 5.0.1 py36_1 conda-forge

The culprit is almost certainly libnetcdf 4.4.1.1 -> 4.5.0

It looks like it's basically this issue again: https://github.com/Unidata/netcdf-c/issues/657

We could fix this either by skipping the tests in xarray's CI or upgrading netCDF-C on conda forge to 4.6.0 or 4.6.1.

Relevant discussion: #153

@pydata/xarray I'm going to issue v0.11.1 in a day or two, unless there's anything else we really want to squeeze in. This is the last release with planned Python 2.7 support (but we could conceivably still do backports for nasty bugs).

See https://travis-ci.org/pydata/xarray/jobs/410459646

Example failure: ``` =================================== FAILURES =================================== __ TestDataArray.testto_and_from_cdms2_classic ___ self = <xarray.tests.test_dataarray.TestDataArray testMethod=test_to_and_from_cdms2_classic> def test_to_and_from_cdms2_classic(self): """Classic with 1D axes""" pytest.importorskip('cdms2')

    original = DataArray(
        np.arange(6).reshape(2, 3),
        [('distance', [-2, 2], {'units': 'meters'}),
         ('time', pd.date_range('2000-01-01', periods=3))],
        name='foo', attrs={'baz': 123})
    expected_coords = [IndexVariable('distance', [-2, 2]),
                       IndexVariable('time', [0, 1, 2])]
    actual = original.to_cdms2()

  assert_array_equal(actual, original)

E ValueError: E error during assertion: E
E Traceback (most recent call last): E File "/home/travis/miniconda/envs/test_env/lib/python2.7/site-packages/numpy/testing/_private/utils.py", line 752, in assert_array_compare E x, y = x[~flagged], y[~flagged] E File "/home/travis/miniconda/envs/test_env/lib/python2.7/site-packages/cdms2/avariable.py", line 1177, in getitem E speclist = self._process_specs([key], {}) E File "/home/travis/miniconda/envs/test_env/lib/python2.7/site-packages/cdms2/avariable.py", line 938, in _process_specs E if Ellipsis in specs: E ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() E
E
E Arrays are not equal E x: TransientVariable([[0, 1, 2], E [3, 4, 5]]) E y: array([[0, 1, 2], E [3, 4, 5]]) ```

Signature: something like array = array.insert_dim(name, coord, axis=-1)

If index has size > 1, tile the array values along the new dimension, possibly using Variable.expand_dims to avoid copies.

In the spirit of our goal for a more rapid release (https://github.com/pydata/xarray/issues/1821), let's aim to issue the 0.10.2 release in the next few days, ideally after the following PRs are merged (all of which are nearly ready):

• [x] fix distributed writes #1793
• [x] einsum for xarray #1968
• [x] Support array_ufunc for xarray objects. #1962

CC @pydata/xarray

I plan to issue this in within the next few days, after merging #1260 (Rasterio support) and #1439 (pydap fix). Let me know if there's anything else critical to get in.

CC @pydata/xarray

This is somewhat time sensitive: https://github.com/kwgoodman/bottleneck/issues/168

Suppose I have two xarray Datasets, each defined along the 'x' axis: python ds1 = xarray.Dataset({'foo': (('x',), [1, 2, 3])}, {'x': [1, 2, 3], 'y': 'a'}) ds2 = xarray.Dataset({'foo': (('x',), [4, 5, 6])}, {'x': [1, 2, 3], 'y': 'b'}) <xarray.Dataset> Dimensions: (x: 3) Coordinates: y |S1 'a' * x (x) int64 1 2 3 Data variables: foo (x) int64 1 2 3

Now I want to stack them along a new MultiIndex 'yx' that consists of y and x levels: python desired = xarray.Dataset({'foo': (('yx',), [1, 2, 3, 4, 5, 6])}, {'yx': pandas.MultiIndex.from_product([['a', 'b'], [1, 2, 3]], names=['y', 'x'])}) <xarray.Dataset> Dimensions: (yx: 6) Coordinates: * yx (yx) MultiIndex - y (yx) object 'a' 'a' 'a' 'b' 'b' 'b' - x (yx) int64 1 2 3 1 2 3 Data variables: foo (yx) int64 1 2 3 4 5 6

How can this be achieved with the minimum effort? What is the missing utility function that we need? I attempted to use set_index and swap_dims but so far have been unsuccessful.

@benbovy any ideas? I think something similar may have come up when we were discussing your set_index PR.

It should be possible to safely merge together variables with values [NaN, 1, 2] and [0, 1, NaN] by using methods such as combine_first (which should also be OK with conflicting values, like the pandas method) and merge (which should raise if values conflict).

See this stackoverflow post for a merge example: http://stackoverflow.com/questions/36731870/how-to-merge-xarray-datasets-with-conflicting-coordinates

If True, squeezed dimensions should be dropped from the resulting object (instead of being retained as scalar)

This is cleaner than require users to separately download data, and it already works, for everything in the xarray-data repository!

ds = xr.tutorial.load_dataset('RASM_example_data')

We might want to rename the file to simply "rasm" to keep things shorter.

CC @rabernat @jhamman

Once we merge #972, I'd like to release v0.8.2.

It fixes several bugs likely to impact users and is almost completely backwards compatible (except for now automatically aligning in broadcast when we previously raised an error).

CC @jhamman in case he has time to try doing the release process sometime in the next few days.

Currently, when objects with with non-unique (duplicated) values in one of their indexes are passed to align, an error surfaces from pandas: InvalidIndexError: Reindexing only valid with uniquely valued Index objects

We could certainly give a more informative error here (see this complaint on StackOverflow), but a bigger issue is that this probably isn't strictly necessary. Instead, we could skip indexes for alignment if they are already equal. This is slightly less principled (a non-unique index may indicate something has gone wrong), but certainly more convenient and more inline with how pandas works (e.g., it even allows arithmetic between objects with non-unique indexes, which I believe does not work currently in xarray).

Currently, we do this as a special case when merging arrays and exactly one has labels (see _align_for_merge in https://github.com/pydata/xarray/pull/950). But we could probably do this in general, either by default or with a flag to enable it (or turn it off). This would then propagate to every xarray operation that uses align under the covers.

As reported on StackOverflow: http://stackoverflow.com/questions/38711915/segmentation-fault-writing-xarray-datset-to-netcdf-or-dataframe/

It appears that we can only open a single file at a time with pynio?

Adding a thread lock via lock=True didn't solve the issue.

cc @david-ian-brown

One option: add a batch_apply method:

This would be a shortcut for split-apply-combine with groupby/apply if the grouping over a dimension is only being done for efficiency reasons.

This function should take several parameters: - The dimension to group over. - The batchsize to group over on this dimension (defaulting to 1). - The func to apply to each group.

At first, this function would be useful just to avoid memory issues. Eventually, it would be nice to add a n_jobs parameter which would automatically dispatch to multiprocessing/joblib. We would need to get pickling (issue #24) working first to be able to do this.

For dask: - [x] default threadpool for dask.array - [x] fix indexing bugs for dask.array - [x] make a decision on (and if necessary implement) renaming "block" to "chunk" - [x] fix repeated use of da.insert

For xray: - [x] update xray for the updated dask (https://github.com/xray/xray/pull/395) - [x] figure out how to handle caching with the .load() method on dask arrays - [x] cleanup the xray documentation on dask array. - [x] write an introductory blog post

Things we can add in an incremental release: - make non-aggregating grouped operations more useable - automatic lazy apply for grouped operations on xray objects

CC @mrocklin

@jhamman writes in #337:

Since ds.groupby('time.season').mean('time') returns a Dataset with a Coordinates variable named time.season, ds.sel(time.season='JJA') doesn't work for Python syntax reasons. So, I don't know if I would change the syntax used in my example (selecting my position). I'm not keen on this constructor: ds.sel(**{'time.season':'JJA'}). I'm wondering if it would be better to name the coordinates returned from "virtual variable" operations without the "time." portion. Just a thought.

I agree, this is awkward. This has been on my to-do list in the back of my head for some time.

My hesitation with just using 'season' for the name of ds['time.season'] is that it would be a little weird to have indexing return something with a different name than what you asked for.

ds['season'] is another optional that initially looks even more appealing, but what if we have more than datetime variable? This is not unheard of (e.g., 'time' and 'forecast_reference_time').

Another option would be to simply support ds['time_season']. Then at least you can do indexing without using ** (e.g., ds.sel(time_season='JJA')).

e.g., we should be able to write ds.sel(time=('2000', '2010')) as an alias for ds.sel(time=slice('2000', '2010'))

I am currently working with station data. In order to keep around latitude and longitude (I use station_id as the coordinate variable), I need to resort to some ridiculous contortions:

python residuals = results['y'] - observations['y'] residuals.dataset.update(results.select_vars('longitude', 'latitude'))

There has got to be an easier way to handle this.

I don't want to revert to some primitive guessing strategy (e.g, looking at attrs['coordinates']) to figure out which extra variables can be safely kept after mathematical operations.

Another approach would be to try to preserve everything in the dataset linked to an DataArray when doing math. But I don't really like this option, either, because it would lead to serious propagation of "linked dataset variables", which are rather surprising and can have unexpected performance consequences (though at least they appear in repr as of #128).

This leaves me to a final alternative: restructuring xray's internals to provide first-class support for coordinates that are not indexes. For example, this would mean promoting ds.coordinates to an actual dictionary stored on a dataset, and allowing it to hold objects that aren't an xray.Coordinate.

Making this change transparent to users would likely require changing the Dataset signature to something like Dataset(variables, coords, attrs). We might (yet again) want to rename Coordinate, to something like IndexVar, to emphasis the notion of "index" and "non-index" coordinates. And we could get rid of the terrible "linked dataset variable".

Once we have non-index coordinates, we need a policy for what to do when adding with two DataArrays for which they differ. I think my preferred approach is to not enforce that they be found on both arrays, but to raise an exception if there are any conflicting values -- unless they are scalar valued, in which case the dropped or turned into a tuple or given different names. (Otherwise there would be cases where you couldn't calculate x[1] - x[0].)

We might even able to keep around multi-dimension coordinates this way (e.g., 2D lat/lon arrays for projected data).... I'll need to think about that one some more.

Right now, we use ds.attrs but the keyword argument is still attributes. We should resolve this inconsistency.

We also use dimensions and coordinates instead of the natural abbreviations dims and coords (although dims is used in the Variable constructor). Should we switch to the abbreviated versions for consistency with attrs?

Note that I switched to attrs in part because of its use in other packages (h5py, pytables and blz). There is not as clear precedent for what to call dimensions and coordinates.

This would make it less ambiguous that this is the preferred way to access and manipulate data in xray.

On a related note, I would like to make XArray more of an internal implementation detail that we only expose to advanced users.