Intended to fix https://github.com/pydata/xarray/issues/4449

The goal is to restore behavior to match what we had prior to https://github.com/pydata/xarray/pull/4379 for all types of data other than np.ndarray objects

Needs tests!

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes https://github.com/pydata/xarray/issues/2298
• [ ] Tests added
• [ ] Tests passed
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Still needs more tests and documentation.

The old calendar was on @jhamman's UCAR account, which he no longer has access to!

xref https://github.com/pydata/xarray/issues/4001

• [x] Closes #5329
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

This means that explicitly opening a file multiple times with open_dataset (e.g., after modifying it on disk) now reopens the file from scratch, rather than reusing a cached version.

If users want to reuse the cached file, they can reuse the same xarray object. We don't need this for handling many files in Dask (the original motivation for caching), because in those cases only a single CachingFileManager is created.

I think this should some long-standing usability issues: #4240, #4862

Conveniently, this also obviates the need for some messy reference counting logic.

• [x] Closes #4240, #4862
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Fixes #324

It definitely doesn't work properly yet, totally mixing up coordinates, data variables and multi-indexes (as shown by the failing tests).

A simple example:

``` In [4]: coords = {'a': ('x', [0, 0, 1, 1]), 'b': ('y', [0, 0, 1, 1])}

In [5]: square = xr.DataArray(np.arange(16).reshape(4, 4), coords=coords, dims=['x', 'y'])

In [6]: square Out[6]: <xarray.DataArray (x: 4, y: 4)> array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) Coordinates: b (y) int64 0 0 1 1 a (x) int64 0 0 1 1 * x (x) int64 0 1 2 3 * y (y) int64 0 1 2 3

In [7]: square.groupby(['a', 'b']).mean() Out[7]: <xarray.DataArray (a: 2, b: 2)> array([[ 2.5, 4.5], [ 10.5, 12.5]]) Coordinates: * a (a) int64 0 1 * b (b) int64 0 1

In [8]: square.groupby(['x', 'y']).mean() Out[8]: <xarray.DataArray (x: 4, y: 4)> array([[ 0., 1., 2., 3.], [ 4., 5., 6., 7.], [ 8., 9., 10., 11.], [ 12., 13., 14., 15.]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) int64 0 1 2 3 ```

More examples: https://gist.github.com/shoyer/5cfa4d5751e8a78a14af25f8442ad8d5

Some utility functions that should be useful for https://github.com/pydata/xarray/issues/1603

Still very much a work in progress -- it would be great if someone has time to finish writing any of these in another PR!

This PR simplifies the fix from https://github.com/pydata/xarray/pull/5731, specifically for the benefit of xarray.corr. There is no need to use map_blocks instead of using where directly.

It is a basically an alternative version of https://github.com/pydata/xarray/pull/5284. It is potentially slightly less efficient to do this masking step when unnecessary, but I doubt this makes a noticeable performance difference in practice (and I doubt this optimization is useful insdie map_blocks, anyways).

One of the PTSA authors told me about the new URL by email.

Type checking may be a little worse if typing-extensions are not installed, but I don't think it's worth the trouble of adding another hard dependency just for one use for TypeGuard.

Note: sadly this doesn't work yet. Mypy (and pylance) don't like the type alias defined with try/except. Any ideas? In the worst case, we could revert the TypeGuard entirely, but that would be a shame...

• [x] Closes #5495
• [x] Passes pre-commit run --all-files

mode="r+" only allows for modifying pre-existing array values in a Zarr store. This makes it a safer default mode when doing a limited region write. It also offers a nice performance bonus when using consolidated metadata, because the store to modify can be opened in "consolidated" mode -- rather than painfully slow non-consolidated mode.

This PR includes several related changes to to_zarr():

1. It adds support for the new mode="r+".
2. consolidated=True in to_zarr() now means "open in consolidated mode" if using using mode="r+", instead of "write in consolidated mode" (which would not make sense for r+).
3. It allows setting consolidated=True when using region, mostly for the sake of fast store opening with r+.
4. Validation in to_zarr() has been reorganized to always use the existing Zarr group, rather than re-opening zar stores from scratch, which could require additional network requests.
5. Incidentally, I've renamed the ZarrStore.ds attribute to ZarrStore.zarr_group, which is a much more descriptive name.

These changes gave me a ~5x boost in write performance in a large parallel job making use of to_zarr with region.

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Also includes a better error message when loading a tutorial dataset but an underlying IO dependency is not found.

• [x] Fixes #5291
• [x] Tests added
• [x] Passes pre-commit run --all-files

The new version checks magic numbers in files on disk, not just already open file objects.

I've also added a bunch of unit-tests.

Fixes GH5295

• [x] Closes #5295
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Fixes the issue noticed in: https://github.com/pydata/xarray/pull/4175#issuecomment-649135372

Let's see if this passes CI...

• [x] Passes isort -rc . && black . && mypy . && flake8

This PR adds support for a region keyword argument to to_zarr(), to support parallel writes to different parts of arrays in a zarr stores, e.g., ds.to_zarr(..., region={'x': slice(1000, 2000)}) to write a dataset over the range 1000:2000 along the x dimension.

This is useful for creating large Zarr datasets without requiring dask. For example, the separate workers in a simulation job might each write a single non-overlapping chunk of a Zarr file. The standard way to handle such datasets today is to first write netCDF files in each process, and then consolidate them afterwards with dask (see #3096).

Creating empty Zarr stores

In order to do so, the Zarr file must be pre-existing with desired variables in the right shapes/chunks. It is desirable to be able to create such stores without actually writing data, because datasets that we want to write in parallel may be very large.

In the example below, I achieve this filling a Dataset with dask arrays, and passing compute=False to to_zarr(). This works, but it relies on an undocumented implementation detail of the compute argument. We should either:

1. Officially document that the compute argument only controls writing array values, not metadata (at least for zarr).
2. Add a new keyword argument or entire new method for creating an unfilled Zarr store, e.g., write_values=False.

I think (1) is maybe the cleanest option (no extra API endpoints).

Unchunked variables

One potential gotcha concerns coordinate arrays that are not chunked, e.g., consider parallel writing of a dataset divided along time with 2D latitude and longitude arrays that are fixed over all chunks. With the current PR, such coordinate arrays would get rewritten by each separate writer.

If a Zarr store does not have atomic writes, then conceivably this could result in corrupted data. The default DirectoryStore has atomic writes and cloud based object stores should also be atomic, so perhaps this doesn't matter in practice, but at the very least it's inefficient and could cause issues for large-scale jobs due to resource contention.

Options include:

1. Current behavior. Variables whose dimensions do not overlap with region are written by to_zarr(). This is likely the most intuitive behavior for writing from a single process at a time.
2. Exclude variables whose dimensions do not overlap with region from being written. This is likely the most convenient behavior for writing from multiple processes at once.
3. Like (2), but issue a warning if any such variables exist instead of silently dropping them.
4. Like (2), but raise an error instead of a warning. Require the user to explicitly drop them with .drop(). This is probably the safest behavior.

I think (4) would be my preferred option. Some users would undoubtedly find this annoying, but the power-users for whom we are adding this feature would likely appreciate it.

Usage example

```python import xarray import dask.array as da

ds = xarray.Dataset({'u': (('x',), da.arange(1000, chunks=100))})

create the new zarr store, but don't write data

path = 'my-data.zarr' ds.to_zarr(path, compute=False)

look at the unwritten data

ds_opened = xarray.open_zarr(path) print('Data before writing:', ds_opened.u.data[::100].compute())

Data before writing: [ 1 100 1 100 100 1 1 1 1 1]

write out each slice (could be in separate processes)

for start in range(0, 1000, 100): selection = {'x': slice(start, start + 100)} ds.isel(selection).to_zarr(path, region=selection)

print('Data after writing:', ds_opened.u.data[::100].compute())

Data after writing: [ 0 100 200 300 400 500 600 700 800 900]

```

• [x] Closes https://github.com/pydata/xarray/issues/3096
• [x] Integration test
• [x] Unit tests
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This should result in a better user experience, e.g., specifically pointing out the attribute with an invalid value.

• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

• [x] Closes #4425
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

Fixes #4283

The underlying issue is that calling .item() on a NumPy array with dtype=datetime64[ns] returns an integer, rather than an np.datetime64 scalar. This is somewhat baffling but works this way because .item() returns native Python types, but datetime.datetime doesn't support nanosecond precision.

pandas.Index.get_loc used to support these integers, but now is more strict. Hence we get errors.

We can fix this by using array[()] to convert 0d arrays into NumPy scalars instead of calling array.item().

I've added a crude regression test. There may well be a better way to test this but I haven't figured it out yet.

• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

See https://github.com/dask/dask/issues/new/choose for an approximate example of what this looks like.

Before:

pandas.MultiIndexSeries.time_to_xarray
======= ========= ==========
--             subset
------- --------------------
dtype     True     False
======= ========= ==========
  int    505±0ms   37.1±0ms
 float   485±0ms   38.3±0ms
======= ========= ==========

After:

pandas.MultiIndexSeries.time_to_xarray
======= ============ ==========
--               subset
------- -----------------------
dtype      True       False
======= ============ ==========
  int    10.7±0.4ms   22.6±1ms
 float   10.0±0.8ms   21.1±1ms
======= ============ ==========

~~There are still some cases where we have to fall back to the existing slow implementation, but hopefully they should now be relatively rare.~~ Edit: now we always use the new implementation

• [x] Closes #2459, closes #4186
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] Passes isort -rc . && black . && mypy . && flake8

• [x] Closes #4176
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Using <pre> messes up the display of nested HTML reprs, e.g., from dask. Now we only use the <pre> tag when displaying raw text reprs.

Before (Jupyter notebook):

After:

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8

See below for the new language, to clarify that documentation is only necessary for "user visible changes."

I added "including notable bug fixes" to indicate that minor bug fixes may not be worth noting (I was thinking of test-suite only fixes in this category) but perhaps that is too confusing.

cc @pydata/xarray for opinions!

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for user visible changes (including notable bug fixes) and api.rst for new API

https://docs.readthedocs.io/en/latest/guides/autobuild-docs-for-pull-requests.html

Don't merge this!

Instead, we'll use RTD's new doc builder instead. For an example, click on "docs/readthedocs.org:xray" below or look at GH4159

• [x] Closes https://github.com/pydata/xarray/issues/4146

We don't support creating a Panel with to_pandas() with any version of pandas at present, so this test was previous broken if pandas < 0.25 was installed.

Fixes https://github.com/pydata/xarray/issues/4030

It looks like this is triggered by the new cartopy version now being installed on RTD (version 0.17.0 -> 0.18.0).

Long term we should fix this, but for now it's better just to disable the warning.

Here's the message from RTD: `` Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/latest/lib/python3.8/site-packages/IPython/sphinxext/ipython_directive.py", line 586, in process_input raise RuntimeError('Non Expected warning in{}line {}'.format(filename, lineno)) RuntimeError: Non Expected warning in/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/plotting.rst` line 732 The full traceback has been saved in /tmp/sphinx-err-qav6jjmm.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!

---

Warning in /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/plotting.rst at block ending on line 732 Specify :okwarning: as an option in the ipython:: block to suppress this message

/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/xarray/plot/facetgrid.py:373: UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations. self.fig.tight_layout() <<<------------------------------------------------------------------------- ``` https://readthedocs.org/projects/xray/builds/10969146/

This is nice for consistency with to_netcdf. It should be useful for cases where users want to update values in existing Zarr datasets.

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

The goal here is to make something extensible that we can live with for quite some time, and to clean up the internals of xarray's backend interface.

Most of these are analogues of existing xarray classes with a cleaned up interface. I have not yet worried about backwards compatibility or tests -- I would appreciate feedback on the approach here.

Several parts of the logic exist for the sake of dask. I've included the word "dask" in comments to facilitate inspection by mrocklin.

CC @rabernat, @pwolfram, @jhamman, @mrocklin -- for review

CC @mcgibbon, @JoyMonteiro -- this is relevant to our discussion today about adding support for appending to netCDF files. Don't let this stop you from getting started on that with the existing interface, though.

I'm not sure how this got here, but it was probably my fault at one point :)

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Follow-up on #3436, needed now that https://github.com/Unidata/cftime/pull/127 has been merged.

I really should have known better than to remove this check -- there was a whole comment I had written explaining why it was there! I guess this is a lesson in why it's always important to write regression tests.

• [x] Closes #3377
• [x] Tests added
• [x] Passes black . && mypy . && flake8

Part of the explicit indexes refactor (https://github.com/pydata/xarray/issues/1603)

No user facing changes.

• [x] Passes black . && mypy . && flake8

xref #2227

Before: indexing.BooleanIndexing.time_indexing 898±0ms

After indexing.BooleanIndexing.time_indexing 401±0ms

• [x] Passes black . && mypy . && flake8

When I read this, I thought it was referring to scattering data somehow :).

Fixes https://github.com/pydata/xarray/issues/3206

Example usage:

In [3]: import pandas as pd
   ...: import numpy as np
   ...: import xarray
   ...: df = pd.DataFrame({
   ...:     'w': range(10),
   ...:     'x': list('abcdefghij'),
   ...:     'y': np.arange(0, 100, 10),
   ...:     'z': np.ones(10),
   ...: }).set_index(['w', 'x', 'y'])
   ...:

In [4]: ds = xarray.Dataset.from_dataframe(df, sparse=True)

In [5]: ds.z.data
Out[5]: <COO: shape=(10, 10, 10), dtype=float64, nnz=10, fill_value=nan>

• [x] Closes #3206, Closes #2139
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

min and max are now working.

notnull was already fixed by one of my earlier PRs.

std/var/median are still broken, but only because sparse hasn't implemented the corresponding NumPy functions yet (nanstd, nanvar and nanmedian).

rank needs pure NumPy implementation (not via bottleneck) if we want it to work on sparse or dask arrays.

• [x] Tests added
• [x] Passes black . && mypy . && flake8

We shouldn't be checking the details of dask's repr.

• [x] Tests added
• [x] Passes black . && mypy . && flake8

I also did a bit of cleanup (e.g., renaming methods) in xarray.core.formatting.

Sparse arrays were previously not shown in the Dataset repr:

<xarray.Dataset>
Dimensions:  (x: 4)
Coordinates:
    y        (x) int64 ...
Dimensions without coordinates: x
Data variables:
    a        (x) float64 ..."""

Now they are:

<xarray.Dataset>
Dimensions:  (x: 4)
Coordinates:
    y        (x) int64 <COO: shape=(4,), nnz=3, fill_value=0>
Dimensions without coordinates: x
Data variables:
    a        (x) float64 <COO: shape=(4,), nnz=4, fill_value=0.0>"""

• [x] Tests added
• [x] Passes black . && mypy . && flake8

It has some questionable coercion logic that no longer seems to be necessary.

Not a user facing change.

• [x] Passes black . && mypy . && flake8

This version should be much more compatible out of the box with duck typing.

No user facing changes.

This should get things building again.

xref https://github.com/pydata/xarray/issues/3154

The upstream issue that required the constraint seems to have been fixed: https://github.com/barronh/pseudonetcdf/issues/69

In theory, this should make codecov wait until reports from Azure are back (and require that all Azure checks pass) before posting its comment.

NetCDF-C and HDF5 are not threads-safe, so it's likely that closing the file object associated with one group could invalidate other open groups from the same file.

Now, we cache a single object corresponding to the root group for each file, and access sub-groups on the fly as needed.

• [x] Closes https://github.com/pydata/xarray/issues/2954
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• Suppressed various warnings.
• PsuedoNetCDF was not getting tested. Now it is.
• Added test coverage for print_versions.py

Using pre-built wheels for NumPy and pandas, so hopefully things will install with reasonable speed.

Adapted from https://github.com/dask/dask/blob/2.0.0/continuous_integration/travis/install.sh

The location of the XML output was being changed by pytest-azurepipelines.

More progress towards https://github.com/pydata/xarray/issues/1603 without any user facing changes.

The only part that's left is explicit index handling in merge.

Azure seems to be working pretty well, e.g., we now use it for test coverage with codecov. I don't see any particular reason to keep Travis around at this point.

The old setup didn't seem to work on CI, even when we explicitly passed the relevant flags.

xref https://github.com/Unidata/MetPy/issues/1077

No user facing changes

See https://github.com/tonybaloney/pytest-azurepipelines

No user facing changes

See https://github.com/pydata/xarray/pull/3058/checks?check_run_id=158702766 for an example failure

No user facing changes.

• Exclude _version.py from coverage, since it was generated from versioneer.
• Remove the "reach" graph from comments.
• Don't give PRs a failing status if they decrease coverage.

Apparently some users have IT polices prohibiting the use of "beta" software!

It's being removed in the next pandas 0.25 release (see https://github.com/pandas-dev/pandas/pull/27101).

• [x] Closes #3057
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This fixes test failures on master, as noted in #2706

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

xref https://github.com/pydata/xarray/issues/2977

Previously, these operations could silently return incorrect results (dask 2.0), or use unbounded amounts of memory (older versions of dask).

This requires a fairly large refactoring, because deciding when to use bottleneck now needs to be done at runtime rather than at import-time. These methods are now constructed as methods rather being injected aftewards into the class, which should also be a much more standard and understable design.

• [x] Closes #2940
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Template time expansion should reduce the noise in the CI output.

Follow on to https://github.com/pydata/xarray/pull/3039

xref https://github.com/astropy/astropy/pull/8445

xref #2940, #2942

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Fixes https://github.com/pydata/xarray/issues/3009

Previously, indexing an ImplicitToExplicitIndexingAdapter object could directly return an ExplicitlyIndexed object, which could not be indexed normally, e.g., x[index] could result in an object that could not be indexed properly. This resulted in broken behavior with dask's new _meta attribute.

I'm pretty sure this fix is appropriate, but it does introduce two failing tests with xarray on dask master. In particular, there are now errors raised inside two tests from dask's blockwise_meta helper function: ```

  return meta.astype(dtype)

E AttributeError: 'ImplicitToExplicitIndexingAdapter' object has no attribute 'astype' ```

cc @mrocklin @pentschev

• [x] Tests added

The first commit here is just https://github.com/pydata/xarray/pull/2858/files

It appears that we have our work cut-out for us -- about 180 failing tests if we turn on these more rigorous checks by default: ``` collected 8308 items

xarray/tests/test_accessors.py ............................................................................... [ 0%] .................................................................. [ 1%] xarray/tests/test_backends.py ...............................x..F.................F........................... [ 2%] ......................x..F.................F......................F.FFFF.FFFFF.FFFFFFF......F.FxFFFF..F....... [ 4%] s...F.F.F.xxxxFFF.FFFF.FFFFF.FFFFFFF......F.FxFFFF..F.......s...F.F.F.xxxxFF..............................x..F [ 5%] .............................................x..F...............ss..............................x..F.......... [ 6%] .......................................x..F..........F...................................x..F................. [ 8%] ...........................x..F..F..............................................x..F.................F........ [ 9%] .......FF..................................x..F.................F...............FF......F..................... [ 10%] ........x..F.................F...............FF......................................................sssssssss [ 11%] sssssss.....................................x......F........ssssssssssssssssssssssssssssssssssssssssssssssssss [ 13%] ssssssssssssssssssssssssssssss.......................................................................... [ 14%] xarray/tests/test_backends_api.py . [ 14%] xarray/tests/test_backends_file_manager.py ........................... [ 14%] xarray/tests/test_backends_locks.py . [ 14%] xarray/tests/test_backends_lru_cache.py ........ [ 14%] xarray/tests/test_cftime_offsets.py .......................................................................... [ 15%] .............................................................................................................. [ 17%] .............................................................................................................. [ 18%] .............................................................................................................. [ 19%] .............................................................................................................. [ 21%] .............................................................................................................. [ 22%] .............................................................................................................. [ 23%] .............................................................................................................. [ 25%] .............................................................................................................. [ 26%] .............................................................................................................. [ 27%] .............................................................................................................. [ 29%] .............................................................................................................. [ 30%] .............................................................................................................. [ 31%] .............................................................................................................. [ 33%] .............................................................................................................. [ 34%] .............................................................................................................. [ 35%] .............................................................................................................. [ 37%] .............................................................................................................. [ 38%] .............................................................................................................. [ 39%] ........ [ 39%] xarray/tests/test_cftimeindex.py ............................................................................. [ 40%] .............................................................................................................. [ 42%] .............................................................................................................. [ 43%] .............................................................................................................. [ 44%] .............................................................................................................. [ 46%] .......................................................................................x [ 47%] xarray/tests/test_cftimeindex_resample.py .................................................................... [ 47%] .............................................................................................................. [ 49%] .............................................................................................................. [ 50%] .............................................................................................................. [ 51%] .............................................................................................................. [ 53%] ................................................................................... [ 54%] xarray/tests/test_coding.py ......... [ 54%] xarray/tests/test_coding_strings.py ................... [ 54%] xarray/tests/test_coding_times.py ............................................................................ [ 55%] .............................................................................................................. [ 56%] .............................................................................................................. [ 58%] .............................. [ 58%] xarray/tests/test_combine.py F.................................................x [ 59%] xarray/tests/test_computation.py ................................. [ 59%] xarray/tests/test_conventions.py ..............................................x..F............s... [ 60%] xarray/tests/test_dask.py ................................................x................... [ 61%] xarray/tests/test_dataarray.py ............F.......................................................FF......... [ 62%] ................................s.s...F...................FFFF........................sss..................... [ 63%] .............................................................................................................. [ 64%] .............................................................................................................. [ 66%] .............................................................................................................. [ 67%] ..........................................................................................................ssss [ 68%] sssssssss [ 68%] xarray/tests/test_dataset.py ..........................F..............F...FF....................F.F........... [ 69%] ...s.FFFFF...................FF..............F..............................FFF.F.FFF.FFFF..FFFFFFF...FFFF..FF [ 71%] FFFF....F.....FFFFFF.......................................................................................... [ 72%] .............................................................................................................. [ 73%] ......................................................................................ssssssssssssssss........ [ 75%] .............................................................................................................. [ 76%] .............................................................................................................. [ 77%] ........................................................................................... [ 78%] xarray/tests/test_distributed.py FFFFFFFFFFsFsFss.FF [ 79%] xarray/tests/test_dtypes.py ....................................... [ 79%] xarray/tests/test_duck_array_ops.py .....................................ss..............ss..............ss... [ 80%] ...ss......ss..............ss......................................................ss......ss................. [ 81%] .............ss..............ss..............ss......ss......ss..............ss............................... [ 83%] .......................ss......ss................ssssssssss..........ssssssssss..........ssssssssss..........s [ 84%] sssssssss..........ssssss..............ssssss..............ssssss..............ssssss..............sssssssssss [ 85%] .s.s.s.s.sssssssssss.s.s.s.s.sssssssssss.s.s.s.s.sssssssssss.s.s.s.s.sssssss.s.s.s.s.s.s.sssssss.s.s.s.s.s.s.s [ 87%] ssssss.s.s.s.s.s.s.sssssss.s.s.s.s.s.s........................................................................ [ 88%] .................................. [ 88%] xarray/tests/test_extensions.py .... [ 88%] xarray/tests/test_formatting.py ............... [ 88%] xarray/tests/test_groupby.py ........ [ 89%] xarray/tests/test_indexing.py ................................................................................ [ 90%] ...................................... [ 90%] xarray/tests/test_interp.py FFFFFFFssFF..FF.........FF...FFFFFx.F.F.... [ 90%] xarray/tests/test_merge.py ................. [ 91%] xarray/tests/test_merge.py ................. [ 91%] xarray/tests/test_missing.py ................................. [ 91%] xarray/tests/test_nputils.py ... [ 91%] xarray/tests/test_options.py ...........x [ 91%] xarray/tests/test_plot.py .................................................................................... [ 12%] .............................................................................................................s [ 28%] .....................................................................................sss...................... [ 44%] ......................... [ 48%] xarray/tests/test_testing.py . [ 48%] xarray/tests/test_tutorial.py ss [ 48%] xarray/tests/test_ufuncs.py FFF..................................................................... [ 59%] xarray/tests/test_utils.py ............................ [ 63%] xarray/tests/test_variable.py ................................................................................ [ 74%] ................................................................................xxxX.......................... [ 91%] .......................................xxxxxxx............... [100%]

============= 180 failed, 7733 passed, 356 skipped, 38 xfailed, 1 xpassed, 10 warnings in 465.51 seconds ============= ```

We don't want 10000 lines printing every single test :)

This build is currently install NumPy 1.11, which isn't supported by xarray. Maybe adding minimum numpy and pandas versions will help.

I've added internal consistency checks to the uses of assert_equal in our test suite, so this shouldn't happen again.

• [x] Closes #2856, closes #2854
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2842
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This deprecation was finished in v0.11.0.

0.12 will already have a big change in dropping Python 2.7 support. I'd rather wait a bit longer to finalize these deprecations to minimize the impact on users.

xref https://github.com/pydata/xarray/issues/2776

These aren't valid for netCDF files.

Fixes #2803

flake8 includes a few more useful checks, but it's annoying to only see it's output in Travis-CI results.

This keeps Travis-CI and pep8speaks in sync.

This lets us remove lots of ugly explicit calls to kwargs.pop().

Apparently I wasn't paying attention in my last PR :)

This is part of the larger project in https://github.com/pydata/xarray/issues/1603

None of this should change public APIs: we're simply making updating indexes explicit (via the indexes dict) rather than implicit (through variables).

Fixes https://github.com/pydata/xarray/issues/2050

I'm not quite sure what was going on, but it passes now.

• [x] Closes #2050

Example usage:

```

import xarray times = xarray.cftime_range('2000', periods=30, freq='MS') da = xarray.DataArray(range(30), [('time', times)]) da.resample(time='1AS').mean() <xarray.DataArray (time: 3)> array([ 5.5, 17.5, 26.5]) Coordinates: * time (time) object 2001-01-01 00:00:00 ... 2003-01-01 00:00:00 ```

• [x] Closes #2717
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I just pushed the 0.11.3 bug-fix release to pypi.

We were pinning very old versions for most of these packages. This should fix the failures on ReadTheDocs after https://github.com/pydata/xarray/pull/2707 goes in.

• [x] Closes #2705

This should make it easier to debug the doc build environment.

This was introduced by the recent indexes refactor, but never made it into a release.

I'll port these to master separately, given that this commit probably won't apply cleanly.

Fixes https://github.com/pydata/xarray/issues/2673

Note that these were only test failures, not a real bug.

• [x] Closes #2673
• [x] Tests added

It was a nice idea to support CFTimeIndex in a pandas.MultiIndex, but pandas seems to have inadvertently broken this, see https://github.com/pandas-dev/pandas/issues/24263

(This should fix our failing CI tests with pandas 0.24 rc)

This documentation revamp builds upon @rabernat's rewrite in #2430.

The main change is that the three paragraph description felt too long to me, so I moved the background paragraph on multi-dimensional arrays into the next section, on "Why xarray". I also ended up rewriting most of that page, and made a few adjustments to the FAQ and related projects pages.

Remove the optional condaforge-rc, netcdf4-dev and pynio-dev builds. These have been continuously failing (due to broken installs), so we shouldn't waste time/energy running them.

The rest of the scientific Python stack doesn't seem to support type annotations yet, but that's OK -- we can use this incrementally in xarray when it seems appropriate, and may check a few bugs. I'm especially excited to use this for internal functions, where we don't always bother with full docstrings (e.g., what is the type of the variables argument?).

This includes: 1. various minor fixes to ensure that "mypy xarray" passes. 2. ~~adding "mypy xarray" to our lint check on Travis-CI.~~

For reference, see "Using mypy with an existing codebase": https://mypy.readthedocs.io/en/stable/existing_code.html

Question: are we OK with (2)? This means Travis-CI will fail if your code causes mypy to error.

xref https://github.com/pydata/xarray/issues/1603

This change switches Dataset.indexes and DataArray.indexes to be backed by explicit dictionaries of indexes, instead of being implicitly defined by the set of coordinates with names matching dimensions.

There are no changes to the public interface yet: these will come later.

My current plan: 1. (This PR) Indexes are recreated from coordinates every time a new DataArray or Dataset is created. 2. (Follow-up PRs) Refactor indexes to be propagated explicitly in xarray operations. This will facilitate future API changes, when indexes will no longer only be associated with dimensions. I will probably add some testing decorator that can be used to mark part of a test as including no creation of default indexes. 3. Add explicit entries into indexes for MultiIndex levels that are checked instead of MultiIndex variables. Still no public API changes (aside from adding more entries to .indexes). 4. Support arbitrary coordinates in indexes.

• [x] Closes #2641
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

These were adding a lot of noise to the output of our test suite -- something like 500 lines of useless warnings!

This frees users from needing to worry about this.

Using __del__ turned up to be easier than using weak references.

• [x] Closes #2560
• [x] Closes #2614
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new AP

There seems to be some sort of dependency issue on Appveyor, but it's not worth tracking down given how we'll be dropping Python 2.7 in the new year anyways.