issues

Per https://github.com/pydata/xarray/issues/8844#issuecomment-2089427222, for the time being this marks test_use_cftime_false_standard_calendar_in_range as an expected failure under NumPy 2. Hopefully we'll be able to fix the upstream issue in pandas eventually.

This PR fixes the issue noted in https://github.com/pydata/xarray/issues/7493#issuecomment-1953091000 that non-nanosecond precision datetime or timedelta values passed to expand_dims would not be cast to nanosecond precision. The underlying issue was that the _possibly_convert_datetime_or_timedelta_index function did not appropriately handle being passed PandasIndexingAdapter objects.

• [x] Fixes https://github.com/pydata/xarray/issues/7493#issuecomment-1953091000
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I finally had a moment to think about this some more following discussion in https://github.com/pydata/xarray/pull/8253. This PR adds a chunk-friendly code path to encode_cf_datetime and encode_cf_timedelta, which enables lazy encoding of time-like values, and by extension, preservation of chunks when writing time-like values to zarr. With these changes, the test added by @malmans2 in #8253 passes.

Though it largely reuses existing code, the lazy encoding implemented in this PR is stricter than eager encoding in a couple ways: 1. It requires either both the encoding units and dtype be prescribed, or neither be prescribed; prescribing one or the other is not supported, since it requires inferring one or the other from the data. In the case that neither is specified, the dtype is set to np.int64 and the units are either "nanoseconds since 1970-01-01" or "microseconds since 1970-01-01" depending on whether we are encoding np.datetime64[ns] values or cftime.datetime objects. In the case of timedelta64[ns] values, the units are set to "nanoseconds". 2. In addition, if an integer dtype is prescribed, but the units are set such that floating point values would be required, it raises instead of modifying the units to enable integer encoding. This is a requirement since the data units may differ between chunks, so overriding could result in inconsistent units.

As part of this PR, since dask requires we know the dtype of the array returned by the function passed to map_blocks, I also added logic to handle casting to the specified encoding dtype in an overflow-and-integer safe manner. This means an informative error message would be raised in the situation described in #8542:

OverflowError: Not possible to cast encoded times from dtype('int64') to dtype('int16') without overflow. Consider removing the dtype encoding, at which point xarray will make an appropriate choice, or explicitly switching to a larger integer dtype.

I eventually want to think about this on the decoding side as well, but that can wait for another PR.

• [x] Closes #7132
• [x] Closes #8230
• [x] Closes #8432
• [x] Closes #8253
• [x] Addresses #8542
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Following several upstream PRs in pandas, this PR deprecates cftime frequency strings "A", "AS", "Q", "M", "H", "T", "S", "L", and "U" in favor of "Y", "YS", "QE", "ME", "h", "min", "s", "ms", and "us". Similarly following pandas, it makes a breaking change to have infer_freq return the latter frequencies instead of the former.

There are a few places in the tests and one place in the code where we need some version-specific logic to retain support for older pandas versions. @aulemahal it would be great if you could take a look to make sure that I handled this breaking change properly / fully in the date_range_like case.

I also took the liberty to transition to using "Y", "YS", "h", "min", "s", "ms", "us", and "ns" within our code, tests, and documentation to reduce the amount of warnings emitted. I have held off on switching to "QE", "ME", and anchored offsets involving "Y" or "YS" in pandas-related code since those are not supported in older versions of pandas.

The deprecation warning looks like this: ```

xr.cftime_range("2000", periods=5, freq="M") <stdin>:1: FutureWarning: 'M' is deprecated and will be removed in a future version. Please use 'ME' instead of 'M'. CFTimeIndex([2000-01-31 00:00:00, 2000-02-29 00:00:00, 2000-03-31 00:00:00, 2000-04-30 00:00:00, 2000-05-31 00:00:00], dtype='object', length=5, calendar='standard', freq='ME') ```

• [x] Closes #8394
• [x] Addresses the convert_calendar and date_range_like test failures in #8091
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

Pandas has introduced some deprecations in how frequency strings are specified:

• Deprecating "A", "A-JAN", etc. in favor of "Y", "Y-JAN", etc. (https://github.com/pandas-dev/pandas/pull/55252)
• Deprecating "AS", "AS-JAN", etc. in favor of "YS", "YS-JAN", etc. (https://github.com/pandas-dev/pandas/pull/55479)
• Deprecating "Q", "Q-JAN", etc. in favor of "QE", "QE-JAN", etc. (https://github.com/pandas-dev/pandas/pull/55553)
• Deprecating "M" in favor of "ME" (https://github.com/pandas-dev/pandas/pull/54061)
• Deprecating "H" in favor of "h" (https://github.com/pandas-dev/pandas/pull/54939)
• Deprecating "T", "S", "L", and "U" in favor of "min", "s", "ms", and "us" (https://github.com/pandas-dev/pandas/pull/54061).

It would be good to carry these deprecations out for cftime frequency specifications to remain consistent.

The remaining failing cftime resample tests in https://github.com/pydata/xarray/issues/8091 happen to be related to a bug that was fixed in the pandas implementation, https://github.com/pandas-dev/pandas/pull/55283, leading answers to change in some circumstances. This PR ports that bug fix to xarray's implementation of resample for data indexed by a CFTimeIndex.

• [x] Fixes remaining failing cftime resample tests in https://github.com/pydata/xarray/issues/8091
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

A simple example where answers change in pandas is the following:

Previously

```

import numpy as np; import pandas as pd index = pd.date_range("2000", periods=5, freq="5D") series = pd.Series(np.arange(index.size), index=index) series.resample("2D", closed="right", label="right", offset="1s").mean() 2000-01-01 00:00:01 0.0 2000-01-03 00:00:01 NaN 2000-01-05 00:00:01 1.0 2000-01-07 00:00:01 NaN 2000-01-09 00:00:01 NaN 2000-01-11 00:00:01 2.0 2000-01-13 00:00:01 NaN 2000-01-15 00:00:01 3.0 2000-01-17 00:00:01 NaN 2000-01-19 00:00:01 NaN 2000-01-21 00:00:01 4.0 Freq: 2D, dtype: float64 ```

Currently

```

import numpy as np; import pandas as pd index = pd.date_range("2000", periods=5, freq="5D") series = pd.Series(np.arange(index.size), index=index) series.resample("2D", closed="right", label="right", offset="1s").mean() 2000-01-01 00:00:01 0.0 2000-01-03 00:00:01 NaN 2000-01-05 00:00:01 NaN 2000-01-07 00:00:01 1.0 2000-01-09 00:00:01 NaN 2000-01-11 00:00:01 2.0 2000-01-13 00:00:01 NaN 2000-01-15 00:00:01 NaN 2000-01-17 00:00:01 3.0 2000-01-19 00:00:01 NaN 2000-01-21 00:00:01 4.0 Freq: 2D, dtype: float64 ```

This PR allows us to reproduce this change in xarray for data indexed by a CFTimeIndex. The bin edges were incorrect in the previous case; see https://github.com/pandas-dev/pandas/pull/52064#issuecomment-1785893752 for @MarcoGorelli's nice explanation as to why.

This PR proposes a fix to #8271. I think the basic issue is that the only time we need to update the needed_units is if the data_delta does not evenly divide the ref_delta. If it does evenly divide it--as it does in the example in #8271--and we try to update the needed_units solely according to the value of the ref_delta, we run the risk of resetting them to something that would be coarser than the data requires. If it does not evenly divide it, we are safe to reset the needed_units because they will be guaranteed to be finer-grained than the data requires.

I modified test_roundtrip_float_times to reflect the example given by @larsbuntemeyer in #8271. @kmuehlbauer let me know if this fix makes sense to you.

• [x] Closes #8271
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This addresses the remaining cftime-related test failures in #7707 by introducing a function that always returns a nanosecond-precision Timestamp object. Despite no corresponding test failures, for safety I grepped and went ahead and replaced the pd.Timestamp constructor with this function in a few other areas. I also updated our documentation to replace any mentions of the "Timestamp-valid range" with "nanosecond-precision range" since Timestamps are now more flexible, and included a note that we have an issue open for relaxing this nanosecond-precision assumption in xarray eventually.

While in principle I think it would be fine if CFTimeIndex.to_datetimeindex returned a DatetimeIndex with non-nanosecond-precision values, since we don't use to_datetimeindex anywhere outside of tests, in its current state it was returning nonsense values: ```

import pandas as pd import xarray as xr times = xr.cftime_range("0001", periods=5) times.to_datetimeindex() DatetimeIndex(['1754-08-30 22:43:41.128654848', '1754-08-31 22:43:41.128654848', '1754-09-01 22:43:41.128654848', '1754-09-02 22:43:41.128654848', '1754-09-03 22:43:41.128654848'], dtype='datetime64[ns]', freq=None) `` This is due to the assumption incftime_to_nptimethat the resulting array will have nanosecond-precision values. We can (and should) address this eventually, but for the sake of quickly supporting pandas version two I decided to be conservative and punt this off to be part of #7493.cftime_to_nptimeis used in places other thanto_datetimeindex`, so modifying it has other impacts downstream.

As suggested by @keewis, to preserve existing behavior in xarray, this PR forces any object passed to format_timestamp to be converted to a string using strftime with a constant format. This addresses the failing tests related to the units encoding in #7420.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

While pandas is getting set to remove the base and loffset arguments in resample, we have not had a chance to emit a deprecation warning for them yet in xarray (https://github.com/pydata/xarray/issues/7420). This PR preserves their functionality in xarray and should hopefully give users some extra time to adapt. Deprecation warnings for each are added so that we can eventually remove them.

I've taken the liberty to define a TimeResampleGrouper object, since we need some way to carry the loffset argument through the resample chain, even though it will no longer be allowed on the pd.Grouper object. Currently it is not particularly complicated, so hopefully it would be straightforward to adapt to what is envisioned in https://github.com/pydata/xarray/issues/6610#issuecomment-1341296800.

• [x] closes #7266
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Following pandas, this PR adds an inclusive argument to xarray.cftime_range and xarray.date_range and deprecates the closed argument. Pandas will be removing the closed argument soon in their date_range implementation, but we will continue supporting it to allow for our own deprecation cycle.

I think we may also need to update our minimum pandas version to 1.4 for this, since earlier versions of pandas do not support the inclusive argument.

• [x] Closes #6985
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I noticed the PR number was off in my what’s new entry in #7284. This fixes that.

This PR enables the origin and offset arguments in resample. This was simple to do in the case of data indexed by a DatetimeIndex, but naturally required changes to our internal implementation of resample for data indexed by a CFTimeIndex. Fortunately those changes were fairly straightforward to port over from pandas.

This does not do anything to address the deprecation of base noted in #7266, but is an important first step toward getting up to speed with the latest version of pandas, both on the DatetimeIndex side and the CFTimeIndex side. This way we will at least be able to handle that deprecation in the same way for each.

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I think things are fairly comprehensively implemented and tested here, but I'm marking this as a draft for now as I want to see if I can reduce the number of cftime resampling tests some, which have multiplied with the addition of these new arguments.

Thanks for the feedback @hmaarrfk. Is this what you had in mind?

• [x] Closes #7237

For example running this script: ```python import numpy as np import xarray as xr

times = [np.datetime64("2000-01-01", "us")] var = xr.Variable(["time"], times) da = xr.DataArray(times) leads to the following warnings: $ python test_warning.py test_warning.py:6: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time. var = xr.Variable(["time"], times) test_warning.py:7: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time. da = xr.DataArray(times) ```

This PR addresses #7175 by converting datetime or timedelta values to nanosecond precision even if pandas does not. For the time being we emit a warning when pandas does not do the conversion, but we do (right now this is only in the development version of pandas). When things stabilize in pandas we can consider relaxing this constraint in xarray as well.

This got a little bit more complicated due to the presence of timezone-aware datetimes in pandas, but hopefully the tests cover those cases now.

• [x] Closes #7175
• [x] Closes #7197
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Currently, the default behavior of cftime.date2num is to return integer values when possible (i.e. when the encoding units allow), and fall back to returning float64 values when that is not possible. Recently, cftime added the option to use float128 as the fallback dtype, which enables greater potential roundtrip precision. This is through the longdouble flag to cftime.date2num, which currently defaults to False. It was intentionally set to False by default, because netCDF does not support storing float128 values in files, and so, without any changes, would otherwise break xarray's encoding procedure.

The desire in cftime, however, is to eventually set this flag to True by default (https://github.com/Unidata/cftime/issues/297). This PR makes the necessary changes in xarray to adapt to this eventual new default. Essentially if the longdouble argument is allowed in the user's version of cftime.date2num, we explicitly set it to False to preserve the current float64 fallback behavior within the context of encoding times. There are a few more places where date2num is used (some additional places in the tests, and in calendar_ops.py), but in those places using float128 values would not present a problem.

At some point we might consider relaxing this behavior in xarray, since it is possible to store float128 values in zarr stores for example, but for the time being the simplest approach seems to be to stick with float64 for all backends (it would be complicated to have backend-specific defaults).

cc: @jswhit

What is your issue?

If any variable in a Dataset has times that cannot be represented as cftime.datetime objects, an error message will be raised. However, this error message will not indicate the problematic variable's name. It would be nice if it did, because it would make it easier for users to determine the source of the error.

cc: @durack1 xref: Unidata/cftime#295

Example

This is a minimal example of the issue. The error message gives no indication that "invalid_times" is the problem:

```

import xarray as xr TIME_ATTRS = {"units": "days since 0001-01-01", "calendar": "noleap"} valid_times = xr.DataArray([0, 1], dims=["time"], attrs=TIME_ATTRS, name="valid_times") invalid_times = xr.DataArray([1e36, 2e36], dims=["time"], attrs=TIME_ATTRS, name="invalid_times") ds = xr.merge([valid_times, invalid_times]) xr.decode_cf(ds) Traceback (most recent call last): File "/Users/spencer/software/xarray/xarray/coding/times.py", line 275, in decode_cf_datetime dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 210, in _decode_datetime_with_pandas raise OutOfBoundsDatetime( pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/spencer/software/xarray/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype result = decode_cf_datetime(example_value, units, calendar, use_cftime) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 277, in decode_cf_datetime dates = _decode_datetime_with_cftime( File "/Users/spencer/software/xarray/xarray/coding/times.py", line 202, in _decode_datetime_with_cftime cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True) File "src/cftime/_cftime.pyx", line 605, in cftime._cftime.num2date File "src/cftime/_cftime.pyx", line 404, in cftime._cftime.cast_to_int OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/spencer/software/xarray/xarray/conventions.py", line 655, in decode_cf vars, attrs, coord_names = decode_cf_variables( File "/Users/spencer/software/xarray/xarray/conventions.py", line 521, in decode_cf_variables new_vars[k] = decode_cf_variable( File "/Users/spencer/software/xarray/xarray/conventions.py", line 369, in decode_cf_variable var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 687, in decode dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype raise ValueError(msg) ValueError: unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed. ```

• [x] Closes #7145
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I'm not sure if there is a better way to do this, but this is one way to address #7145. The error message for the example now looks like:

```

xr.decode_cf(ds) Traceback (most recent call last): File "/Users/spencer/software/xarray/xarray/coding/times.py", line 275, in decode_cf_datetime dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 210, in _decode_datetime_with_pandas raise OutOfBoundsDatetime( pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/spencer/software/xarray/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype result = decode_cf_datetime(example_value, units, calendar, use_cftime) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 277, in decode_cf_datetime dates = _decode_datetime_with_cftime( File "/Users/spencer/software/xarray/xarray/coding/times.py", line 202, in _decode_datetime_with_cftime cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True) File "src/cftime/_cftime.pyx", line 605, in cftime._cftime.num2date File "src/cftime/_cftime.pyx", line 404, in cftime._cftime.cast_to_int OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/spencer/software/xarray/xarray/conventions.py", line 523, in decode_cf_variables new_vars[k] = decode_cf_variable( File "/Users/spencer/software/xarray/xarray/conventions.py", line 369, in decode_cf_variable var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 688, in decode dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime) File "/Users/spencer/software/xarray/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype raise ValueError(msg) ValueError: unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/spencer/software/xarray/xarray/conventions.py", line 659, in decode_cf vars, attrs, coord_names = decode_cf_variables( File "/Users/spencer/software/xarray/xarray/conventions.py", line 534, in decode_cf_variables raise type(e)(f"Failed to decode variable {k!r}: {e}") ValueError: Failed to decode variable 'invalid_times': unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed. ```

This was essentially enabled by @dcherian in #6556, but we did not remove the error that prevented computing the mean of a dask-backed cftime array. This PR removes that error, and adds some tests. One minor modification in _timedelta_to_seconds was needed for compatibility with scalar cftime arrays.

This happens to address the second part of #5897, so I added a regression test for that. It seems like we decided to simply document the behavior in the first part (https://github.com/pydata/xarray/issues/5898, https://github.com/dcherian/xarray/commit/99bfe128066ec3ef1b297650a47e2dd0a45801a8), but I'm not sure if we intend to change that behavior eventually or not.

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR simplifies the tests for the calendar attribute on the dt accessor when using a datetime64[ns]-dtype DataArray. Instead of creating random-valued datetime arrays, we can use arrays of zeros (i.e. 1970-01-01), since the values of the datetimes should not be relevant to these tests (only their type matters).

I suspect this should address #6906, because it eliminates the need to convert to datetime64[ns], though I still feel as though there is a more fundamental pandas issue lurking there.

The development version of pandas raises an OutOfBoundsTimedelta error instead of an OverflowError in pd.to_timedelta if the timedelta cannot be represented with nanosecond precision. Therefore we must also be ready to catch that when decoding times.

The OutOfBoundsTimedelta exception was added in pandas version 1.1, which is prior to our current minimum version (1.2), so it should be safe to import without a version check.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6589
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

It seems rare that decode_cf_variable would be called on variables that contain datetime-like objects already, but in the case that it is, it seems best to let those variables pass through unmodified.

• [x] Closes #6453
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

On the topic of FutureWarning's related to indexing in pandas (#5721), I noticed some another kind of warning in the CFTimeIndex tests: /Users/spencer/software/xarray/xarray/tests/test_cftimeindex.py:350: FutureWarning: 'kind' argument in get_slice_bound is deprecated and will be removed in a future version. Do not pass it. result = index.get_slice_bound("0001", "left", kind) I think it's safe to silence these by removing the kind argument from these tests. We never used it anyway in CFTimeIndex. This is sort of a follow-up to #5359.

• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I think this would be good to do anyway, but I'm also curious to see if it fixes the cftime plotting tests in #5743.

This addresses the test_datetime_mean failures reported in #5366. Pandas now requires that we make sure the units of datetime arrays are specified explicitly in array constructors: https://github.com/pandas-dev/pandas/issues/36615#issuecomment-860040013.

• [x] Passes pre-commit run --all-files

nc-time-axis version 1.3.0 was released today (thanks @bjlittle!), which includes various fixes for incompatibilities with the latest version of cftime. This means that our tests that depend on nc-time-axis should now pass.

• [x] Closes #5344

Pandas recently deprecated the kind argument in Index._maybe_cast_slice_bound, and removed its use in several internal calls: https://github.com/pandas-dev/pandas/pull/41378. This led to some errors in the CFTimeIndex tests in our upstream build. We never made use of it in CFTimeIndex._maybe_cast_slice_bound so the simplest fix for backwards compatibility seems to be to make it optional for now -- in previous versions of pandas it was required -- and remove it when our minimum version of pandas is at least 1.3.0.

• [x] Closes #5356
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This fixes the issue in #5093, by ensuring that we always convert the calendar to lowercase before checking if it is one of the standard calendars in the decoding and encoding process. I've been careful to test that the calendar attribute is faithfully roundtripped despite this, uppercase letters and all.

~~I think part of the reason this went unnoticed for a while was that we could still decode times like this if cftime was installed; it is only in the case when cftime was not installed that our logic failed. This is because cftime.num2date already converts the calendar to lowercase internally.~~

Upon re-reading @pont-us's issue description, while it didn't cause an error, the behavior was incorrect with cftime installed too. I updated the test to check the dtype is np.datetime64 as well.

• [x] Closes #5093
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

It seems that pandas did not include the change that led to #5006 in their latest release. Perhaps it is safer to just catch either error regardless of the pandas version.

• [x] Closes #5147

The exception that was raised in pandas when a datetime.timedelta object outside the range that could be expressed in units of nanoseconds was passed to the pandas.TimedeltaIndex constructor changed from an OverflowError to an OutOfBoundsTimedelta error in the development version of pandas. This PR adjusts our exception handling logic in CFTimeIndex.__sub__ and CFTimeIndex.__rsub__ to account for this.

• [x] closes #4947

As suggested by @dcherian, this was quite similar to rolling; it was useful in particular to follow how the tests were implemented there.

• [x] Closes #3741
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #4097
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Following up on #4684, this PR makes changes to our encoding / decoding process such that cftime.datetime objects can be roundtripped exactly. In the process, because it made the tests cleaner to define, I added cftime offsets for millisecond and microsecond frequency as well.

As I note in the what's new, exact roundtripping requires cftime of at least version 1.4.1, which included improvements to cftime.num2date (https://github.com/Unidata/cftime/pull/176, https://github.com/Unidata/cftime/pull/188) and cftime.date2num (https://github.com/Unidata/cftime/pull/178, https://github.com/Unidata/cftime/pull/225).

• [x] Closes #4045
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR cleans up the logic used to encode and decode times with pandas so that by default we use int64 values in both directions for all precisions down to nanosecond. If a user specifies an encoding (or a file is read in) such that float values would be required, things still work as they did before. I do this mainly by following the approach I described here: https://github.com/pydata/xarray/issues/4045#issuecomment-626257580.

In the process of doing this I made a few changes to coding.times._decode_datetime_with_pandas: - I removed the checks on the minimum and maximum dates to decode, as the issue those checks were imposed for (#975) was fixed in pandas way back in 2016 (https://github.com/pandas-dev/pandas/issues/14068). - I used an alternate approach for fixing #2002, which allows us to continue to use the optimization made in #1414 without having to cast the input array to a float dtype first.

Note this will change the default units that are chosen for encoding times in some instances -- previously we would never default to anything more precise than seconds -- but I think this change is for the better.

cc: @aldanor

@hmaarrfk this overlaps a little with your work in #4400, so I'm giving you credit here too (I hope you don't mind!).

• [x] Closes #4870
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

As of cftime > 1.4.0, the return type of cftime.date2num can either be an integer or float. An integer dtype is used if the times can all be encoded exactly with the provided units; otherwise a float dtype is used. This causes problems in our current encoding pipeline, because we call cftime.date2num on dates one at a time through np.vectorize, and np.vectorize infers the type of the full returned array based on the result of the first function evaluation. If the first result is an integer, then the full array will be assumed to have an integer dtype, and any values that should be floats are cast as integers.

What happened:

``` In [1]: import cftime; import numpy as np; import xarray as xr

In [2]: times = np.array([cftime.DatetimeGregorian(2000, 1, 1), cftime.DatetimeGregorian(2000, 1, 1, 1)])

In [3]: xr.coding.times._encode_datetime_with_cftime(times, "days since 2000-01-01", calendar="gregorian") Out[3]: array([0, 0]) ```

What you expected to happen:

In [3]: xr.coding.times._encode_datetime_with_cftime(times, "days since 2000-01-01", calendar="gregorian") Out[3]: array([0. , 0.04166667])

A solution here would be to encode the times with a list comprehension instead, and cast the final result to an array, in which case NumPy infers the dtype in a more sensible way.

Environment:

Currently CFTimeIndex uses the default repr it inherits from pandas.Index. This just displays a potentially-truncated version of the values in the index, along with the index's data type and length, e.g.: CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00, 2000-01-03 00:00:00, 2000-01-04 00:00:00, 2000-01-05 00:00:00, 2000-01-06 00:00:00, 2000-01-07 00:00:00, 2000-01-08 00:00:00, 2000-01-09 00:00:00, 2000-01-10 00:00:00, ... 2000-12-22 00:00:00, 2000-12-23 00:00:00, 2000-12-24 00:00:00, 2000-12-25 00:00:00, 2000-12-26 00:00:00, 2000-12-27 00:00:00, 2000-12-28 00:00:00, 2000-12-29 00:00:00, 2000-12-30 00:00:00, 2000-12-31 00:00:00], dtype='object', length=366) It would be nice if the repr also included an indication of the calendar type of the index, since different indexes could have different calendar types. For example: CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00, 2000-01-03 00:00:00, 2000-01-04 00:00:00, 2000-01-05 00:00:00, 2000-01-06 00:00:00, 2000-01-07 00:00:00, 2000-01-08 00:00:00, 2000-01-09 00:00:00, 2000-01-10 00:00:00, ... 2000-12-22 00:00:00, 2000-12-23 00:00:00, 2000-12-24 00:00:00, 2000-12-25 00:00:00, 2000-12-26 00:00:00, 2000-12-27 00:00:00, 2000-12-28 00:00:00, 2000-12-29 00:00:00, 2000-12-30 00:00:00, 2000-12-31 00:00:00], dtype='object', length=366, calendar='proleptic_gregorian')

This is a minor cleanup to remove our use of calendar-naive cftime datetime objects (it just occurs in one test). The behavior of the cftime.datetime constructor is set to change in Unidata/cftime#202. By default it will create a calendar-aware datetime with a Gregorian calendar, instead of a calendar-naive datetime. In xarray we don't have a real need to use calendar-naive datetimes, so I think it's just best to remove our use of them.

• [x] Passes isort . && black . && mypy . && flake8

cftime is planning on renaming the base class for its datetime objects from cftime.datetime to cftime.datetime_base. See discussion in https://github.com/Unidata/cftime/issues/198 and https://github.com/Unidata/cftime/pull/199. This PR adds the appropriate logic in xarray to handle this in a backwards-compatible way.

In the documentation in places where we refer to :py:class:`cftime.datetime` objects, I have modified things to read ``cftime`` datetime. Being more generic is probably better in any case, as in most instances we do not explicitly mean that the base class can be used, only subclasses of the base class.

cc: @jswhit

• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

use_cftime was recently added as an option to decode_cf and open_dataset to give users a little more control over how times are decoded (#2759). It would be good if it was also available for open_zarr. This perhaps doesn't have quite the importance, because open_zarr only works for single data stores, so there is no risk of decoding times to different types (e.g. as there was for open_mfdataset, #1263); however, it would still be nice to be able to silence serialization warnings that result from decoding times to cftime objects in some instances, e.g. #2754.

This PR adds support for datetime strings formatted following the default cftime format (YYYY-MM-DD hh:mm:ss) in cftime_range and partial datetime string indexing.

• [x] Closes #4337
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Going through pandas.TimedeltaIndex within duck_array_ops._to_pytimedelta leads to overflow problems (presumably it casts to a "timedelta64[ns]" type internally). This PR updates the logic to directly use NumPy to do the casting, first to "timedelta64[us]", then to datetime.timedelta.

• [x] Closes #4341
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Closes #4265

The change that broke these tests in NumPy master has now been relaxed to trigger a DeprecationWarning (https://github.com/numpy/numpy/pull/16943).

Noticed this today. This note in the DatetimeAccessor docstring is no longer relevant; these fields have been calendar-aware for some time.

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This adds a days_in_month accessor to CFTimeIndex, which allows for easy computation of monthly time weights for non-standard calendars: ``` In [1]: import xarray as xr

In [2]: times = xr.cftime_range("2000", periods=24, freq="MS", calendar="noleap")

In [3]: da = xr.DataArray(times, dims=["time"])

In [4]: da.dt.days_in_month Out[4]: <xarray.DataArray 'days_in_month' (time: 24)> array([31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]) Coordinates: * time (time) object 2000-01-01 00:00:00 ... 2001-12-01 00:00:00 ```

This simplifies the "Calculating Seasonal Averages from Timeseries of Monthly Means" example @jhamman wrote for the docs a while back, which I've taken the liberty of updating.

The ability to add this feature to xarray is thanks in large part to @huard, who added a daysinmonth attribute to cftime.datetime objects late last year: https://github.com/Unidata/cftime/pull/138.

• [x] Closes #3928
• [x] Passes isort -rc . && black . && mypy . && flake8

The warning we want to avoid in these tests is: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)

Other warnings could occur, but shouldn't cause the tests to fail. This modifies these tests to only fail if a warning with this message occurs.

The warning that is occurring seems to be stemming from within the netcdf4-python library: DeprecationWarning: tostring() is deprecated. Use tobytes() instead. attributes = {k: var.getncattr(k) for k in var.ncattrs()}

With Unidata/cftime#163 merged, this test, which we temporarily xfailed in #3885, should pass with cftime master.

xref: #3869

• [x] Closes #3751
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This fixes the errors identified when #3751 was created by allowing one to subtract a pd.Index of cftime.datetime objects from a CFTimeIndex.

Some new errors have come up too (not associated with any updates I made here), which I still need to work on identifying the source of: ``` ____ test_indexing_in_series_getitem[365_day] _____

series = 0001-01-01 00:00:00 1 0001-02-01 00:00:00 2 0002-01-01 00:00:00 3 0002-02-01 00:00:00 4 dtype: int64 index = CFTimeIndex([0001-01-01 00:00:00, 0001-02-01 00:00:00, 0002-01-01 00:00:00, 0002-02-01 00:00:00], dtype='object') scalar_args = [cftime.DatetimeNoLeap(0001-01-01 00:00:00)] range_args = ['0001', slice('0001-01-01', '0001-12-30', None), slice(None, '0001-12-30', None), slice(cftime.DatetimeNoLeap(0001-01...:00), cftime.DatetimeNoLeap(0001-12-30 00:00:00), None), slice(None, cftime.DatetimeNoLeap(0001-12-30 00:00:00), None)]

@requires_cftime
def test_indexing_in_series_getitem(series, index, scalar_args, range_args):
    for arg in scalar_args:

      assert series[arg] == 1

test_cftimeindex.py:597:

../../../pandas/pandas/core/series.py:884: in getitem return self._get_with(key)

self = 0001-01-01 00:00:00 1 0001-02-01 00:00:00 2 0002-01-01 00:00:00 3 0002-02-01 00:00:00 4 dtype: int64 key = cftime.DatetimeNoLeap(0001-01-01 00:00:00)

def _get_with(self, key):
    # other: fancy integer or otherwise
    if isinstance(key, slice):
        # _convert_slice_indexer to determing if this slice is positional
        #  or label based, and if the latter, convert to positional
        slobj = self.index._convert_slice_indexer(key, kind="getitem")
        return self._slice(slobj)
    elif isinstance(key, ABCDataFrame):
        raise TypeError(
            "Indexing a Series with DataFrame is not "
            "supported, use the appropriate DataFrame column"
        )
    elif isinstance(key, tuple):
        try:
            return self._get_values_tuple(key)
        except ValueError:
            # if we don't have a MultiIndex, we may still be able to handle
            #  a 1-tuple.  see test_1tuple_without_multiindex
            if len(key) == 1:
                key = key[0]
                if isinstance(key, slice):
                    return self._get_values(key)
            raise

    if not isinstance(key, (list, np.ndarray, ExtensionArray, Series, Index)):

      key = list(key)

E TypeError: 'cftime._cftime.DatetimeNoLeap' object is not iterable

../../../pandas/pandas/core/series.py:911: TypeError ```

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is particularly useful for removing microsecond noise that can sometimes be added from decoding times via cftime.num2date, though also applies more generally. The methods used here for rounding dates in the integer domain are copied from pandas.

On a somewhat more internal note, this adds an asi8 property to CFTimeIndex, which encodes the dates as integer values representing microseconds since 1970-01-01; this encoding is made exact via the exact_cftime_datetime_difference function. It's possible this could be useful in other contexts.

Some examples:

``` In [1]: import xarray as xr

In [2]: times = xr.cftime_range("2000", periods=5, freq="17D")

In [3]: time = xr.DataArray(times, dims=["time"], name="time")

In [4]: time.dt.floor("11D") Out[4]: <xarray.DataArray 'floor' (time: 5)> array([cftime.DatetimeGregorian(1999-12-31 00:00:00), cftime.DatetimeGregorian(2000-01-11 00:00:00), cftime.DatetimeGregorian(2000-02-02 00:00:00), cftime.DatetimeGregorian(2000-02-13 00:00:00), cftime.DatetimeGregorian(2000-03-06 00:00:00)], dtype=object) Coordinates: * time (time) object 2000-01-01 00:00:00 ... 2000-03-09 00:00:00

In [5]: time.dt.ceil("11D") Out[5]: <xarray.DataArray 'ceil' (time: 5)> array([cftime.DatetimeGregorian(2000-01-11 00:00:00), cftime.DatetimeGregorian(2000-01-22 00:00:00), cftime.DatetimeGregorian(2000-02-13 00:00:00), cftime.DatetimeGregorian(2000-02-24 00:00:00), cftime.DatetimeGregorian(2000-03-17 00:00:00)], dtype=object) Coordinates: * time (time) object 2000-01-01 00:00:00 ... 2000-03-09 00:00:00

In [6]: time.dt.round("11D") Out[6]: <xarray.DataArray 'round' (time: 5)> array([cftime.DatetimeGregorian(1999-12-31 00:00:00), cftime.DatetimeGregorian(2000-01-22 00:00:00), cftime.DatetimeGregorian(2000-02-02 00:00:00), cftime.DatetimeGregorian(2000-02-24 00:00:00), cftime.DatetimeGregorian(2000-03-06 00:00:00)], dtype=object) Coordinates: * time (time) object 2000-01-01 00:00:00 ... 2000-03-09 00:00:00 ```

@max-sixty @shoyer -- I agree we've let these linger far too long. This should hopefully get things back to being green.

• [x] Passes isort -rc . && black . && mypy . && flake8

MCVE Code Sample

``` In [1]: import xarray as xr

In [2]: times = xr.cftime_range('0001', periods=3, freq='500Y')

In [3]: da = xr.DataArray(range(3), dims=['time'], coords=[times])

In [4]: da.interp(time=['0002-05-01'])

TypeError Traceback (most recent call last) <ipython-input-4-f781cb4d500e> in <module> ----> 1 da.interp(time=['0002-05-01'])

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/dataarray.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 1353 kwargs=kwargs, 1354 assume_sorted=assume_sorted, -> 1355 coords_kwargs, 1356 ) 1357 return self._from_temp_dataset(ds)

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/dataset.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 2565 if k in var.dims 2566 } -> 2567 variables[name] = missing.interp(var, var_indexers, method, kwargs) 2568 elif all(d not in indexers for d in var.dims): 2569 # keep unrelated object array

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/missing.py in interp(var, indexes_coords, method, *kwargs) 607 new_dims = broadcast_dims + list(destination[0].dims) 608 interped = interp_func( --> 609 var.transpose(original_dims).data, x, destination, method, kwargs 610 ) 611

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/missing.py in interp_func(var, x, new_x, method, kwargs) 683 ) 684 --> 685 return _interpnd(var, x, new_x, func, kwargs) 686 687

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/missing.py in _interpnd(var, x, new_x, func, kwargs) 698 699 def _interpnd(var, x, new_x, func, kwargs): --> 700 x, new_x = _floatize_x(x, new_x) 701 702 if len(x) == 1:

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/missing.py in _floatize_x(x, new_x) 556 # represented by float. 557 xmin = x[i].values.min() --> 558 x[i] = x[i]._to_numeric(offset=xmin, dtype=np.float64) 559 new_x[i] = new_x[i]._to_numeric(offset=xmin, dtype=np.float64) 560 return x, new_x

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/variable.py in _to_numeric(self, offset, datetime_unit, dtype) 2001 """ 2002 numeric_array = duck_array_ops.datetime_to_numeric( -> 2003 self.data, offset, datetime_unit, dtype 2004 ) 2005 return type(self)(self.dims, numeric_array, self._attrs)

~/Software/miniconda3/envs/xarray-tests/lib/python3.7/site-packages/xarray/core/duck_array_ops.py in datetime_to_numeric(array, offset, datetime_unit, dtype) 410 if array.dtype.kind in "mM": 411 return np.where(isnull(array), np.nan, array.astype(dtype)) --> 412 return array.astype(dtype) 413 414

TypeError: float() argument must be a string or a number, not 'datetime.timedelta' ```

Problem Description

In principle we should be able to get this to work. The issue stems from the following logic in datetime_to_numeric: https://github.com/pydata/xarray/blob/45fd0e63f43cf313b022a33aeec7f0f982e1908b/xarray/core/duck_array_ops.py#L402-L404 Here we are relying on pandas to convert an array of datetime.timedelta objects to an array with dtype timedelta64[ns]. If the array of datetime.timedelta objects cannot be safely converted to timedelta64[ns] (e.g. due to an integer overflow) then this line is silently a no-op which leads to the error downstream at the dtype conversion step. This is my fault originally for suggesting this approach, https://github.com/pydata/xarray/pull/2668#discussion_r247271576.

~~To solve this I think we'll need to write our own logic to convert datetime.timedelta objects to numeric values instead of relying on pandas/NumPy.~~ (as @huard notes we should be able to use NumPy directly here for the conversion). We should not consider ourselves beholden to using nanosecond resolution for a couple of reasons: 1. datetime.timedelta objects do not natively support nanosecond resolution; they have microsecond resolution natively, which corresponds with a NumPy timedelta range of +/- 2.9e5 years. 2. One motivation/use-case for cftime dates is that they can represent long time periods that cannot be represented using a standard DatetimeIndex. We should do everything we can to support this with a CFTimeIndex.

@huard @dcherian this is an important issue we'll need to solve to be able to use a fixed offset for cftime dates for an application like polyfit/polyval.

xref: #3349 and #3631.

Output of xr.show_versions()

This uses encoding['dtype'] over data.dtype when possible within CFMaskCoder.encode to decide what type to cast encoding['missing_value'] or encoding['_FillValue'] to; this is one way to fix #3624. Another possible way would be to ensure the times have the proper dtype coming from CFDatetimeCoder.encode. I'm not sure what is the preferred solution.

cc: @andersy005, @spencerahill

• [x] Closes #3624
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

MCVE Code Sample

``` In [1]: import numpy as np; import pandas as pd; import xarray as xr

In [2]: times = pd.date_range('2000', periods=3)

In [3]: da = xr.DataArray(times, dims=['a'], coords=[[1, 2, 3]], name='foo')

In [4]: da.encoding['_FillValue'] = 1.0e20

In [5]: da.encoding['dtype'] = np.dtype('float64')

In [6]: da.to_dataset().to_netcdf('test.nc')

OverflowError Traceback (most recent call last) <ipython-input-6-cbc6b2cfdf9a> in <module> ----> 1 da.to_dataset().to_netcdf('test.nc')

~/Software/xarray/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf) 1548 unlimited_dims=unlimited_dims, 1549 compute=compute, -> 1550 invalid_netcdf=invalid_netcdf, 1551 ) 1552

~/Software/xarray/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf) 1071 # to be parallelized with dask 1072 dump_to_store( -> 1073 dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims 1074 ) 1075 if autoclose:

~/Software/xarray/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims) 1117 variables, attrs = encoder(variables, attrs) 1118 -> 1119 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims) 1120 1121

~/Software/xarray/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims) 291 writer = ArrayWriter() 292 --> 293 variables, attributes = self.encode(variables, attributes) 294 295 self.set_attributes(attributes)

~/Software/xarray/xarray/backends/common.py in encode(self, variables, attributes) 380 # All NetCDF files get CF encoded by default, without this attempting 381 # to write times, for example, would fail. --> 382 variables, attributes = cf_encoder(variables, attributes) 383 variables = {k: self.encode_variable(v) for k, v in variables.items()} 384 attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}

~/Software/xarray/xarray/conventions.py in cf_encoder(variables, attributes) 758 _update_bounds_encoding(variables) 759 --> 760 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()} 761 762 # Remove attrs from bounds variables (issue #2921)

~/Software/xarray/xarray/conventions.py in <dictcomp>(.0) 758 _update_bounds_encoding(variables) 759 --> 760 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()} 761 762 # Remove attrs from bounds variables (issue #2921)

~/Software/xarray/xarray/conventions.py in encode_cf_variable(var, needs_copy, name) 248 variables.UnsignedIntegerCoder(), 249 ]: --> 250 var = coder.encode(var, name=name) 251 252 # TODO(shoyer): convert all of these to use coders, too:

~/Software/xarray/xarray/coding/variables.py in encode(self, variable, name) 163 if fv is not None: 164 # Ensure _FillValue is cast to same dtype as data's --> 165 encoding["_FillValue"] = data.dtype.type(fv) 166 fill_value = pop_to(encoding, attrs, "_FillValue", name=name) 167 if not pd.isnull(fill_value):

OverflowError: Python int too large to convert to C long ```

Expected Output

I think this should succeed in writing to a netCDF file (it worked in xarray 0.14.0 and earlier).

Problem Description

I think this (admittedly very subtle) issue was introduced in https://github.com/pydata/xarray/pull/3502. Essentially at the time data enters CFMaskCoder.encode it does not necessarily have the dtype it will ultimately be encoded with. In the case of this example, data has type int64, but when it will be stored in the netCDF file it will be a double-precision float.

A possible solution here might be to rely on encoding['dtype'] (if it exists) to determine the type to cast the encoding values for '_FillValue' and 'missing_value' to, instead of relying solely on data.dtype (maybe use that as a fallback).

cc: @spencerahill

Output of xr.show_versions()

This fixes test_cf_datetime_nan for upcoming releases of pandas. See failure class (2) reported in #3673.

• [x] Tests added
• [x] Passes black . && mypy . && flake8

It looks like https://github.com/dask/dask/pull/5684, which adds nanmedian to dask (nice!), caused the error message to change for when one tries to reduce an array over all axes via median (i.e. it no longer contains 'dask', because xarray now dispatches to the newly added dask function instead of failing before trying that).

@dcherian do you have thoughts on how to best address this? Should we just remove that check in test_reduce? ``` =================================== FAILURES =================================== _____ TestVariable.test_reduce _______

error = <class 'NotImplementedError'>, pattern = 'dask'

@contextmanager
def raises_regex(error, pattern):
    __tracebackhide__ = True
    with pytest.raises(error) as excinfo:

      yield

xarray/tests/init.py:104:

self = <xarray.tests.test_dask.TestVariable object at 0x7fd14f8e9c88>

def test_reduce(self):
    u = self.eager_var
    v = self.lazy_var
    self.assertLazyAndAllClose(u.mean(), v.mean())
    self.assertLazyAndAllClose(u.std(), v.std())
    with raise_if_dask_computes():
        actual = v.argmax(dim="x")
    self.assertLazyAndAllClose(u.argmax(dim="x"), actual)
    with raise_if_dask_computes():
        actual = v.argmin(dim="x")
    self.assertLazyAndAllClose(u.argmin(dim="x"), actual)
    self.assertLazyAndAllClose((u > 1).any(), (v > 1).any())
    self.assertLazyAndAllClose((u < 1).all("x"), (v < 1).all("x"))
    with raises_regex(NotImplementedError, "dask"):

      v.median()

xarray/tests/test_dask.py:220:

self = <xarray.Variable (x: 4, y: 6)> dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray> dim = None, axis = None, skipna = None, kwargs = {}

def wrapped_func(self, dim=None, axis=None, skipna=None, **kwargs):

  return self.reduce(func, dim, axis, skipna=skipna, **kwargs)

xarray/core/common.py:46:

self = <xarray.Variable (x: 4, y: 6)> dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray> func = <function _create_nan_agg_method.\<locals>.f at 0x7fd16c228378> dim = None, axis = None, keep_attrs = None, keepdims = False, allow_lazy = True kwargs = {'skipna': None} input_data = dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>

def reduce(
    self,
    func,
    dim=None,
    axis=None,
    keep_attrs=None,
    keepdims=False,
    allow_lazy=None,
    **kwargs,
):
    """Reduce this array by applying `func` along some dimension(s).

    Parameters
    ----------
    func : function
        Function which can be called in the form
        `func(x, axis=axis, **kwargs)` to return the result of reducing an
        np.ndarray over an integer valued axis.
    dim : str or sequence of str, optional
        Dimension(s) over which to apply `func`.
    axis : int or sequence of int, optional
        Axis(es) over which to apply `func`. Only one of the 'dim'
        and 'axis' arguments can be supplied. If neither are supplied, then
        the reduction is calculated over the flattened array (by calling
        `func(x)` without an axis argument).
    keep_attrs : bool, optional
        If True, the variable's attributes (`attrs`) will be copied from
        the original object to the new one.  If False (default), the new
        object will be returned without attributes.
    keepdims : bool, default False
        If True, the dimensions which are reduced are left in the result
        as dimensions of size one
    **kwargs : dict
        Additional keyword arguments passed on to `func`.

    Returns
    -------
    reduced : Array
        Array with summarized data and the indicated dimension(s)
        removed.
    """
    if dim == ...:
        dim = None
    if dim is not None and axis is not None:
        raise ValueError("cannot supply both 'axis' and 'dim' arguments")

    if dim is not None:
        axis = self.get_axis_num(dim)

    if allow_lazy is not None:
        warnings.warn(
            "allow_lazy is deprecated and will be removed in version 0.16.0. It is now True by default.",
            DeprecationWarning,
        )
    else:
        allow_lazy = True

    input_data = self.data if allow_lazy else self.values

    if axis is not None:
        data = func(input_data, axis=axis, **kwargs)
    else:

      data = func(input_data, **kwargs)

xarray/core/variable.py:1534:

values = dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray> axis = None, skipna = None, kwargs = {} func = <function nanmedian at 0x7fd16c226bf8>, nanname = 'nanmedian'

def f(values, axis=None, skipna=None, **kwargs):
    if kwargs.pop("out", None) is not None:
        raise TypeError(f"`out` is not valid for {name}")

    values = asarray(values)

    if coerce_strings and values.dtype.kind in "SU":
        values = values.astype(object)

    func = None
    if skipna or (skipna is None and values.dtype.kind in "cfO"):
        nanname = "nan" + name
        func = getattr(nanops, nanname)
    else:
        func = _dask_or_eager_func(name)

    try:

      return func(values, axis=axis, **kwargs)

xarray/core/duck_array_ops.py:307:

a = dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray> axis = None, out = None

def nanmedian(a, axis=None, out=None):

  return _dask_or_eager_func("nanmedian", eager_module=nputils)(a, axis=axis)

xarray/core/nanops.py:144:

args = (dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>,) kwargs = {'axis': None} dispatch_args = (dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>,) wrapped = <function nanmedian at 0x7fd1737bcea0>

def f(*args, **kwargs):
    if list_of_args:
        dispatch_args = args[0]
    else:
        dispatch_args = args[array_args]
    if any(isinstance(a, dask_array.Array) for a in dispatch_args):
        try:
            wrapped = getattr(dask_module, name)
        except AttributeError as e:
            raise AttributeError(f"{e}: requires dask >={requires_dask}")
    else:
        wrapped = getattr(eager_module, name)

  return wrapped(*args, **kwargs)

xarray/core/duck_array_ops.py:47:

a = dask.array<array, shape=(4, 6), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray> axis = None, keepdims = False, out = None

@derived_from(np)
def nanmedian(a, axis=None, keepdims=False, out=None):
    """
    This works by automatically chunking the reduced axes to a single chunk
    and then calling ``numpy.nanmedian`` function across the remaining dimensions
    """
    if axis is None:
        raise NotImplementedError(

          "The da.nanmedian function only works along an axis or a subset of axes.  "
            "The full algorithm is difficult to do in parallel"
        )

E NotImplementedError: The da.nanmedian function only works along an axis or a subset of axes. The full algorithm is difficult to do in parallel

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/dask/array/reductions.py:1299: NotImplementedError

During handling of the above exception, another exception occurred:

self = <xarray.tests.test_dask.TestVariable object at 0x7fd14f8e9c88>

def test_reduce(self):
    u = self.eager_var
    v = self.lazy_var
    self.assertLazyAndAllClose(u.mean(), v.mean())
    self.assertLazyAndAllClose(u.std(), v.std())
    with raise_if_dask_computes():
        actual = v.argmax(dim="x")
    self.assertLazyAndAllClose(u.argmax(dim="x"), actual)
    with raise_if_dask_computes():
        actual = v.argmin(dim="x")
    self.assertLazyAndAllClose(u.argmin(dim="x"), actual)
    self.assertLazyAndAllClose((u > 1).any(), (v > 1).any())
    self.assertLazyAndAllClose((u < 1).all("x"), (v < 1).all("x"))
    with raises_regex(NotImplementedError, "dask"):

      v.median()

xarray/tests/test_dask.py:220:

self = <contextlib._GeneratorContextManager object at 0x7fd14f8bcc50> type = <class 'NotImplementedError'> value = NotImplementedError('The da.nanmedian function only works along an axis or a subset of axes. The full algorithm is difficult to do in parallel') traceback = <traceback object at 0x7fd154597bc8>

def __exit__(self, type, value, traceback):
    if type is None:
        try:
            next(self.gen)
        except StopIteration:
            return False
        else:
            raise RuntimeError("generator didn't stop")
    else:
        if value is None:
            # Need to force instantiation so we can reliably
            # tell if we get the same exception back
            value = type()
        try:

          self.gen.throw(type, value, traceback)

E AssertionError: exception NotImplementedError('The da.nanmedian function only works along an axis or a subset of axes. The full algorithm is difficult to do in parallel') did not match pattern 'dask' ```

This is a possible fix for the issue @mathause described in https://github.com/pydata/xarray/issues/3535#issuecomment-554317768. @TomNicholas does this seem like a safe change to make in combine_by_coords?

• [x] Closes #3535
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Per https://github.com/pydata/xarray/pull/3431#discussion_r337620810, this removes outdated code leftover from the netcdftime -> cftime transition. Currently the minimum version of netCDF4 that xarray tests against is 1.4, which does not include netcdftime, and instead specifies cftime as a required dependency.

• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

The main issue is that we were resampling the same time indexes across a large range of frequencies, in some cases producing very long results, e.g. resampling an index that spans 27 years to a frequency of 12 hours.

This modifies the primary test so that it constructs time indexes whose ranges are based on the frequencies we resample to. Now in total the tests in test_cftimeindex_resample.py take around 6 seconds.

@jwenfai I did some coverage analysis offline, and these tests produce the same coverage that we had before (I found it necessary to be sure to test cases where the reference index had either a shorter or longer frequency than the resample frequency). Do you think what I have here is sufficient? I think we could potentially shorten things even more, but I'm not sure if it's worth the effort.

• [x] Closes #2874

See below for the new profiling results; now the longest cftime tests are no longer associated with resample.

$ pytest -k cftime --durations=50 ... 0.18s call xarray/tests/test_backends.py::TestScipyInMemoryData::test_roundtrip_cftime_datetime_data 0.11s call xarray/tests/test_backends.py::TestScipyFilePath::test_roundtrip_cftime_datetime_data 0.10s call xarray/tests/test_backends.py::TestNetCDF4Data::test_roundtrip_cftime_datetime_data 0.09s call xarray/tests/test_backends.py::TestNetCDF4ClassicViaNetCDF4Data::test_roundtrip_cftime_datetime_data 0.09s call xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_cftime_datetime_data 0.08s teardown xarray/tests/test_cftime_offsets.py::test_add_year_end_onOffset[julian-(2, 12)-()-<YearEnd: n=-1, month=12>-(1, 12)-()] 0.06s call xarray/tests/test_backends.py::TestNetCDF3ViaNetCDF4Data::test_roundtrip_cftime_datetime_data 0.06s call xarray/tests/test_backends.py::TestGenericNetCDFData::test_roundtrip_cftime_datetime_data 0.05s call xarray/tests/test_backends.py::TestScipyFileObject::test_roundtrip_cftime_datetime_data 0.04s call xarray/tests/test_conventions.py::TestCFEncodedDataStore::test_roundtrip_cftime_datetime_data 0.03s call xarray/tests/test_dataset.py::test_differentiate_cftime[True] 0.03s call xarray/tests/test_dataset.py::test_trapz_datetime[cftime-True] 0.02s call xarray/tests/test_coding_times.py::test_contains_cftime_datetimes_dask_3d[standard] 0.02s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_out_of_range[2500-gregorian] 0.02s call xarray/tests/test_dataset.py::test_differentiate_cftime[False] 0.02s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-right-None-4A-MAY] 0.02s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_in_range[gregorian] 0.02s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-None-right-11Q-JUN] 0.02s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_out_of_range[2500-proleptic_gregorian] 0.02s call xarray/tests/test_backends.py::test_use_cftime_true[1500-gregorian] 0.02s call xarray/tests/test_backends.py::test_use_cftime_true[2500-proleptic_gregorian] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2000-gregorian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-None-right-4A-MAY] 0.01s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_out_of_range[2500-standard] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[1500-julian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-left-right-4A-MAY] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-right-None-11Q-JUN] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-None-right-4A-MAY] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-left-None-7M] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-left-None-4A-MAY] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-left-right-4A-MAY] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-None-None-11Q-JUN] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-right-right-4A-MAY] 0.01s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_out_of_range[1500-proleptic_gregorian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-left-None-11Q-JUN] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2500-julian] 0.01s call xarray/tests/test_backends.py::test_use_cftime_standard_calendar_default_out_of_range[1500-gregorian] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[1500-proleptic_gregorian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-left-right-11Q-JUN] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2000-standard] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2500-standard] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-None-None-4A-MAY] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-right-right-11Q-JUN] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-right-right-7M] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2500-gregorian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-24-left-right-7M] 0.01s call xarray/tests/test_backends.py::test_use_cftime_true[2000-proleptic_gregorian] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-right-None-7M] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-None-right-7M] 0.01s call xarray/tests/test_cftimeindex_resample.py::test_resample[longer_da_freq-31-left-None-11Q-JUN]

Code Sample, a copy-pastable example if possible

In xarray 0.12.0 the following example produces a Dataset with no indexes: ``` In [1]: import xarray as xr

In [2]: da = xr.DataArray([1], [('x', [0])], name='a')

In [3]: da.to_dataset().isel(x=0).expand_dims('x').indexes Out[3]: ```

Expected Output

In xarray 0.11.3 the roundtrip sequence above properly recovers the initial index along the 'x' dimension: ``` In [1]: import xarray as xr

In [2]: da = xr.DataArray([1], [('x', [0])], name='a')

In [3]: da.to_dataset().isel(x=0).expand_dims('x').indexes Out[3]: x: Int64Index([0], dtype='int64', name='x') ```

Output of xr.show_versions()

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

For now I've held off on making these changes dask-compatible (I could do it, but I'm not sure it is worth the extra complexity).

Based on @shoyer's suggestion in https://github.com/pydata/xarray/issues/2754#issuecomment-461983092.

• [x] Closes #1263; Closes #2754
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] closes #1084
• [x] passes git diff upstream/master | flake8 --diff
• [x] tests added / passed
• [x] whatsnew entry

This work in progress PR is a start on implementing a NetCDFTimeIndex, a subclass of pandas.Index, which closely mimics pandas.DatetimeIndex, but uses netcdftime._netcdftime.datetime objects. Currently implemented in the new index are: - Partial datetime-string indexing (using strictly ISO8601-format strings, using a date parser implemented by @shoyer in https://github.com/pydata/xarray/issues/1084#issuecomment-274372547) - Field-accessors for year, month, day, hour, minute, second, and microsecond, to enable groupby operations on attributes of date objects

This index is meant as a step towards improving the handling of non-standard calendars and dates outside the range Timestamp('1677-09-21 00:12:43.145225') to Timestamp('2262-04-11 23:47:16.854775807').

For now I have pushed only the code and some tests for the new index; I want to make sure the index is solid and well-tested before we consider integrating it into any of xarray's existing logic or writing any documentation.

Regarding the index, there are a couple remaining outstanding issues (that at least I'm aware of):

1. Currently one can create non-sensical datetimes using netcdftime._netcdftime.datetime objects. This means one can attempt to index with an out-of-bounds string or datetime without raising an error. Could this possibly be addressed upstream? For example: ``` In [1]: from netcdftime import DatetimeNoLeap

In [2]: DatetimeNoLeap(2000, 45, 45) Out[2]: netcdftime._netcdftime.DatetimeNoLeap(2000, 45, 45, 0, 0, 0, 0, -1, 1) 2. I am looking to enable this index to be used in pandas.Series and pandas.DataFrame objects as well; this requires implementing aget_valuemethod. I have taken @shoyer's suggested simplified approach from https://github.com/pydata/xarray/issues/1084#issuecomment-275963433, and tweaked it to also allow for slice indexing, so I think this is most of the way there. A remaining to-do for me, however, is to implement something to allow for integer-indexing outside ofiloc, e.g. if you have a pandas.Seriesseries, indexing with the syntaxseries[1]orseries[1:3]```.

Hopefully this is a decent start; in particular I'm not an expert in writing tests so please let me know if there are improvements I can make to the structure and / or style I've used so far. I'm happy to make changes. I appreciate your help.

Thanks for the suggestion, @shoyer. - [x] Closes #2688 - [x] Tests added - [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

cc: @jwenfai

• [x] Closes #2761
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Code Sample, a copy-pastable example if possible

```python In [1]: import xarray

In [2]: xarray.cftime_range('2000', periods=3, calendar='standard').values Out[2]: array([cftime.DatetimeProlepticGregorian(2000, 1, 1, 0, 0, 0, 0, -1, 1), cftime.DatetimeProlepticGregorian(2000, 1, 2, 0, 0, 0, 0, -1, 1), cftime.DatetimeProlepticGregorian(2000, 1, 3, 0, 0, 0, 0, -1, 1)], dtype=object) ```

Problem description

When writing cftime_range I used dates from a proleptic Gregorian calendar when the calendar type was specified as 'standard'. While this is consistent with Python's built-in datetime.datetime (which uses a proleptic Gregorian calendar), this differs from the behavior in cftime.num2date and ultimately the CF conventions, which state that 'standard' should refer to the true Gregorian calendar. My inclination is that considering "cf" is in the name of cftime_range, we should adhere to those conventions as closely as possible (and hence the way I initially coded things was a mistake).

Expected Output

python In [2]: xarray.cftime_range('2000', periods=3, calendar='standard').values Out[2]: array([cftime.DatetimeGregorian(2000, 1, 1, 0, 0, 0, 0, -1, 1), cftime.DatetimeGregorian(2000, 1, 2, 0, 0, 0, 0, -1, 1), cftime.DatetimeGregorian(2000, 1, 3, 0, 0, 0, 0, -1, 1)], dtype=object)

Do others agree that we should fix this? If we were to make this change, would it be appropriate to consider it a bug and simply make the breaking change immediately, or might we need a deprecation cycle?

Code Sample, a copy-pastable example if possible

Currently something like the following raises an error: ``` In [1]: import xarray as xr

In [2]: import pandas as pd

In [3]: import numpy as np

In [4]: times = xr.cftime_range('2000', periods=3)

In [5]: series = pd.Series(np.array([0., np.nan, 1.]), index=times)

In [6]: series Out[6]: 2000-01-01 00:00:00 0.0 2000-01-02 00:00:00 NaN 2000-01-03 00:00:00 1.0 dtype: float64

In [7]: series.dropna()

TypeError Traceback (most recent call last) <ipython-input-7-45eb0c023203> in <module> ----> 1 series.dropna()

~/pandas/pandas/core/series.py in dropna(self, axis, inplace, **kwargs) 4169 4170 if self._can_hold_na: -> 4171 result = remove_na_arraylike(self) 4172 if inplace: 4173 self._update_inplace(result)

~/pandas/pandas/core/dtypes/missing.py in remove_na_arraylike(arr) 539 return arr[notna(arr)] 540 else: --> 541 return arr[notna(lib.values_from_object(arr))]

~/pandas/pandas/core/series.py in getitem(self, key) 801 key = com.apply_if_callable(key, self) 802 try: --> 803 result = self.index.get_value(self, key) 804 805 if not is_scalar(result):

~/xarray-dev/xarray/xarray/coding/cftimeindex.py in get_value(self, series, key) 321 """Adapted from pandas.tseries.index.DatetimeIndex.get_value""" 322 if not isinstance(key, slice): --> 323 return series.iloc[self.get_loc(key)] 324 else: 325 return series.iloc[self.slice_indexer(

~/xarray-dev/xarray/xarray/coding/cftimeindex.py in get_loc(self, key, method, tolerance) 300 else: 301 return pd.Index.get_loc(self, key, method=method, --> 302 tolerance=tolerance) 303 304 def _maybe_cast_slice_bound(self, label, side, kind):

~/pandas/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance) 2595 'backfill or nearest lookups') 2596 try: -> 2597 return self._engine.get_loc(key) 2598 except KeyError: 2599 return self._engine.get_loc(self._maybe_cast_indexer(key))

~/pandas/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

~/pandas/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

TypeError: '[ True False True]' is an invalid key ```

Problem description

We currently rely on this in the resampling logic within xarray for a Series indexed by a DatetimeIndex: https://github.com/pydata/xarray/blob/dc87dea52351835af472d131f70a7f7603b3100e/xarray/core/groupby.py#L268 It would be nice if we could do the same with a Series indexed by a CFTimeIndex, e.g. in #2593.

Expected Output

In [7]: series.dropna() Out[7]: 2000-01-01 00:00:00 0.0 2000-01-03 00:00:00 1.0 dtype: float64

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

``` In [1]: import xarray

In [2]: times = xarray.cftime_range('2000', periods=3)

In [3]: times - times[0]

TypeError Traceback (most recent call last) <ipython-input-3-97cbca76a8af> in <module> ----> 1 times - times[0]

~/xarray-dev/xarray/xarray/coding/cftimeindex.py in sub(self, other) 417 return CFTimeIndex(np.array(self) - other.to_pytimedelta()) 418 else: --> 419 return CFTimeIndex(np.array(self) - other) 420 421 def _add_delta(self, deltas):

~/xarray-dev/xarray/xarray/coding/cftimeindex.py in new(cls, data, name) 238 result = object.new(cls) 239 result._data = np.array(data, dtype='O') --> 240 assert_all_valid_date_type(result._data) 241 result.name = name 242 return result

~/xarray-dev/xarray/xarray/coding/cftimeindex.py in assert_all_valid_date_type(data) 194 raise TypeError( 195 'CFTimeIndex requires cftime.datetime ' --> 196 'objects. Got object of {}.'.format(date_type)) 197 if not all(isinstance(value, date_type) for value in data): 198 raise TypeError(

TypeError: CFTimeIndex requires cftime.datetime objects. Got object of <class 'datetime.timedelta'>. ```

Problem description

This should result in a pandas.TimedeltaIndex, as is the case for a pandas.DatetimeIndex: ``` In [4]: import pandas

In [5]: times = pandas.date_range('2000', periods=3)

In [6]: times - times[0] Out[6]: TimedeltaIndex(['0 days', '1 days', '2 days'], dtype='timedelta64[ns]', freq=None) ```

Expected Output

``` In [1]: import xarray

In [2]: times = xarray.cftime_range('2000', periods=3)

In [3]: times - times[0] Out[3]: TimedeltaIndex(['0 days', '1 days', '2 days'], dtype='timedelta64[ns]', freq=None) ```

Output of xr.show_versions()

• [x] Closes #2671
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2649
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I think this should be an appropriate fix for #2649, but I'd appreciate input from those who are more experienced dealing with timezones in NumPy/pandas.

My understanding is that NumPy dates are stored as UTC and do not carry any timezone information. Therefore converting the ref_date with tz_convert(None) here, which converts it to UTC and removes the timezone information, should be appropriate for encoding.

Currently, while we indeed do always decode to UTC, I'm not sure how well we test that. In addition this tests both the np.datetime64 and cftime.datetime pathways.

• [x] Closes #2197
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

My main goal with this was to help out with #2593 (xarray's built-in interpolation method is compatible with cftime coordinates, so this refactor would simplify things there). While doing this I realized that I could also add the simple bug-fix for #2197.

cc: @jwenfai

• [x] Tests added

It turns out there was a remaining bug in cftime (https://github.com/Unidata/cftime/issues/106) that impacted the results of the dayofwk and dayofyr attributes of cftime objects generated by their replace method, which we use when parsing dates from strings, and in some offset arithmetic.

A workaround is to add a dayofwk=-1 argument to each replace call where the dayofwk or dayofyr would be expected to change. I've fixed this bug upstream in cftime (https://github.com/Unidata/cftime/pull/108), but it will only be available in a future version. Would it be appropriate to use this workaround in xarray?

This would fix this doc page for instance:

• [x] Closes #2623
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is related to a bug fixed in https://github.com/pandas-dev/pandas/pull/18020#issuecomment-340477318 (this should return a TimedeltaIndex): ``` In [2]: times = pd.date_range('2000', periods=3)

In [3]: times - np.datetime64('2000-01-01') Out[3]: DatetimeIndex(['1970-01-01', '1970-01-02', '1970-01-03'], dtype='datetime64[ns]', freq='D') Subtracting a `Timestamp` object seems to work in all versions: In [4]: times - pd.Timestamp('2000-01-01') Out[4]: TimedeltaIndex(['0 days', '1 days', '2 days'], dtype='timedelta64[ns]', freq=None) ```

This was caught by @jwenfai in #2593. I hope no one was inadvertently trying to use this argument before.

Should this need a what's new entry?

As mentioned in https://github.com/pydata/xarray/issues/2597#issuecomment-446151329, the dayofyr and dayofwk attributes of cftime.datetime objects do not always work in versions of cftime prior to 1.0.2. This issue comes up in the latest doc build:

This updates the documentation environment to use the most recent version (1.0.3.4), which should fix things.

• [x] Closes #2597
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Minor follow-up to #2541

As discussed in #2437 and #2505, this sets the option enable_cftimeindex to True by default.

• [x] Fully documented, including whats-new.rst for all changes.

• [x] Closes #2521 (remove if there is no corresponding issue, which should only be the case for minor changes)

cftime version 1.0.2.1 (currently only installed on Windows, because it hasn't appeared on conda-forge yet) includes some changes that improve the precision of datetime arithmetic, which causes some results of infer_datetime_units to change. These changes aren't really a concern, because it doesn't impact our ability to round-trip dates; it just changes the units dates are encoded with in some cases. For that reason I've just deleted the tests where the answers change across versions.

• [x] Closes #2272
• [x] Tests added (for all bug fixes or enhancements)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Just a minor followup to #2509 to remove Dataset.T from the documentation.

• [x] Closes #2484
• [x] Tests added (for all bug fixes or enhancements)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

``` In [1]: import xarray as xr

In [2]: start_dates = xr.cftime_range('1999-12', periods=12, freq='M')

In [3]: end_dates = start_dates.shift(1, 'M')

In [4]: end_dates - start_dates

TypeError Traceback (most recent call last) <ipython-input-4-43c24409020b> in <module>() ----> 1 end_dates - start_dates

/Users/spencerclark/xarray-dev/xarray/xarray/coding/cftimeindex.pyc in sub(self, other) 365 366 def sub(self, other): --> 367 return CFTimeIndex(np.array(self) - other) 368 369

TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'CFTimeIndex'

```

Problem description

Subtracting one DatetimeIndex from another produces a TimedeltaIndex: ``` In [5]: import pandas as pd

In [6]: start_dates = pd.date_range('1999-12', periods=12, freq='M')

In [7]: end_dates = start_dates.shift(1, 'M')

In [8]: end_dates - start_dates Out[8]: TimedeltaIndex(['31 days', '29 days', '31 days', '30 days', '31 days', '30 days', '31 days', '31 days', '30 days', '31 days', '30 days', '31 days'], dtype='timedelta64[ns]', freq=None) ``` This should also be straightforward to enable for CFTimeIndexes and would be useful, for example, in the problem described in https://github.com/pydata/xarray/issues/2481#issue-369639339.

Expected Output

``` In [1]: import xarray as xr

In [2]: start_dates = xr.cftime_range('1999-12', periods=12, freq='M')

In [3]: end_dates = start_dates.shift(1, 'M')

In [4]: end_dates - start_dates Out[4]: TimedeltaIndex(['31 days', '29 days', '31 days', '30 days', '31 days', '30 days', '31 days', '31 days', '30 days', '31 days', '30 days', '31 days'], dtype='timedelta64[ns]', freq=None) ```

Per @shoyer's comment, https://github.com/pydata/xarray/pull/2431#discussion_r221976257, this cleans up _parse_array_of_cftime_strings, making it robust to multi-dimensional arrays in the process.

• [x] Closes #2244
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

With the latest version of pandas, checking the date_type of a CFTimeIndex produces a FutureWarning: ``` In [1]: import xarray as xr

In [2]: times = xr.cftime_range('2000', periods=5)

In [3]: times.date_type /Users/spencerclark/xarray-dev/xarray/xarray/coding/cftimeindex.py:161: FutureWarning: CFTimeIndex.data is deprecated and will be removed in a future version if self.data: Out[3]: cftime._cftime.DatetimeProlepticGregorian `` I think it was a typo to begin with to useself.dataincftimeindex.get_date_type(my mistake). Here I switch to usingself._data`, which is used elsewhere when internally referencing values of the index.

• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

As discussed in https://github.com/pydata/xarray/pull/2398#pullrequestreview-156804917, this enables the use of differentiate and interp on DataArrays/Datasets with cftime.datetime coordinates.

• [x] Closes #2142
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I took the approach first discussed here by @shoyer and followed pandas by creating simplified offset classes for use with cftime objects to implement a CFTimeIndex-enabled cftime_range function. I still may clean things up a bit and add a few more tests, but I wanted to post this in its current state to show some progress, as I think it is more or less working. I will try to ping folks when it is ready for a more detailed review.

Here are a few examples: ``` In [1]: import xarray as xr

In [2]: xr.cftime_range('2000-02-01', '2002-05-05', freq='3M', calendar='noleap') Out[2]: CFTimeIndex([2000-02-28 00:00:00, 2000-05-31 00:00:00, 2000-08-31 00:00:00, 2000-11-30 00:00:00, 2001-02-28 00:00:00, 2001-05-31 00:00:00, 2001-08-31 00:00:00, 2001-11-30 00:00:00, 2002-02-28 00:00:00], dtype='object')

In [3]: xr.cftime_range('2000-02-01', periods=4, freq='3A-JUN', calendar='noleap') Out[3]: CFTimeIndex([2000-06-30 00:00:00, 2003-06-30 00:00:00, 2006-06-30 00:00:00, 2009-06-30 00:00:00], dtype='object')

In [4]: xr.cftime_range(end='2000-02-01', periods=4, freq='3A-JUN') Out[4]: CFTimeIndex([1990-06-30 00:00:00, 1993-06-30 00:00:00, 1996-06-30 00:00:00, 1999-06-30 00:00:00], dtype='object') ```

Hopefully the offset classes defined here would also be useful for implementing things like resample for CFTimeIndex objects (#2191) and CFTimeIndex.shift (#2244).

• [x] Closes #2165
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

The issue is that both is_monotonic_decreasing and is_monotonic_increasing return True for a length-1 index; therefore an additional check is needed to make sure the length of the index is greater than 1 in CFTimeIndex._maybe_cast_slice_bound. This is similar to how things are done in DatetimeIndex._maybe_cast_slice_bound in pandas.

Code Sample, a copy-pastable example if possible

``` In [1]: import xarray as xr

In [2]: import cftime

In [3]: index = xr.CFTimeIndex([cftime.DatetimeNoLeap(1, 1, 1)])

In [4]: da = xr.DataArray([1], coords=[index], dims=['time'])

In [5]: da.sel(time=slice('0001', '0001')) Out[5]: <xarray.DataArray (time: 0)> array([], dtype=int64) Coordinates: * time (time) object ```

Problem description

When a CFTimeIndex is created with a single element, slicing using strings does not work; the example above should behave analogously to as it does when using a DatetimeIndex: ``` In [9]: import pandas as pd

In [10]: index = pd.DatetimeIndex(['2000-01-01'])

In [11]: da = xr.DataArray([1], coords=[index], dims=['time'])

In [12]: da.sel(time=slice('2000', '2000')) Out[12]: <xarray.DataArray (time: 1)> array([1]) Coordinates: * time (time) datetime64[ns] 2000-01-01 ``` I have a fix for this, which I will push shortly.

Expected Output

In [5]: da.sel(time=slice('0001', '0001')) Out[5]: <xarray.DataArray (time: 1)> array([1]) Coordinates: * time (time) object 0001-01-01 00:00:00

Output of xr.show_versions()

• [x] Closes #2127
• [x] Tests added
• [x] Tests passed

I can confirm the docs now build properly locally:

Code Sample, a copy-pastable example if possible

``` In [1]: from itertools import product

In [2]: import numpy as np

In [3]: import xarray as xr

In [4]: from cftime import DatetimeNoLeap

In [5]: dates = [DatetimeNoLeap(year, month, 1) for year, month in product(range ...: (1, 3), range(1, 13))]

In [6]: with xr.set_options(enable_cftimeindex=True): ...: da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo') ...:

In [7]: da.to_netcdf('test.nc')

TypeError Traceback (most recent call last) <ipython-input-7-306dbf0ba669> in <module>() ----> 1 da.to_netcdf('test.nc')

/Users/spencerclark/xarray-dev/xarray/xarray/core/dataarray.pyc in to_netcdf(self, args, kwargs) 1514 dataset = self.to_dataset() 1515 -> 1516 return dataset.to_netcdf(args, **kwargs) 1517 1518 def to_dict(self):

/Users/spencerclark/xarray-dev/xarray/xarray/core/dataset.pyc in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims) 1143 return to_netcdf(self, path, mode, format=format, group=group, 1144 engine=engine, encoding=encoding, -> 1145 unlimited_dims=unlimited_dims) 1146 1147 def to_zarr(self, store=None, mode='w-', synchronizer=None, group=None,

/Users/spencerclark/xarray-dev/xarray/xarray/backends/api.pyc in to_netcdf(dataset, path_or_file, mode, format, group, engine, writer, encoding, unlimited_dims) 681 try: 682 dataset.dump_to_store(store, sync=sync, encoding=encoding, --> 683 unlimited_dims=unlimited_dims) 684 if path_or_file is None: 685 return target.getvalue()

/Users/spencerclark/xarray-dev/xarray/xarray/core/dataset.pyc in dump_to_store(self, store, encoder, sync, encoding, unlimited_dims) 1073 1074 store.store(variables, attrs, check_encoding, -> 1075 unlimited_dims=unlimited_dims) 1076 if sync: 1077 store.sync()

/Users/spencerclark/xarray-dev/xarray/xarray/backends/common.pyc in store(self, variables, attributes, check_encoding_set, unlimited_dims) 356 """ 357 --> 358 variables, attributes = self.encode(variables, attributes) 359 360 self.set_attributes(attributes)

/Users/spencerclark/xarray-dev/xarray/xarray/backends/common.pyc in encode(self, variables, attributes) 441 # All NetCDF files get CF encoded by default, without this attempting 442 # to write times, for example, would fail. --> 443 variables, attributes = cf_encoder(variables, attributes) 444 variables = OrderedDict([(k, self.encode_variable(v)) 445 for k, v in variables.items()])

/Users/spencerclark/xarray-dev/xarray/xarray/conventions.pyc in cf_encoder(variables, attributes) 575 """ 576 new_vars = OrderedDict((k, encode_cf_variable(v, name=k)) --> 577 for k, v in iteritems(variables)) 578 return new_vars, attributes

python2/cyordereddict/_cyordereddict.pyx in cyordereddict._cyordereddict.OrderedDict.init (python2/cyordereddict/_cyordereddict.c:1225)()

//anaconda/envs/xarray-dev/lib/python2.7/_abcoll.pyc in update(args, *kwds) 569 self[key] = other[key] 570 else: --> 571 for key, value in other: 572 self[key] = value 573 for key, value in kwds.items():

/Users/spencerclark/xarray-dev/xarray/xarray/conventions.pyc in <genexpr>((k, v)) 575 """ 576 new_vars = OrderedDict((k, encode_cf_variable(v, name=k)) --> 577 for k, v in iteritems(variables)) 578 return new_vars, attributes

/Users/spencerclark/xarray-dev/xarray/xarray/conventions.pyc in encode_cf_variable(var, needs_copy, name) 232 variables.CFMaskCoder(), 233 variables.UnsignedIntegerCoder()]: --> 234 var = coder.encode(var, name=name) 235 236 # TODO(shoyer): convert all of these to use coders, too:

/Users/spencerclark/xarray-dev/xarray/xarray/coding/times.pyc in encode(self, variable, name) 384 data, 385 encoding.pop('units', None), --> 386 encoding.pop('calendar', None)) 387 safe_setitem(attrs, 'units', units, name=name) 388 safe_setitem(attrs, 'calendar', calendar, name=name)

/Users/spencerclark/xarray-dev/xarray/xarray/coding/times.pyc in encode_cf_datetime(dates, units, calendar) 338 339 if units is None: --> 340 units = infer_datetime_units(dates) 341 else: 342 units = _cleanup_netcdf_time_units(units)

/Users/spencerclark/xarray-dev/xarray/xarray/coding/times.pyc in infer_datetime_units(dates) 254 reference_date = dates[0] if len(dates) > 0 else '1970-01-01' 255 reference_date = format_cftime_datetime(reference_date) --> 256 unique_timedeltas = np.unique(np.diff(dates)).astype('timedelta64[ns]') 257 units = _infer_time_units_from_diff(unique_timedeltas) 258 return '%s since %s' % (units, reference_date)

TypeError: Cannot cast datetime.timedelta object from metadata [Y] to [ns] according to the rule 'same_kind' ```

Problem description

This seems to be an edge case that was not covered in the tests I added in #1252. Strangely if I cast the result of np.unique(np.diff(dates)) as an array before converting to 'timedelta64[ns]' objects things work: ``` In [9]: np.unique(np.diff(dates)).astype('timedelta64[ns]')

TypeError Traceback (most recent call last) <ipython-input-9-5d53452b676f> in <module>() ----> 1 np.unique(np.diff(dates)).astype('timedelta64[ns]')

TypeError: Cannot cast datetime.timedelta object from metadata [Y] to [ns] according to the rule 'same_kind'

In [10]: np.array(np.unique(np.diff(dates))).astype('timedelta64[ns]') Out[10]: array([2419200000000000, 2592000000000000, 2678400000000000], dtype='timedelta64[ns]') ``` Might anyone have any ideas as to what the underlying issue is? The fix could be as simple as that, but I don't understand why that makes a difference.

Expected Output

da.to_netcdf('test.nc') should succeed without an error.

Output of xr.show_versions()

cftime is now needed to build the documentation: http://xarray.pydata.org/en/latest/time-series.html#non-standard-calendars-and-dates-outside-the-timestamp-valid-range

Sorry I neglected this in #1252!

Addresses https://github.com/pydata/xarray/pull/1252#issuecomment-381131366

Perhaps I should have waited until cftime was up on conda-forge, but once that happens I can update this PR to use that in setting up the CI environments rather than pip.

I made updates to the installing and time series pages of the docs. Does this need a what's new entry? I'm not sure which heading I would classify it under.

I've been experimenting with distributed recently and have run into an issue when saving a result directly to a file using the netcdf4 engine. I've found if I compute things before saving to a file (thus loading the result into memory before calling to_netcdf) things work OK. I attached a minimum working example below.

Can others reproduce this? Part of me thinks there must be something wrong with my setup, because I'm somewhat surprised something like this wouldn't have come up already (apologies in advance if that's the case).

``` In [1]: import dask

In [2]: import distributed

In [3]: import netCDF4

In [4]: import xarray as xr

In [5]: dask.version Out[5]: '0.15.0'

In [6]: distributed.version Out[6]: '1.17.1'

In [7]: netCDF4.version Out[7]: '1.2.9'

In [8]: xr.version Out[8]: '0.9.6'

In [9]: da = xr.DataArray([1., 2., 3.])

In [10]: da.to_netcdf('no-dask.nc')

In [11]: da.chunk().to_netcdf('dask.nc') # Not using distributed yet

In [12]: c = distributed.Client() # Launch a LocalCluster (now using distributed)

In [13]: c Out[13]: <Client: scheduler='tcp://127.0.0.1:44576' processes=16 cores=16>

In [14]: da.chunk().to_netcdf('dask-distributed-netcdf4.nc', engine='netcdf4')

EOFError Traceback (most recent call last) <ipython-input-14-98490239a35f> in <module>() ----> 1 da.chunk().to_netcdf('dask-distributed-netcdf4.nc', engine='netcdf4')

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/core/dataarray.py in to_netcdf(self, args, kwargs) 1349 dataset = self.to_dataset() 1350 -> 1351 dataset.to_netcdf(args, **kwargs) 1352 1353 def to_dict(self):

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims) 975 return to_netcdf(self, path, mode, format=format, group=group, 976 engine=engine, encoding=encoding, --> 977 unlimited_dims=unlimited_dims) 978 979 def unicode(self):

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, writer, encoding, unlimited_dims) 571 try: 572 dataset.dump_to_store(store, sync=sync, encoding=encoding, --> 573 unlimited_dims=unlimited_dims) 574 if path_or_file is None: 575 return target.getvalue()

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/core/dataset.py in dump_to_store(self, store, encoder, sync, encoding, unlimited_dims) 916 unlimited_dims=unlimited_dims) 917 if sync: --> 918 store.sync() 919 920 def to_netcdf(self, path=None, mode='w', format=None, group=None,

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/backends/netCDF4_.py in sync(self) 334 def sync(self): 335 with self.ensure_open(autoclose=True): --> 336 super(NetCDF4DataStore, self).sync() 337 self.ds.sync() 338

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/backends/common.py in sync(self) 200 201 def sync(self): --> 202 self.writer.sync() 203 204 def store_dataset(self, dataset):

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/xarray/backends/common.py in sync(self) 177 import dask 178 if LooseVersion(dask.version) > LooseVersion('0.8.1'): --> 179 da.store(self.sources, self.targets, lock=GLOBAL_LOCK) 180 else: 181 da.store(self.sources, self.targets)

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/dask/array/core.py in store(sources, targets, lock, regions, compute, kwargs) 922 dsk = sharedict.merge((name, updates), *[src.dask for src in sources]) 923 if compute: --> 924 Array._get(dsk, keys, kwargs) 925 else: 926 from ..delayed import Delayed

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/dask/base.py in _get(cls, dsk, keys, get, kwargs) 102 get = get or _globals['get'] or cls._default_get 103 dsk2 = optimization_function(cls)(ensure_dict(dsk), keys, kwargs) --> 104 return get(dsk2, keys, **kwargs) 105 106 @classmethod

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, **kwargs) 1762 if sync: 1763 try: -> 1764 results = self.gather(packed) 1765 finally: 1766 for f in futures.values():

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct) 1261 else: 1262 return self.sync(self._gather, futures, errors=errors, -> 1263 direct=direct) 1264 1265 @gen.coroutine

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/client.py in sync(self, func, args, kwargs) 487 return future 488 else: --> 489 return sync(self.loop, func, args, **kwargs) 490 491 def str(self):

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, args, kwargs) 232 e.wait(1000000) 233 if error[0]: --> 234 six.reraise(error[0]) 235 else: 236 return result[0]

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 684 if value.traceback is not tb: 685 raise value.with_traceback(tb) --> 686 raise value 687 688 else:

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/utils.py in f() 221 raise RuntimeError("sync() called from thread of running loop") 222 yield gen.moment --> 223 result[0] = yield make_coro() 224 except Exception as exc: 225 logger.exception(exc)

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/tornado/gen.py in run(self) 1013 1014 try: -> 1015 value = future.result() 1016 except Exception: 1017 self.had_exception = True

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/tornado/concurrent.py in result(self, timeout) 235 return self._result 236 if self._exc_info is not None: --> 237 raise_exc_info(self._exc_info) 238 self._check_done() 239 return self._result

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/tornado/util.py in raise_exc_info(exc_info)

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/tornado/gen.py in run(self) 1019 1020 if exc_info is not None: -> 1021 yielded = self.gen.throw(*exc_info) 1022 exc_info = None 1023 else:

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct) 1154 six.reraise(type(exception), 1155 exception, -> 1156 traceback) 1157 if errors == 'skip': 1158 bad_keys.add(key)

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 683 value = tp() 684 if value.traceback is not tb: --> 685 raise value.with_traceback(tb) 686 raise value 687

/nbhome/skc/miniconda3/envs/research/lib/python3.6/site-packages/distributed/protocol/pickle.py in loads() 57 def loads(x): 58 try: ---> 59 return pickle.loads(x) 60 except Exception: 61 logger.info("Failed to deserialize %s", x[:10000], exc_info=True)

EOFError: Ran out of input ```

If I load the data into memory first by invoking compute() things work OK:

In [15]: da.chunk().compute().to_netcdf('dask-distributed-netcdf4.nc', engine='netcdf4')

@jhamman I'm sorry I missed this in #1920. The time decoding tests in the temporary Travis build with the new netcdftime library are all skipped because they are tagged with @requires_netCDF4 decorators rather than @requires_netcdftime ones. This PR fixes that.

In a local environment (after swapping these decorators) with the new netcdftime, I'm actually getting a failure, so there may be a bug we need to sort out upstream.

• [x] closes #1127
• [x] tests added / passed
• [x] passes git diff upstream/master | flake8 --diff
• [ ] whatsnew entry

@shoyer here I'm reusing existing requirements files. Is this along the lines of what you were looking for in #1127?

I think this should solve the AppVeyor test failures in #1252, as it should install version 1.2.7 of netCDF4, rather than version 1.2.4.

I recently tried to apply the argmin function on a dataset that I opened using xr.open_mfdataset and encountered an unexpected error. Applying argmin on the same dataset opened using xr.open_dataset works fine. Below is an example with some toy data. Could this be a bug, or is there something I'm doing wrong? I appreciate your help.

``` ipython In [1]: import xarray as xr In [2]: import numpy as np In [3]: xr.DataArray(np.random.rand(2, 3, 4), coords=[np.arange(2), np.arange(3), np.arange(4)], dims=['x', 'y', 'z']).to_dataset(name='test').to_netcdf('test_mfdataset.nc') In [4]: xr.open_dataset('test_mfdataset.nc').test.argmin('x').values Out[4]: array([[1, 1, 1, 1], [1, 0, 1, 0], [1, 1, 0, 1]])

In [5]: xr.open_mfdataset('test_mfdataset.nc').test.argmin('x').values

IndexError Traceback (most recent call last) <ipython-input-8-ccac9ca40874> in <module>() ----> 1 xr.open_mfdataset('test_mfdataset.nc').test.argmin('x').values

//anaconda/lib/python2.7/site-packages/xarray/core/dataarray.py in values(self) 353 def values(self): 354 """The array's data as a numpy.ndarray""" --> 355 return self.variable.values 356 357 @values.setter

//anaconda/lib/python2.7/site-packages/xarray/core/variable.py in values(self) 286 def values(self): 287 """The variable's data as a numpy.ndarray""" --> 288 return _as_array_or_item(self._data_cached()) 289 290 @values.setter

//anaconda/lib/python2.7/site-packages/xarray/core/variable.py in _data_cached(self) 252 def _data_cached(self): 253 if not isinstance(self._data, (np.ndarray, PandasIndexAdapter)): --> 254 self._data = np.asarray(self._data) 255 return self._data 256

//anaconda/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order) 472 473 """ --> 474 return array(a, dtype, copy=False, order=order) 475 476 def asanyarray(a, dtype=None, order=None):

//anaconda/lib/python2.7/site-packages/dask/array/core.py in array(self, dtype, kwargs) 852 853 def array(self, dtype=None, kwargs): --> 854 x = self.compute() 855 if dtype and x.dtype != dtype: 856 x = x.astype(dtype)

//anaconda/lib/python2.7/site-packages/dask/base.py in compute(self, kwargs) 35 36 def compute(self, kwargs): ---> 37 return compute(self, **kwargs)[0] 38 39 @classmethod

//anaconda/lib/python2.7/site-packages/dask/base.py in compute(args, kwargs) 108 for opt, val in groups.items()]) 109 keys = [var._keys() for var in variables] --> 110 results = get(dsk, keys, *kwargs) 111 112 results_iter = iter(results)

//anaconda/lib/python2.7/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, kwargs) 55 results = get_async(pool.apply_async, len(pool._pool), dsk, result, 56 cache=cache, queue=queue, get_id=_thread_get_id, ---> 57 kwargs) 58 59 return results

//anaconda/lib/python2.7/site-packages/dask/async.py in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs) 479 _execute_task(task, data) # Re-execute locally 480 else: --> 481 raise(remote_exception(res, tb)) 482 state['cache'][key] = res 483 finish_task(dsk, key, state, results, keyorder.get)

IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (3,4) (4,1) (1,3)

Traceback

File "//anaconda/lib/python2.7/site-packages/dask/async.py", line 264, in execute_task result = _execute_task(task, data) File "//anaconda/lib/python2.7/site-packages/dask/async.py", line 246, in _execute_task return func(*args2) File "//anaconda/lib/python2.7/site-packages/toolz/functoolz.py", line 381, in call ret = f(ret) File "//anaconda/lib/python2.7/site-packages/dask/array/reductions.py", line 450, in arg_agg return _arg_combine(data, axis, argfunc)[0] File "//anaconda/lib/python2.7/site-packages/dask/array/reductions.py", line 416, in _arg_combine arg = (arg + offsets)[tuple(inds)] ```