home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 1419602897

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1419602897 I_kwDOAMm_X85UnWvR 7197 Unstable pandas causes CF datetime64 issues 1828519 closed 0     4 2022-10-23T02:30:39Z 2022-10-26T16:00:35Z 2022-10-26T16:00:35Z CONTRIBUTOR      

What happened?

The Satpy project has a CI environment that installs numpy, pandas, and xarray (and a couple other packages) from their unstable sources (nightly builds, github source, etc). In the last week or two this environment has started failing with various datetime64 issues. It all seems to be caused by some recent change in pandas, but I can't place exactly what the problem is nor the commit/PR that started it. It seems there are a couple datetime related PRs.

What did you expect to happen?

Datetime or datetime64 objects should be allowed to be in whatever units they need to be in (days or minutes or nanoseconds. It seems parts of xarray (or pandas) assume datetime64[ns] but a change in pandas is no longer doing this conversion automatically (from datetime64[X] to datetime64[ns].

Minimal Complete Verifiable Example

You should be able to take any environment with modern xarray and install dev pandas with:

python -m pip install --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ --trusted-host pypi.anaconda.org --no-deps --pre --upgrade pandas

Then run this snippet:

```Python import xarray as xr import numpy as np from xarray.coding.times import CFDatetimeCoder

a = xr.DataArray(np.arange(1.0), dims=("time",), coords={"time": [np.datetime64('2018-05-30T10:05:00')]})

CFDatetimeCoder().encode(a.coords["time"]) ```

I haven't been able to generate a higher-level MVCE yet, but I'm hoping this little snippet will make the issue obvious to someone familiar with xarray internals.

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```

----> 1 CFDatetimeCoder().encode(a.coords["time"])

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:676, in CFDatetimeCoder.encode(self, variable, name) 672 dims, data, attrs, encoding = unpack_for_encoding(variable) 673 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes( 674 variable 675 ): --> 676 (data, units, calendar) = encode_cf_datetime( 677 data, encoding.pop("units", None), encoding.pop("calendar", None) 678 ) 679 safe_setitem(attrs, "units", units, name=name) 680 safe_setitem(attrs, "calendar", calendar, name=name)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:612, in encode_cf_datetime(dates, units, calendar) 609 dates = np.asarray(dates) 611 if units is None: --> 612 units = infer_datetime_units(dates) 613 else: 614 units = _cleanup_netcdf_time_units(units)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:394, in infer_datetime_units(dates) 392 print("Formatting datetime object") 393 reference_date = dates[0] if len(dates) > 0 else "1970-01-01" --> 394 reference_date = format_cftime_datetime(reference_date) 395 unique_timedeltas = np.unique(np.diff(dates)) 396 units = _infer_time_units_from_diff(unique_timedeltas)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:405, in format_cftime_datetime(date) 400 def format_cftime_datetime(date): 401 """Converts a cftime.datetime object to a string with the format: 402 YYYY-MM-DD HH:MM:SS.UUUUUU 403 """ 404 return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:06d}".format( --> 405 date.year, 406 date.month, 407 date.day, 408 date.hour, 409 date.minute, 410 date.second, 411 date.microsecond, 412 )

AttributeError: 'numpy.datetime64' object has no attribute 'year' ```

Anything else we need to know?

No response

Environment

``` INSTALLED VERSIONS ------------------ commit: None python: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59) [GCC 10.3.0] python-bits: 64 OS: Linux OS-release: 5.19.0-76051900-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.10.0 pandas: 2.0.0.dev0+422.g6c46013c54 numpy: 1.23.4 scipy: 1.10.0.dev0+1848.f114d8b netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.0 h5py: 3.7.0 Nio: None zarr: 2.13.0a3.dev5 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.4dev cfgrib: None iris: None bottleneck: 1.3.5 dask: 2022.10.0+6.gc8dc3955 distributed: None matplotlib: 3.7.0.dev473+gc450aa7baf cartopy: 0.20.3 seaborn: None numbagg: None fsspec: 2022.5.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.3.0 pip: 22.2.2 conda: None pytest: 7.1.1 IPython: 8.2.0 sphinx: 5.0.0 ```
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7197/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 2 rows from issues_id in issues_labels
  • 4 rows from issue in issue_comments
Powered by Datasette · Queries took 0.983ms · About: xarray-datasette