home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 844712857

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
844712857 MDU6SXNzdWU4NDQ3MTI4NTc= 5093 open_dataset uses cftime, not datetime64, when calendar attribute is "Gregorian" 9010180 closed 0     2 2021-03-30T15:12:09Z 2021-04-20T14:17:42Z 2021-04-18T10:17:08Z NONE      

What happened:

I used xarray.open_dataset to open a NetCDF file whose time coordinate had the calendar attribute set to Gregorian. All dates were within the Timestamp-valid range.

The resulting dataset represented the time co-ordinate as a cftime._cftime.DatetimeGregorian.

What you expected to happen:

I expected the dataset to represent the time co-ordinate as a datetime64[ns], as documented here and here.

Minimal Complete Verifiable Example:

```python import xarray as xr import numpy as np import pandas as pd

def print_time_type(dataset): print(dataset.time.dtype, type(dataset.time[0].item()))

da = xr.DataArray( data=[32, 16, 8], dims=["time"], coords=dict( time=pd.date_range("2014-09-06", periods=3), reference_time=pd.Timestamp("2014-09-05"), ), )

Create dataset and confirm type of time

ds1 = xr.Dataset({"myvar": da}) print_time_type(ds1) # prints "datetime64[ns]" <class 'int'>

Manually set time attributes to "Gregorian" rather

than default "proleptic_gregorian".

ds1.time.encoding["calendar"] = "Gregorian" ds1.reference_time.encoding["calendar"] = "Gregorian" ds1.to_netcdf("test-capitalized.nc")

ds2 = xr.open_dataset("test-capitalized.nc") print_time_type(ds2)

prints "object <class 'cftime._cftime.DatetimeGregorian'>"

Workaround: add "Gregorian" to list of standard calendars.

xr.coding.times._STANDARD_CALENDARS.add("Gregorian") ds3 = xr.open_dataset("test-capitalized.nc") print_time_type(ds3) # prints "datetime64[ns]" <class 'int'> ```

Anything else we need to know?:

The documentation for the use_cftime parameter of open_dataset says:

If None (default), attempt to decode times to np.datetime64[ns] objects; if this is not possible, decode times to cftime.datetime objects.

In practice, we are getting some cftime.datetimes even for times which are interpretable and representable as np.datetime64[ns]s. In particular, we have some NetCDF files in which the time variable has a calendar attribute with a value of Gregorian (with a capital ‘G’). CF conventions allow this:

When this standard defines string attributes that may take various prescribed values, the possible values are generally given in lower case. However, applications programs should not be sensitive to case in these attributes.

However, xarray regards Gregorian as a non-standard calendar and falls back to cftime.datetime. If (as in the example) Gregorian is added to xr.coding.times._STANDARD_CALENDARS, the times are read as np.datetime64[ns]s.

Suggested fix: in xarray.coding.times._decode_datetime_with_pandas, change ‘if calendar not in _STANDARD_CALENDARS:’ to ‘if calendar.lower() not in _STANDARD_CALENDARS:’.

Environment:

Output of <tt>xr.show_versions()</tt> INSTALLED VERSIONS ------------------ commit: None python: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.8.0-48-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.17.1.dev39+g45b4436b pandas: 1.2.3 numpy: 1.20.2 scipy: None netCDF4: 1.5.6 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20210108 pip: 21.0.1 conda: None pytest: None IPython: None sphinx: None
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/5093/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 0 rows from issues_id in issues_labels
  • 2 rows from issue in issue_comments
Powered by Datasette · Queries took 2.801ms · About: xarray-datasette