issue_comments

That would be great @andrewpauling! I think this is the relevant code in xarray: https://github.com/pydata/xarray/blob/1155f5646e07100e4acda18db074b148f1213b5d/xarray/core/variable.py#L244-L250

I want to say arguably we could use the _possibly_convert_objects function on datetime64 and timedelta64 data as well; you'll see this goes through a pandas.Series to do the casting, which has built-in logic to check that the values can be represented with nanosecond precision. But it's up to you how you ultimately want to go about things.

I agree this casting behavior is a bit surprising. If we wanted to be a little more transparent, we could also warn when attempting to cast non-nanosecond-precision datetime64 data to nanosecond precision. I'm not sure what others think; I know pandas doesn't do this, but it could be friendlier for users.

Thanks @spencerkclark for the response, that makes sense. I have been able to get what I needed to work using cftime, it just seemed strange to me that xarray behaved as it did with datetime64.

I agree it would be nice if an error was raised when dates can't be represented, would this be difficult to implement? I have been hoping to contribute to some open-source projects so if it's not too complex I'd be happy to tackle it, and if you have any advice on where to start with this problem that would be great.

Thanks @andrewpauling -- I do think there's a bug here, but this issue happens to be more complicated than it might seem on the surface :).

Xarray standardizes around nanosecond precision for np.datetime64 dtypes, and casts any NumPy array of dtype datetime64 to nanosecond precision. This is mainly motivated by pandas -- pandas requires nanosecond precision -- which xarray relies on for time indexing and other time-related operations through things like pandas.DatetimeIndex or the pandas.Series.dt accessor. As you've noted this is unfortunate since it limits the supported time range for np.datetime64 types (see, e.g., discussion in https://github.com/pydata/xarray/issues/789).

Addressing this fully would be a challenge (we've discussed this at times in the past). It was concluded that for dates outside the representable range that cftime dates would be used, and that over time we would build up infrastructure to enable some of the nice things you can do with np.datetime64 types with cftime objects. The functionality now largely exists, and a nice benefit of doing this through cftime is that we also gain compatibility with non-standard calendar types, e.g. DatetimeNoLeap. I encourage you to try and take advantage of that, and please let us know if there is something missing that you would like to see implemented or improved!

This is a long way of saying, without a fair amount of work (i.e. addressing this issue upstream in pandas) xarray is unlikely to relax its approach for the precision of np.datetime64 dtypes, and will continue casting to nanosecond precision.

However, the fact that your example silently results in non-sensical times should be considered a bug; instead, following pandas, I would argue we should raise an error if the dates cannot be represented with nanosecond precision.