html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/121#issuecomment-42893119,https://api.github.com/repos/pydata/xarray/issues/121,42893119,MDEyOklzc3VlQ29tbWVudDQyODkzMTE5,1217238,2014-05-12T21:51:14Z,2014-05-12T21:51:33Z,MEMBER,"Those precision issues are unfortunate but perhaps unavoidable in this case because you are representing dates as floating point numbers -- the units are in ""days"" but the frequency between time points is measured in ""hours"".
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,33272937
https://github.com/pydata/xarray/issues/121#issuecomment-42890839,https://api.github.com/repos/pydata/xarray/issues/121,42890839,MDEyOklzc3VlQ29tbWVudDQyODkwODM5,2443309,2014-05-12T21:28:44Z,2014-05-12T21:28:44Z,MEMBER,"Ok, I just spent a few minutes working through a possible (although not ideal) solution for this. It works although it is a bit ugly and quite a bit slower than the standard calendar option. This option returns a `numpy` array with a dtype: `datetime64[ns]`. The downside is that it uses a `datetime` object as an intermediary.
``` python
In [1]:
import pandas as pd
from netCDF4 import num2date, date2num
import datetime
import numpy as np
from xray.conventions import decode_cf_datetime as decode
units = 'days since 0001-01-01'
# pandas time range
times = pd.date_range('2001-01-01-00', end='2001-06-30-23', freq='H')
# numpy array of numeric dates on noleap calendar
noleap_time = date2num(times.to_pydatetime(), units, calendar='noleap')
# numpy array of numeric dates on standard calendar
std_time = date2num(times.to_pydatetime(), units, calendar='standard')
# decoding function using datetime intermediary
def nctime_to_nptime(times):
new = np.empty(len(times), dtype='M8[ns]')
for i, t in enumerate(times):
new[i] = np.datetime64(datetime.datetime(*t.timetuple()[:6]))
return new
In [2]:
# decode noleap_time
%timeit nctime_to_nptime(decode(noleap_time, units, calendar='noleap'))
noleap_datetimes = nctime_to_nptime(num2date(noleap_time, units, calendar='noleap'))
print 'dtype:', noleap_datetimes.dtype
print noleap_datetimes
10 loops, best of 3: 38.8 ms per loop
dtype: datetime64[ns]
['2000-12-31T16:00:00.000000000-0800' '2000-12-31T17:00:00.000000000-0800'
'2000-12-31T18:00:00.000000000-0800' ...,
'2001-06-30T14:00:00.000000000-0700' '2001-06-30T15:00:00.000000000-0700'
'2001-06-30T16:00:00.000000000-0700']
In [3]:
# decode std_time using vectorized converter
%timeit decode(std_time, units, calendar='standard')
standard_datetimes = decode(std_time, units, calendar='standard')
print 'dtype:', standard_datetimes.dtype
print standard_datetimes
1000 loops, best of 3: 243 µs per loop
dtype: datetime64[ns]
['2000-12-31T16:00:00.000000000-0800' '2000-12-31T16:59:59.000000000-0800'
'2000-12-31T17:59:59.000000000-0800' ...,
'2001-06-30T13:59:59.000000000-0700' '2001-06-30T14:59:59.000000000-0700'
'2001-06-30T15:59:59.000000000-0700']
```
Two three things to notice here:
- the `decode_cf_datetime` is having a precision issue on the seconds when using the numpy timedelta approach (used by standard calendars).
- The new function is substantially slower than the numpy vectorization approach (can't get around that though)
- the `dtype` of the returned array matches regardless of the calendar
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,33272937
https://github.com/pydata/xarray/issues/121#issuecomment-42798808,https://api.github.com/repos/pydata/xarray/issues/121,42798808,MDEyOklzc3VlQ29tbWVudDQyNzk4ODA4,1217238,2014-05-12T06:09:09Z,2014-05-12T06:09:21Z,MEMBER,"Timedelta operations are used in exactly one place in xray: speeding up decoding of dates from netCDF if a standard calendar is being used. Otherwise, that sort of stuff is left up to the user.
If dates with non-standard calendars can generally be most usefully expressed as a pandas.DatetimeIndex, then let's go ahead and default to decoding them into datetime64 arrays. The relevant function to modify is [here](https://github.com/xray-pydata/xray/blob/master/xray/conventions.py#L60) (see also [here](https://github.com/xray-pydata/xray/blob/master/xray/conventions.py#L249)) if you'd like to make a pull request!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,33272937
https://github.com/pydata/xarray/issues/121#issuecomment-42793512,https://api.github.com/repos/pydata/xarray/issues/121,42793512,MDEyOklzc3VlQ29tbWVudDQyNzkzNTEy,2443309,2014-05-12T03:25:35Z,2014-05-12T03:25:35Z,MEMBER,"I think the simplest option would be to develop a function to cast the `netCDF4.datetime` objects into an array of `numpy.datetime64` objects.
Does `xray` do a lot of timedelta operations? If not, I'd say its probably best to live with (for now) the discrepancies between the calendars. Just be sure that you always use the `netCDF4.date2num` when going back to ordinal times.
I've run into issues like this repeatedly and I think it would be really nice if the `datetime` module was calendar aware -- probably not going to happen. The closest thing I've found to this is the `netCDF4 netcdftime.py` which is written in pure python but could be expanded to handle the issues we're talking about.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,33272937
https://github.com/pydata/xarray/issues/121#issuecomment-42789906,https://api.github.com/repos/pydata/xarray/issues/121,42789906,MDEyOklzc3VlQ29tbWVudDQyNzg5OTA2,1217238,2014-05-12T01:31:11Z,2014-05-12T01:31:11Z,MEMBER,"Yes, this is certainly related to #118. Virtual variables work by using pandas.DatetimeIndex methods, but if you're not using a standard calendar, you end up with an object array of netCDF4.datetime objects instead of an array of numpy.datetime64 objects (which can be turned into a DatetimeIndex).
Unfortunately, we do need to be able to make a DatetimeIndex to be able to use its (very quick) calculations for properties like year. The alternative is to write our own implementation, which would likely mean far slower pure-python code. We could also write a function to cast an array into a DatetimeIndex from datetime objects, which I'm guessing would be your preferred solution, even though there are issues like the difference between dates, as DatetimeIndex objects and numpy's datetime64 always assume a standard gregorian calendar.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,33272937