@jhamman sorry for making this thread longer than it should. But I don't think you disagree! You are just more optimist than I am :wink:

But you are right. We need a champion from the climate community. And if either of you open the issue I will be the second man on the hill.

Regarding the non-standard calendar support, is it worth opening issues in numpy / pandas?

Worth? Yes. Any hope to actually get it in there? No...

I want to teach xray in my fall physical oceanography class

@rabernat I hear you! I suffer from the same problem. But I think we should teach students how to defend themselves from bad practices and tools limitations. (We have both in there! Talking about non-standard calendar and accepted but non-recommended standards.)

3) Use xray without time support

Not ideal but that is probably the way to go. My guess is that you can load the data in xray, but you cannot get those dates into the pandas indexing machinery, right? That is not too bad since there are not many "dates operations" that you can do anyway. Most of the time we only need to translate the time information into a label for the figures. If that is the case you are OK with the tools we have now.

@ocefpaf To be clear, by "strongly prefer to get this fix upstream" I mostly meant that I am reluctant to include this in xray.

Great. We don't need another interpretation of the CF-standards out there!

See the WOA13 dataset above as an example of the problems this brings. The data has no calendar and use units months since. CF discourage the use of months for obvious reasons and no calendar translates to:

gregorian or standard Mixed Gregorian/Julian calendar as defined by Udunits.

Which one! Year zero is invalid in one and (kind of) accepted in the other!!

Long story short: the dataset is valid CF but it is unclear how to interpret the dates. No one cares because people using that dataset know that t16 means Autumn (Oh no wait!! I am in the South Hemisphere, so that means Spring :stuck_out_tongue_winking_eye:)

PS: Note that there is a none calendar too in the CF conventions that is more adequate for datasets like that.

Just so you see how messy this can be. The year 0000, because it does not exist, is used to store climatology in COARDS and CF. The coards package issued a warning but did the wrong thing:

python from coards import parse units = 'days since 0000-01-01 00:00:00' parse(0, units) coards/__init__.py:60: UserWarning: Shifted data 366 days to the future, since year zero does not exist. UserWarning) datetime.datetime(1, 1, 1, 0, 0)

If someone accidentally saves that date back with this object it will no longer be year 0 and other CDMs might choke by not recognizing it as climatology. cf_units does the right thing:

python import cf_units u = cf_units.Unit('days since 0000-01-01 00:00:00', calendar=cf_units.CALENDAR_NO_LEAP) ut = u.utime() ut.num2date(0) 0-01-01 00:00:00

But that object is pretty much useless for pandas and xray because of the non-standard calendar. The object is useful to annotate plots or to save the data metadata back the same way it was before.

I would also strongly prefer to get this fix in netCDF4 upstream rather than in xray, if possible.

I disagree. Maybe not even xray should "fix" this. There are two issues here:

1) Interpreting year 0 as 1; 2) Non-standard calendars.

Maybe (2) should be in np.datetime64 (and pandas), but even that is hard to become a reality due to its niche use.

(1) is an UDUNITS ~~aberration~~ interpretation that made into some Conventions (CF and COARDS) and does not belong in the netCDF4 package. Does xray wants to became a CF CDM? I sure hope not! I have iris for that but for everything else I use xray.

And if people really want to get those niche date specification from xray they can just read the raw time data and parse it using one of the several UDUNITS wrappers or convention compliance/checkers out there. Here is another one that should do the right thing here for that case: https://pypi.python.org/pypi/coards/1.0.5

@rabernat here is an example on how to read those dates using cf_units (big download for a small example :wink:):

http://nbviewer.ipython.org/gist/ocefpaf/d14bd8ad24f4e1a47b19

My point with this example is: netCDF4-python should not do that for you! And, of xray wants to go down that road, it must be done with some warnings and good documentation!

By accepting these rules (by rules I mean how UDUNITS interprets the dates and how that interpretation is part of the CF-convention) you are leaving the world of "read my data" and entering a world of "interpret my data."

cf_units interprets your data as CF-compliant and does what you expect from that Common Data Model.

@ocefpaf NCAR is one of the lead institutions in terms of the CF conventions. Yet the CESM POP model, also developed at NCAR, has this "year 0" issue! To me this suggests that any practical, real-world application will need to deal with special cases like this one.

I know :wink: that is why I use cf_units!

PS: do you have an OPenDAP endpoint for you data? I would like to make a few tests here.

Note that I am not questioning the validity of those dates neither its use. But they are out there and people need to deal with that ~~monstrosity~~ feature. If xray decides to support this it has to be implemented very carefully and with some warnings...

Can you point me to where in the CF Conventions or UDUNITS the valid time coordinate units defined?

This edge case is defined here: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#climatological-statistics

NB: People that follow the COARDS conventions will also need this. Any Udunits wrapper can deal with that for you:

``` python import cf_units u = cf_units.Unit('days since 0000-01-01 00:00:00', calendar=cf_units.CALENDAR_NO_LEAP) ut = u.utime()

Returns a fake datetime object (See http://scitools.org.uk/iris/docs/latest/iris/iris/unit.html#iris.unit.Unit.num2date)

ut.num2date(0) 0-01-01 00:00:00

Note that python datetime cannot take year = 0, but udunits did the "right" thing.

ut.date2num(datetime(1, 1, 1)) 365.0 ```

We should think about whether or not this fix belongs in xray or netCDF4. I am of the opinion that if the CF Conventions do in fact support the units in question, we should apply this fix in the netCDF package. If they don't, I don't think we wan to support it here either.

I strongly disagree with that. netCDF is not one CDM and should not follow the CF-conventions! I know that iris does follow CF very closely (annoyingly in fact) and xray "kind of" follow (which is great BTW). However, if such conventions were adopted in the netCDF package, how will we load non-CF files?