issue_comments

I submitted an feature request over at numpy. I'm going to close this now.

@rabernat - thanks for the PR and keep them coming.

I am the least knowledgeable person here regarding numpy and pandas development. The issues should probably be opened by someone else.

Calendar support is numpy is conceivable, but it will pretty much require fixing numpy dtypes first so that they can be parametrized and extended by third parties in Python (this is on the roadmap). Right now the datetime64 type itself is pretty buggy, in large part because it's written in C code that nobody is maintaining.

.

For pandas, I think the bigger issue is that pandas only does datetime64 with ns resolution. Simply adding us support would go a long ways toward solving this. See here for some discussion on the pandas side: https://github.com/pydata/pandas/issues/7307

On Fri, Aug 14, 2015 at 9:17 AM, Joe Hamman notifications@github.com wrote:

Worth? Yes. Any hope to actually get it in there? No... I think I disagree. There is almost no chance anyone outside of the climate community is going to spend time on this but, if calendar support was added in a responsible way to numpy and pandas, I don't see why they wouldn't be interested. So it will need to come from the climate users community, which IMHO, is under represented in the dev community.

If you don't want open the issue, I will.

Reply to this email directly or view it on GitHub: https://github.com/xray/xray/pull/523#issuecomment-131166492

Worth? Yes. Any hope to actually get it in there? No...

I think I disagree. There is almost no chance anyone outside of the climate community is going to spend time on this but, if calendar support was added in a responsible way to numpy and pandas, I don't see why they wouldn't be interested. So it will need to come from the climate users community, which IMHO, is under represented in the dev community.

If you don't want open the issue, I will.

Regarding the non-standard calendar support, is it worth opening issues in numpy / pandas?

Ok, thanks for the thoughtful discussion. I understand why you both feel this shouldn't be implemented in xray. My one objection to the discussion is that I don't think that climatological time is such a "niche" issue--processing climate model output (much of which has no specific calendar date associated with it but still has seasonal cycles, etc.) seems like one of the most useful applications for xray.

I come at this from a very practical point of view. I need to use certain datasets (e.g. WOA13, POP model output) for my research and teaching. I want to teach xray in my fall physical oceanography class---it is perfect for teaching because it "just works" and allows students to quickly load data and do basic analysis easily. This PR request would allow me to do that. I understand the counter-arguments, but I don't know exactly how I should proceed if this can't be part of xray. The options seem to be:

1) Don't use the datasets 2) Don't use xray 3) Use xray without time support

Is there something else I'm missing?

@ocefpaf To be clear, by "strongly prefer to get this fix upstream" I mostly meant that I am reluctant to include this in xray.

I would like it to be straightforward for others to extend our reading capabilities for netcdfs by adding custom logic like this for their own equivalent of xray.open_dataset that builds on what we have in xray.

I would also strongly prefer to get this fix in netCDF4 upstream rather than in xray, if possible.

Here is another example of a dataset with a similar time encoding issue. This is a valid OpenDAP URL: http://data.nodc.noaa.gov/thredds/dodsC/woa/WOA13/DATAv2/temperature/netcdf/decav/1.00/woa13_decav_t16_01v2.nc

The time units are 'months since 0000-01-01 00:00:00'. This is the very popular World Ocean Atlas. Since it is a monthly climatology, the dates are referenced to year 0.

@ocefpaf Unfortunately no OPenDAP endpoint, but I put a sample file on our ftp server ftp://ftp.ldeo.columbia.edu/pub/rpa/pop_sample/b40.1850.track1.2deg.wcm.007.pop.h.0100-01.nc

@ocefpaf NCAR is one of the lead institutions in terms of the CF conventions. Yet the CESM POP model, also developed at NCAR, has this "year 0" issue! To me this suggests that any practical, real-world application will need to deal with special cases like this one.

I have a few general comments,
1. Can you point me to where in the CF Conventions or UDUNITS the valid time coordinate units defined? 2. We should think about whether or not this fix belongs in xray or netCDF4. I am of the opinion that if the CF Conventions do in fact support the units in question, we should apply this fix in the netCDF package. If they don't, I don't think we wan to support it here either. 3. @shoyer is out for a bit but he will almost certainly want to weigh in on this.

If we end up going this route, you'll want to add some unit tests in test_conventions.py.