issues

• Allows conversion to and from: NetCDF4, scipy.io.netcdf and in memory storage.
• Added general test cases, and cases for specific backend stores.

Previously conflicting _FillValue and missing_value only raised an AssertionError, now it's more informative.

The right solution to this is to figure out how to successfully round trip endian-ness, but that seems to be a deeper issue inside netCDF4 (https://github.com/Unidata/netcdf4-python/issues/346)

Instead we force all data to little endian before netCDF4 write.

This makes it possible to read datasets that don't follow CF datetime conventions perfectly, such as the following example which (surprisingly) comes from NCEP/NCAR (you'd think they would follow CF!)

``` ds = xray.open_dataset('http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GEFS/Global_1p0deg_Ensemble/members/GEFS_Global_1p0deg_Ensemble_20150114_1200.grib2/GC') print ds['time'].encoding['units']

u'Hour since 2015-01-14T12:00:00Z' ```

This would allow you to have coordinates which are offsets from a time coordinates which comes in handy when dealing with forecast data where the 'time' coordinate might be the forecast run time and you then want a 'lead' coordinate which is an offset from the run time.

Restructured Backends to make CF conventions handling consistent.

Among other things this includes: - EncodedDataStores which can wrap other stores and allow for modular encoding/decoding. - Trivial indices ds['x'] = ('x', np.arange(10)) are no longer stored on disk and are only created when accessed. - AbstractDataStore API change. Shouldn't effect external users. - missing_value attributes now function like _FillValue

All current tests are passing (though it could use more new ones).

Removed the object oriented encoding/decoding scheme in favor of a model where encoding/decoding happens when a dataset is stored to/ loaded from a DataStore.

Conventions can now be enforced at the DataStore level by overwriting the Datastore.store() and Datastore.load() methods, or as an optional arg to Dataset.load_store, Dataset.dump_to_store.

Includes miscellaneous cleanup.

``` Data objects now have a swap-able backend store.

• Allows conversion to and from: NetCDF4, scipy.io.netcdf and in memory storage.
• Added general test cases, and cases for specific backend stores.
• Dataset.translate() can now optionally copy the object.
• Fixed most unit tests, test_translate_consistency still fails. ```

decode_cf_variable was still using da.data instead of da.values.

It now also works with DataArray as input.

same as #153, but without tests.

concat_over=True indicates that concat should concat over all variables that are not the same in the set of datasets that are to be concatenated.

Internally Datasets convert time coordinates to pandas.DatetimeIndex. The backend function convert_to_cf_variable will convert these datetimes back to CF style times, but the original units were not being preserved.

Fixed a bug in which writes would fail if Datasets contained 0d variables.

Also added the ability to open Datasets directly from NetCDF3 bytestrings.

Small bug fix, and a test.

This was causing an error in the following situation:

ds = xray.Dataset() ds['time'] = ('time', [np.datetime64('2001-05-01') for i in range(5)]) ds['variable'] = ('time', np.arange(5.)) ds.to_netcdf('test.nc') ds = xray.open_dataset('./test.nc') ss = ds.indexed(time=slice(0, 2)) ss.dumps()

Thanks @shoyer for the fix.

In an attempt to coerce all datetime arrays to nano second resolutoin utils.as_safe_array() was creating copies of any datetime64 array (via the astype method). This was causing unexpected behavior (bugs) for things such as concatenation over times. (see below).

``` import xray import pandas as pd

ds = xray.Dataset() ds['time'] = ('time', pd.date_range('2011-09-01', '2011-09-11')) times = [ds.indexed(time=[i]) for i in range(10)] ret = xray.Dataset.concat(times, 'time') print ret['time']

<xray.DataArray 'time' (time: 10)> array(['1970-01-02T07:04:40.718526408-0800', '1969-12-31T16:00:00.099966608-0800', '1969-12-31T16:00:00.041748384-0800', '1969-12-31T16:00:00.041748360-0800', '1969-12-31T16:00:00.041748336-0800', '1969-12-31T16:00:00.041748312-0800', '1969-12-31T16:00:00.041748288-0800', '1969-12-31T16:00:00.041748264-0800', '1969-12-31T16:00:00.041748240-0800', '1969-12-31T16:00:00.041748216-0800'], dtype='datetime64[ns]') Attributes: Empty ```

decode_cf_variable will not save encoding for any 'object' dtypes.

When encoding cf variables check if dtype is np.datetime64 as well as DatetimeIndex.

fixes akleeman/xray/issues/39

BUG: dataset_repr was failing on empty datasets.

ds = xray.open_dataset('http://motherlode.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p5deg/files/GFS_Global_0p5deg_20140303_0000.grib2', decode_cf=False) In [4]: ds['lat'].dtype Out[4]: dtype('O')

This makes serialization fail.

Both Issues https://github.com/akleeman/xray/pull/20 and https://github.com/akleeman/xray/pull/21 are dealing with similar conceptual issues. Namely sometimes the user may want fine control over how a dataset is stored (integer packing, time units and calendars ...). Taking time as an example, the current model interprets the units and calendar in order to create a DatetimeIndex, but then throws out those attributes so that if the dataset were re-serialized the units may not be preserved.

One proposed solution to this issue is to include a distinct set of encoding attributes that would hold things like 'scale_factor', and 'add_offset' allowing something like this

``` ds['time'] = ('time', pd.date_range('1999-01-05', periods=10)) ds['time'].encoding['units'] = 'days since 1989-08-19' ds.dump('netcdf.nc')

ncdump -h ... int time(time) ; time:units = "days since 1989-08-19" ; ... ```

The encoding attributes could also handle masking, scaling, compression etc ...