issue_comments

When I look at current master, it seems this is already fixed

xarray 0.11.3:

https://github.com/pydata/xarray/blob/d58e0769c7d600d5c8c3c2fb52aa28856c4d3290/xarray/core/dataset.py#L167

master:

https://github.com/pydata/xarray/blob/9c0bbf744a5235b4187f87de49175e6776d813cb/xarray/core/dataset.py#L179

I'd like this one to be merged very much. Is there anything holding this back?

Also, it might be nice to update the documentation with info on this.

Thanks, @darothen ! However, I believe I shouldn't have to, but xarray should do the right thing automatically.

Maybe I want to use the dataset after saving for something else, which would mean I'd have to do something like d_.drop(['category', 'species']).to_netcdf(...), which I don't like too much ...

Sure :smile:

I first open a DataArray:

grid_eu_coarse_noborder = xr.open_dataarray('tno-macc3_eu-no-nld-no-border_area.nc')
grid_eu_coarse_noborder

<xarray.DataArray 'TNO-MACC3_EU' (species: 12, category: 15, lon: 720, lat: 672)>
[87091200 values with dtype=float64]
Coordinates:
  * lat       (lat) float64 30.03 30.09 30.16 30.22 30.28 30.34 30.41 30.47 ...
  * category  (category) object 'pow' 'res' 'inc' 'pei' 'exf' 'sol' 'tra1' ...
  * species   (species) object 'CH4' 'CO' 'NH3' 'NMVOC' 'NOX' 'BC_1' ...
  * lon       (lon) float64 -29.94 -29.81 -29.69 -29.56 -29.44 -29.31 -29.19 ...

from which I select using .loc[] and convert to Dataset:

grid_eu_coarse_noborder.loc['NOX', 'pow'].to_dataset()

<xarray.Dataset>
Dimensions:       (lat: 672, lon: 720)
Coordinates:
  * lat           (lat) float64 30.03 30.09 30.16 30.22 30.28 30.34 30.41 ...
    category      <U3 'pow'
    species       <U3 'NOX'
  * lon           (lon) float64 -29.94 -29.81 -29.69 -29.56 -29.44 -29.31 ...
Data variables:
    TNO-MACC3_EU  (lon, lat) float64 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ...

which then cannot be saved:

grid_eu_coarse_noborder.loc['NOX', 'pow'].to_dataset().to_netcdf('test.nc')


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/core/utils.py in __len__(self)
    395         try:
--> 396             return self.shape[0]
    397         except IndexError:

IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-8-fb260f3bec4b> in <module>()
----> 1 grid_eu_coarse_noborder.loc['NOX', 'pow'].to_dataset().to_netcdf('test.nc')

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims)
    951         return to_netcdf(self, path, mode, format=format, group=group,
    952                          engine=engine, encoding=encoding,
--> 953                          unlimited_dims=unlimited_dims)
    954 
    955     def __unicode__(self):

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/api.py in to_netcdf(dataset, path, mode, format, group, engine, writer, encoding, unlimited_dims)
    567     try:
    568         dataset.dump_to_store(store, sync=sync, encoding=encoding,
--> 569                               unlimited_dims=unlimited_dims)
    570         if isinstance(path, BytesIO):
    571             return path.getvalue()

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/core/dataset.py in dump_to_store(self, store, encoder, sync, encoding, unlimited_dims)
    891 
    892         store.store(variables, attrs, check_encoding,
--> 893                     unlimited_dims=unlimited_dims)
    894         if sync:
    895             store.sync()

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/common.py in store(self, variables, attributes, *args, **kwargs)
    233         cf_variables, cf_attrs = cf_encoder(variables, attributes)
    234         AbstractWritableDataStore.store(self, cf_variables, cf_attrs,
--> 235                                         *args, **kwargs)
    236 
    237

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, unlimited_dims)
    202         self.set_attributes(attributes)
    203         self.set_variables(variables, check_encoding_set,
--> 204                            unlimited_dims=unlimited_dims)
    205 
    206     def set_attributes(self, attributes):

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/common.py in set_variables(self, variables, check_encoding_set, unlimited_dims)
    214             check = vn in check_encoding_set
    215             target, source = self.prepare_variable(
--> 216                 name, v, check, unlimited_dims=unlimited_dims)
    217             self.writer.add(source, target)
    218

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/netCDF4_.py in prepare_variable(self, name, variable, check_encoding, unlimited_dims)
    272 
    273         if self.format == 'NETCDF4':
--> 274             variable, datatype = _nc4_values_and_dtype(variable)
    275         else:
    276             variable = encode_nc3_variable(variable)

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/backends/netCDF4_.py in _nc4_values_and_dtype(var)
     78     if var.dtype.kind == 'U':
     79         # this entire clause should not be necessary with netCDF4>=1.0.9
---> 80         if len(var) > 0:
     81             var = var.astype('O')
     82         dtype = str

/home/eb/software/LAMOSpy/0.2-intel-2016a-Python-3.5.1/lib/python3.5/site-packages/xarray-0.9.1-py3.5.egg/xarray/core/utils.py in __len__(self)
    396             return self.shape[0]
    397         except IndexError:
--> 398             raise TypeError('len() of unsized object')
    399 
    400

TypeError: len() of unsized object

Sounds good. However, maybe from_ncdataset would avoid possible name confusion given that there's a Dataset in both xarray and netCDF4? On the other hand, at least in this context one of the options wouldn't make sense ... What do you think?

sounds good to me, @sjpfenninger, but I don't know the internals ...

That's strange, I cannot reproduce the test failure on my system ... Any ideas?

I'm not really happy with my implementation; the coordinate/variable handling in lines 1870-1884 are far from elegant. But is there a nicer way to do this?

I wouldn't allow full control here, but maybe a keyword argument for choosing whether to take the "lower", "upper" or "mean" labels would be appropriate.

I don't see how 'mean' could work, as the coord might be of dtype str. But I'm implementing 'upper' and 'lower'. If the user wants something different, she can always just swap the coord manually.

Actually, the current implementation leads to 0.0 for arrays inside a Dataset which don't have dim as a dimension. I personally would find it more intuitive if those arrays would not be touched at all.

Do you agree, or do you prefer this 0.0 for unaffected arrays?

can you give a pointer to where this would need to be done in the code?

If you can give me a pointer where in the code this would have to be implemented, I could give it a try myself ...

Thanks!