issue_comments

@ocefpaf where you able to make any progress on this?

Sorry but no. I will look at it again but I do remember seeing some extra conda install commands issued after the env was create, and that may lead to undesirable up-downgrade of packages. If possible, the RTD env should install everything in one go, with a single conda env create environment.yml.

PS: Not sure what your timezone is but due to my ignorance on RTD it would be nice if we could touch base on gitter at some point. We can probably figure this out quickly.

@ocefpaf where you able to make any progress on this? (reminder: NetCDF4 is not installed properly on RTD, with ImportError libmfhdf.so.0: cannot open shared object file: No such file or directory)

If you're busy, I could try to do some trials by adding some other packages to the configuration. This is quite silly, but for some reason I have no such problems with netCDF4 on my salem RTD builds, with a similar config (the only difference I see is the higher number of packages: https://github.com/fmaussion/salem/blob/master/docs/environment.yml)

We are talking about two different issues here. I'll open another issue for the rasm file.

Ah OK. Sorry for the noise.

We are talking about two different issues here. I'll open another issue for the rasm file.

The error is visible here the latest build logs are available here thanks!

Thanks @fmaussion I can see that there are multiple conda install commands the the subsequent ones change the env. For example:

``` conda install --yes --name latest sphinx==1.3.5 Pygments==2.1.1 docutils==0.12 mock pillow==3.0.0 sphinx_rtd_theme==0.1.7 alabaster>=0.7,<0.8,!=0.7.5

Fetching package metadata: .... Solving package specifications: .........

Package plan for installation in environment /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/latest:

The following NEW packages will be INSTALLED:

jbig:             2.1-0       
sphinx_rtd_theme: 0.1.7-py27_0

The following packages will be DOWNGRADED:

freetype:         2.6.3-1      --> 2.5.5-1     
jpeg:             9b-0         --> 8d-2        
libtiff:          4.0.6-7      --> 4.0.6-2     
pillow:           3.4.2-py27_0 --> 3.0.0-py27_1
pygments:         2.1.3-py27_1 --> 2.1.1-py27_0
sphinx:           1.4.8-py27_0 --> 1.3.5-py27_0
tk:               8.5.19-0     --> 8.5.18-0

```

That jpeg change is not desirable!

I also cannot see the full log from conda env create --name latest --file /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/environment.yml.

It's the same environment, I'm literally typing "git checkout master" or "git checkout v0.8.2" in my xarray-dev conda environment (on Python 3.5)

I am not sure it is due to the change I mention above. I will make a few experiments and report back.

It's the same environment, I'm literally typing "git checkout master" or "git checkout v0.8.2" in my xarray-dev conda environment (on Python 3.5)

On Fri, Nov 11, 2016 at 10:59 AM, Fabien Maussion notifications@github.com wrote:

Because you have a RuntimeWarning when reading already

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1106#issuecomment-260029534, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1iqw2jD48U-qC5Kjed0Ut3Lz_BS4ks5q9LsjgaJpZM4Ku57C .

Because you have a RuntimeWarning when reading already

I thought I was able to solve the datetime issue by updating netCDF4 -- do you have the same version in both environments?

I get the same error attempting to same RASM on xarray v0.8.2.

On xarray master, I get a different error, related to encoding datetimes:

``` In [1]: import xarray as xr

In [2]: ds = xr.tutorial.load_dataset('rasm') /Users/shoyer/dev/xarray/xarray/conventions.py:386: RuntimeWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy netCDF4.datetime objects instead, reason: dates out of range result = decode_cf_datetime(example_value, units, calendar)

In [3]: ds.to_netcdf('rasm.nc', format='NETCDF3_CLASSIC', engine='scipy')

TypeError Traceback (most recent call last) <ipython-input-3-f834a013f387> in <module>() ----> 1 ds.to_netcdf('rasm.nc', format='NETCDF3_CLASSIC', engine='scipy')

/Users/shoyer/dev/xarray/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding) 801 from ..backends.api import to_netcdf 802 return to_netcdf(self, path, mode, format=format, group=group, --> 803 engine=engine, encoding=encoding) 804 805 def unicode(self):

/Users/shoyer/dev/xarray/xarray/backends/api.py in to_netcdf(dataset, path, mode, format, group, engine, writer, encoding) 513 store = store_cls(path, mode, format, group, writer) 514 try: --> 515 dataset.dump_to_store(store, sync=sync, encoding=encoding) 516 if isinstance(path, BytesIO): 517 return path.getvalue()

/Users/shoyer/dev/xarray/xarray/core/dataset.py in dump_to_store(self, store, encoder, sync, encoding) 747 variables, attrs = encoder(variables, attrs) 748 --> 749 store.store(variables, attrs, check_encoding) 750 if sync: 751 store.sync()

/Users/shoyer/dev/xarray/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set) 230 # All NetCDF files get CF encoded by default, without this attempting 231 # to write times, for example, would fail. --> 232 cf_variables, cf_attrs = cf_encoder(variables, attributes) 233 AbstractWritableDataStore.store(self, cf_variables, cf_attrs, 234 check_encoding_set)

/Users/shoyer/dev/xarray/xarray/conventions.py in cf_encoder(variables, attributes) 1067 """ 1068 new_vars = OrderedDict((k, encode_cf_variable(v, name=k)) -> 1069 for k, v in iteritems(variables)) 1070 return new_vars, attributes

/Users/shoyer/dev/xarray/xarray/conventions.py in <genexpr>(.0) 1067 """ 1068 new_vars = OrderedDict((k, encode_cf_variable(v, name=k)) -> 1069 for k, v in iteritems(variables)) 1070 return new_vars, attributes

/Users/shoyer/dev/xarray/xarray/conventions.py in encode_cf_variable(var, needs_copy, name) 721 var, needs_copy = maybe_encode_offset_and_scale(var, needs_copy) 722 var, needs_copy = maybe_encode_fill_value(var, needs_copy) --> 723 var = maybe_encode_dtype(var, name) 724 var = maybe_encode_bools(var) 725 var = ensure_dtype_not_object(var)

/Users/shoyer/dev/xarray/xarray/conventions.py in maybe_encode_dtype(var, name) 634 data = string_to_char(np.asarray(data, 'S')) 635 dims = dims + ('string%s' % data.shape[-1],) --> 636 data = data.astype(dtype=dtype) 637 var = Variable(dims, data, attrs, encoding) 638 return var

TypeError: float() argument must be a string or a number, not 'datetime.datetime' ```

The error is visible here

the latest build logs are available here

thanks!

Can you point me to the error and some details on RTD. They may need to update the conda version to get it to work.

OK, so this didn't help. I'm happy with leaving netCDF4 out of RTD, but I'm unable to convert the rasm file because of the bug I mentioned above.

@fmaussion I am away from a laptop to test this but the following change should fix it (depending on the conda version you have there) :

yaml channels: - conda-forge - defaults

By explicitly adding defaults below conda-forge you will ensure that the right set of packages will be downloaded thanks to the channel preference feature in recent conda version.

I wonder if you can reproduce this?

python import xarray as xr ds = xr.tutorial.load_dataset('rasm') ds.to_netcdf('rasm.nc', format='NETCDF3_CLASSIC', engine='scipy')

I have not seen that one before!

Maybe it's an issue because that file already exists?

I wanted to be sure that IO works well with the scipy engine, but I get this error below. Does that look familiar to you?

``` python import xarray as xr ds = xr.open_dataset('/home/mowglie/Downloads/rasm.nc') ds.Tair.encoding['dtype'] = 'float32' ds.to_netcdf('/home/mowglie/rasm.nc', format='NETCDF3_CLASSIC', engine='scipy')

ValueError Traceback (most recent call last) /home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/backends/api.py in to_netcdf(dataset, path, mode, format, group, engine, writer, encoding) 353 try: --> 354 dataset.dump_to_store(store, sync=sync, encoding=encoding) 355 if isinstance(path, BytesIO):

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/core/dataset.py in dump_to_store(self, store, encoder, sync, encoding) 729 if sync: --> 730 store.sync() 731

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/backends/scipy_.py in sync(self) 147 super(ScipyDataStore, self).sync() --> 148 self.ds.flush() 149

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in flush(self) 388 if hasattr(self, 'mode') and self.mode in 'wa': --> 389 self._write() 390 sync = flush

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write(self) 400 self._write_gatt_array() --> 401 self._write_var_array() 402

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write_var_array(self) 448 for name in variables: --> 449 self._write_var_metadata(name) 450 # Now that we have the metadata, we know the vsize of

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write_var_metadata(self, name) 466 for dimname in var.dimensions: --> 467 dimid = self._dims.index(dimname) 468 self._pack_int(dimid)

ValueError: '2' is not in list

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) <ipython-input-22-4077b7ef5b8c> in <module>() 2 ds = xr.open_dataset('/home/mowglie/Downloads/rasm.nc') 3 ds.Tair.encoding['dtype'] = 'float32' ----> 4 ds.to_netcdf('/home/mowglie/rasm.nc', format='NETCDF3_CLASSIC', engine='scipy')

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding) 780 from ..backends.api import to_netcdf 781 return to_netcdf(self, path, mode, format=format, group=group, --> 782 engine=engine, encoding=encoding) 783 784 dump = utils.function_alias(to_netcdf, 'dump')

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/backends/api.py in to_netcdf(dataset, path, mode, format, group, engine, writer, encoding) 357 finally: 358 if sync: --> 359 store.close() 360 361 if not sync:

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/xarray/backends/scipy_.py in close(self) 149 150 def close(self): --> 151 self.ds.close() 152 153 def exit(self, type, value, tb):

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in close(self) 277 if not self.fp.closed: 278 try: --> 279 self.flush() 280 finally: 281 self.variables = {}

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in flush(self) 387 """ 388 if hasattr(self, 'mode') and self.mode in 'wa': --> 389 self._write() 390 sync = flush 391

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write(self) 399 self._write_dim_array() 400 self._write_gatt_array() --> 401 self._write_var_array() 402 403 def _write_numrecs(self):

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write_var_array(self) 447 # Set the metadata for all variables. 448 for name in variables: --> 449 self._write_var_metadata(name) 450 # Now that we have the metadata, we know the vsize of 451 # each record variable, so we can calculate recsize.

/home/mowglie/.bin/conda/envs/climate/lib/python3.5/site-packages/scipy/io/netcdf.py in _write_var_metadata(self, name) 465 self._pack_int(len(var.dimensions)) 466 for dimname in var.dimensions: --> 467 dimid = self._dims.index(dimname) 468 self._pack_int(dimid) 469

ValueError: '2' is not in list ```

Yes, please open a PR in xarray-data!

Yes, with the following code the file size goes down to 9mb:

python import xarray as xr ds = xr.open_dataset('/home/tt/Downloads/rasm.nc') ds.Tair.encoding['dtype'] = 'float32' ds.to_netcdf('/home/tt/rasm.nc', format='NETCDF3_CLASSIC')

I'm happy to do the conversion if you want. One could compress the file a little bit by changing the data to float32, too

Another option would be to switch all the example files (https://github.com/pydata/xarray-data) to use netCDF3. That's probably a good idea for maximum compatibility in any case, because we can read netCDF3 with only scipy installed.

cc @jhamman