html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1849#issuecomment-489236748,https://api.github.com/repos/pydata/xarray/issues/1849,489236748,MDEyOklzc3VlQ29tbWVudDQ4OTIzNjc0OA==,12465248,2019-05-03T20:54:50Z,2019-05-03T20:54:50Z,CONTRIBUTOR,"@dcherian Thanks,
First, I think you're right that the `encoding['contiguous']=True` is coming from the input file. That was not clear to me (and I did not read the xarray code to verify). But it makes sense.
Second, my example shows something more slightly complicated than the original example which was also not clear to me. In my case the unlimited dimension (`time`) is chunked and is being successfully written in both cases (before and after work around). The error/ failure is happening on the a variable that contains the unlimited dimension but which has `encoding['contiguous']=True` for the variable.
This makes sense upon a slightly more nuanced reading of the netcdf4 manual (as quoted my markelg)
> ""contiguous: if True (default False), the variable data is stored contiguously on disk. Default False. Setting to True for a variable with an unlimited dimension will trigger an error.""
The last sentence apparently means that for *any variable with an unlimited dimension* the use of `contiguous=True` triggers an error. That was not clear to me until I looked a bit harder at this. I think that slightly refines the strategy of how to deal with the problem.
I propose that the solution should be *both*
a) delete encoding['contiguous'] if it is True when asked to write out a variable containing an unlimited dimension.
b) raise an informative warning that the variable was chunked because it contained an unlimited dimension. (If a user hates warnings, they could can handle this deletion herself. One the other hand, there's really nothing else to do, so I'm not sure the warning is necessary... I dont have strong opinion on this, but the code is fiddling with the encodings under the hood, so a warning seems polite).
A final question: should the encoding['contiguous'] be removed from the xarray variable or should it just be removed for purposes of writing it to ncdf4 on disk? I suppose a user could be writing the xarray dataset to another format that might allow what netcdf does not allow. This should be an easy detail.
I'll make a PR with the above and we can evaluate the concrete changes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-489164504,https://api.github.com/repos/pydata/xarray/issues/1849,489164504,MDEyOklzc3VlQ29tbWVudDQ4OTE2NDUwNA==,2448579,2019-05-03T16:51:40Z,2019-05-03T16:51:40Z,MEMBER,"Because it's set in your input file. Both example files in this thread have `_Storage = 'contiguous'` set on all variables but the dimensions are not unlimited, so these files are compliant. Here's the output of `ncdump -sh sample.nc` (second comment in this thread)
```
netcdf sample {
dimensions:
y = 6 ;
variables:
float x(y) ;
x:_FillValue = NaNf ;
x:_Storage = ""contiguous"" ;
x:_Endianness = ""little"" ;
int64 y(y) ;
y:_Storage = ""contiguous"" ;
y:_Endianness = ""little"" ;
// global attributes:
:_NCProperties = ""version=1,netcdflibversion=4.4.1.1,hdf5libversion=1.8.18"" ;
:_SuperblockVersion = 0 ;
:_IsNetcdf4 = 1 ;
:_Format = ""netCDF-4"" ;
```
When you ask xarray to write out an unlimited dimension, it doesn't delete `encoding['contiguous']` and then netCDF4 raises an error (I think).
It's probable that the underlying software you're using to write has probably changed versions and is setting it by default. You can check this by comparing the output of `ncdump -sh file.nc` on cheyenne and your local machine
If this is right, the solution would be either
a) delete `encoding['contiguous']` if it is True when asked to write out an unlimited dimension.
b) raise a warning and ask the user to do the deletion before writing.
My preference is for (a).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-489156658,https://api.github.com/repos/pydata/xarray/issues/1849,489156658,MDEyOklzc3VlQ29tbWVudDQ4OTE1NjY1OA==,12465248,2019-05-03T16:25:19Z,2019-05-03T16:40:52Z,CONTRIBUTOR,"Here's what I understand so far.
For my file, i write it with (""ensured"") and without (""unensured"") the workaround (actually @markelg for discovering this).
```
(base) jamesmcc@cheyenne3[1021]:/glade/scratch/jamesmcc/florence_cutout_routelink_ensemble_run/ensemble> grep '_Storage' ensured_ncdsh.txt
feature_id:_Storage = ""contiguous"" ;
latitude:_Storage = ""contiguous"" ;
longitude:_Storage = ""contiguous"" ;
time:_Storage = ""chunked"" ;
member:_Storage = ""contiguous"" ;
crs:_Storage = ""chunked"" ;
order:_Storage = ""chunked"" ;
elevation:_Storage = ""chunked"" ;
streamflow:_Storage = ""chunked"" ;
q_lateral:_Storage = ""chunked"" ;
velocity:_Storage = ""chunked"" ;
Head:_Storage = ""chunked"" ;
(base) jamesmcc@cheyenne3[1022]:/glade/scratch/jamesmcc/florence_cutout_routelink_ensemble_run/ensemble> grep '_Storage' unensured_ncdsh.txt
feature_id:_Storage = ""contiguous"" ;
latitude:_Storage = ""contiguous"" ;
longitude:_Storage = ""contiguous"" ;
time:_Storage = ""chunked"" ;
member:_Storage = ""contiguous"" ;
crs:_Storage = ""chunked"" ;
```
The error that is thrown is, just the tail end of it:
```
/glade/p/cisl/nwc/jamesmcc/anaconda3/lib/python3.7/site-packages/xarray/backends/netCDF4_.py in prepare_variable(self, name, variable, check_encoding, unlimited_dims)
466 least_significant_digit=encoding.get(
467 'least_significant_digit'),
--> 468 fill_value=fill_value)
469 _disable_auto_decode_variable(nc4_var)
470
netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.createVariable()
netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__init__()
netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
RuntimeError: NetCDF: Invalid argument
```
If I go to line 464 in `xarray/backends/netCDF4_.py`, I see that the variable it is failing on is `crs`. If I
```
print(name)
crs
encoding.get('contiguous', False)
True
```
but the ncdump -sh shows it's actually chunked. I'm not sure this is exactly what's raising the error down the line, but these two things seem to be at odds.
My current question is ""why does `encoding.get('contiguous', False)` return `True`?""
If you have any insights let me know. I probably wont have time to mess with this until next week. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-488870732,https://api.github.com/repos/pydata/xarray/issues/1849,488870732,MDEyOklzc3VlQ29tbWVudDQ4ODg3MDczMg==,2448579,2019-05-02T23:44:00Z,2019-05-02T23:44:00Z,MEMBER,does ncdump -sh show whether contiguous is true?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-488865903,https://api.github.com/repos/pydata/xarray/issues/1849,488865903,MDEyOklzc3VlQ29tbWVudDQ4ODg2NTkwMw==,12465248,2019-05-02T23:19:14Z,2019-05-02T23:19:14Z,CONTRIBUTOR,"I could be persuaded.
I just dont understand how 'contiguous' gets set on the encoding of these variables and if that is appropriate. Does that seem obvious/clear to anyone?
I still dont understand why this is happening for me. I made some fairly small modifications to some code that never threw this error in the past. The small mods could have done it, but the identical code on my laptop did not throw this error on a small sample dataset. Then I went to cheyenne, where all bets are off!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-488843571,https://api.github.com/repos/pydata/xarray/issues/1849,488843571,MDEyOklzc3VlQ29tbWVudDQ4ODg0MzU3MQ==,2448579,2019-05-02T21:44:29Z,2019-05-02T21:44:29Z,MEMBER,@jmccreight Are you up for sending in a PR to raise an informative error message?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-488841260,https://api.github.com/repos/pydata/xarray/issues/1849,488841260,MDEyOklzc3VlQ29tbWVudDQ4ODg0MTI2MA==,12465248,2019-05-02T21:36:41Z,2019-05-02T21:36:41Z,CONTRIBUTOR,"I apparently have this problem too.
Thanks @gerritholl for the workaround. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-369543072,https://api.github.com/repos/pydata/xarray/issues/1849,369543072,MDEyOklzc3VlQ29tbWVudDM2OTU0MzA3Mg==,8441217,2018-03-01T10:09:40Z,2018-03-01T10:09:40Z,CONTRIBUTOR,"I also ran into this issue : to_netcdf fails for my dataset.
Here is how to reproduce the error : (the testfile is attached here [1.zip](https://github.com/pydata/xarray/files/1770670/1.zip) )
```
import netCDF4
import xarray as xr
print(netCDF4.__version__)
print(xr.__version__)
ds = xr.open_dataset('testfile')
ds.to_netcdf('outfile.ok')
ds.to_netcdf('outfile.not.ok', unlimited_dims=['datetime'])
```
And the ouput I get :
> 1.3.1
> 0.10.1
> Traceback (most recent call last):
> File ""bug.py"", line 6, in
> ds.to_netcdf('outfile', unlimited_dims=['datetime'])
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/core/dataset.py"", line 1133, in to_netcdf
> unlimited_dims=unlimited_dims)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/backends/api.py"", line 632, in to_netcdf
> unlimited_dims=unlimited_dims)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/core/dataset.py"", line 1070, in dump_to_store
> unlimited_dims=unlimited_dims)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/backends/common.py"", line 280, in store
> unlimited_dims=unlimited_dims)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 348, in set_variables
> super(NetCDF4DataStore, self).set_variables(*args, **kwargs)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/backends/common.py"", line 317, in set_variables
> name, v, check, unlimited_dims=unlimited_dims)
> File ""/home/pinaultf/miniconda3/envs/defaultenv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 393, in prepare_variable
> fill_value=fill_value)
> File ""netCDF4/_netCDF4.pyx"", line 2437, in netCDF4._netCDF4.Dataset.createVariable
> File ""netCDF4/_netCDF4.pyx"", line 3439, in netCDF4._netCDF4.Variable.__init__
> File ""netCDF4/_netCDF4.pyx"", line 1638, in netCDF4._netCDF4._ensure_nc_success
> RuntimeError: NetCDF: Invalid argument
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-366288761,https://api.github.com/repos/pydata/xarray/issues/1849,366288761,MDEyOklzc3VlQ29tbWVudDM2NjI4ODc2MQ==,2443309,2018-02-16T16:41:03Z,2018-02-16T16:41:03Z,MEMBER,"@markelg - thanks for digging into this a bit. Based on what you're saying, I think we need to raise an informative error here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-366255764,https://api.github.com/repos/pydata/xarray/issues/1849,366255764,MDEyOklzc3VlQ29tbWVudDM2NjI1NTc2NA==,6883049,2018-02-16T14:50:21Z,2018-02-16T15:10:40Z,CONTRIBUTOR,"This happened to me today after introducing some modifications in a code that was working fine. I have tried to trace it without success. Finally, I found a workaround which consist on removing the ""contiguous"" entry from the .encoding attributes. This works with gerritholl's file:
```
import xarray as xr
ds = xr.open_dataset(""sample.nc"")
del ds.x.encoding[""contiguous""]
del ds.y.encoding[""contiguous""]
ds.to_netcdf(""sample2.nc"", unlimited_dims=[""y""])
```
So it seems that this entry in the encoding dictionaries is triggering the error.
OK, so I guess that this explains it, from the netCDF4 documentation:
""contiguous: if True (default False), the variable data is stored contiguously on disk. Default False. Setting to True for a variable with an unlimited dimension will trigger an error.""
This is quite an obscure error right now, so maybe we could force contiguous to be False when unlimited_dims is being used, or either raise a more informative error.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-361468306,https://api.github.com/repos/pydata/xarray/issues/1849,361468306,MDEyOklzc3VlQ29tbWVudDM2MTQ2ODMwNg==,2443309,2018-01-30T03:47:33Z,2018-01-30T03:47:33Z,MEMBER,Thanks for the report. This seems like a bug to me and I'm frankly not sure why it isn't working. I'll look into it more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700
https://github.com/pydata/xarray/issues/1849#issuecomment-359523925,https://api.github.com/repos/pydata/xarray/issues/1849,359523925,MDEyOklzc3VlQ29tbWVudDM1OTUyMzkyNQ==,500246,2018-01-22T18:45:35Z,2018-01-22T18:45:35Z,CONTRIBUTOR,"Not sure if the attachment came through. Trying again:
[sample.nc.gz](https://github.com/pydata/xarray/files/1653187/sample.nc.gz)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,290572700