html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/463#issuecomment-143373357,https://api.github.com/repos/pydata/xarray/issues/463,143373357,MDEyOklzc3VlQ29tbWVudDE0MzM3MzM1Nw==,380927,2015-09-25T23:11:39Z,2015-09-25T23:11:39Z,NONE,"OK, I'll try. Thanks. But I originally tested if netCDF4 can work with a closed/reopened variable like this: ``` python In [1]: import netCDF4 In [2]: a = netCDF4.Dataset(""temp.nc"", mode=""w"") In [3]: a.createDimension(""lon"") Out[3]: (unlimited): name = 'lon', size = 0 In [4]: a.createVariable(""lon"", ""f8"", dimensions=(""lon"")) Out[4]: float64 lon(lon) unlimited dimensions: lon current shape = (0,) filling on, default _FillValue of 9.969209968386869e+36 used In [5]: v = a.variables['lon'] In [6]: v Out[6]: float64 lon(lon) unlimited dimensions: lon current shape = (0,) filling on, default _FillValue of 9.969209968386869e+36 used In [7]: a.close() In [8]: v Out[8]: --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) /home/cp/.pyenv/versions/miniconda3-3.16.0/envs/xray-3.5.0/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj) 695 type_pprinters=self.type_printers, 696 deferred_pprinters=self.deferred_printers) --> 697 printer.pretty(obj) 698 printer.flush() 699 return stream.getvalue() /home/cp/.pyenv/versions/miniconda3-3.16.0/envs/xray-3.5.0/lib/python3.5/site-packages/IPython/lib/pretty.py in pretty(self, obj) 381 if callable(meth): 382 return meth(obj, self, cycle) --> 383 return _default_pprint(obj, self, cycle) 384 finally: 385 self.end_group() /home/cp/.pyenv/versions/miniconda3-3.16.0/envs/xray-3.5.0/lib/python3.5/site-packages/IPython/lib/pretty.py in _default_pprint(obj, p, cycle) 501 if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs: 502 # A user-provided repr. Find newlines and replace them with p.break_() --> 503 _repr_pprint(obj, p, cycle) 504 return 505 p.begin_group(1, '<') /home/cp/.pyenv/versions/miniconda3-3.16.0/envs/xray-3.5.0/lib/python3.5/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle) 683 """"""A pprint that just redirects to the normal repr function."""""" 684 # Find newlines and replace them with p.break_() --> 685 output = repr(obj) 686 for idx,output_line in enumerate(output.splitlines()): 687 if idx: netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__repr__ (netCDF4/_netCDF4.c:25045)() netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__unicode__ (netCDF4/_netCDF4.c:25243)() netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.dimensions.__get__ (netCDF4/_netCDF4.c:27486)() netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._getdims (netCDF4/_netCDF4.c:26297)() RuntimeError: NetCDF: Not a valid ID In [9]: a = netCDF4.Dataset(""temp.nc"") In [10]: v Out[10]: class 'netCDF4._netCDF4.Variable'> lon(lon) dimensions: lon shape = (0,) on, default _FillValue of 9.969209968386869e+36 used ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,94328498 https://github.com/pydata/xarray/issues/463#issuecomment-143338384,https://api.github.com/repos/pydata/xarray/issues/463,143338384,MDEyOklzc3VlQ29tbWVudDE0MzMzODM4NA==,380927,2015-09-25T20:02:42Z,2015-09-25T20:02:42Z,NONE,"I've only put the try - except there to conditionally set the breakpoint. How does it make a difference if the self.store.close is called? It it is not called then the dataset remains opened which should not cause the weird behaviour reported above? Nevertheless I have updated my branch to use a contextmanager because it is a better solution but I still have this strange behaviour of only printing the variable altering the test outcome. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,94328498 https://github.com/pydata/xarray/issues/463#issuecomment-143222580,https://api.github.com/repos/pydata/xarray/issues/463,143222580,MDEyOklzc3VlQ29tbWVudDE0MzIyMjU4MA==,380927,2015-09-25T13:27:59Z,2015-09-25T13:27:59Z,NONE,"I've pushed a few commits trying this out to https://github.com/cpaulik/xray/tree/closing_netcdf_backend . I can open a WIP PR if this would be easier to discuss there. There are however a few tests that keep failing and I can not figure out why. e.g.: `test_backends.py::NetCDF4ViaDaskDataTest::test_compression_encoding`: If I set a breakpoint at [line 941 of dataset.py](https://github.com/cpaulik/xray/blob/closing_netcdf_backend/xray/core/dataset.py#L941) and just continue the test fails. If I however evaluate `self.variables.items()` or even `self.variables` at the breakpoint I get the correct output and the test passes when continued. I can not really see the difference between me evaluating this in `ipdb` and the code that is on the line. The error I get when running the test without interference is: ``` shell test_backends.py::NetCDF4ViaDaskDataTest::test_compression_encoding FAILED ====================================================== FAILURES ======================================================= __________________________________ NetCDF4ViaDaskDataTest.test_compression_encoding ___________________________________ self = def test_compression_encoding(self): data = create_test_data() data['var2'].encoding.update({'zlib': True, 'chunksizes': (5, 5), 'fletcher32': True}) > with self.roundtrip(data) as actual: test_backends.py:502: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/contextlib.py:17: in __enter__ return self.gen.next() test_backends.py:596: in roundtrip yield ds.chunk() ../core/dataset.py:942: in chunk for k, v in self.variables.items()]) ../core/dataset.py:935: in maybe_chunk token2 = tokenize(name, token if token else var._data) /home/cpa/.virtualenvs/xray/local/lib/python2.7/site-packages/dask/base.py:152: in tokenize return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest() ../core/indexing.py:301: in __repr__ (type(self).__name__, self.array, self.key)) ../core/utils.py:377: in __repr__ return '%s(array=%r)' % (type(self).__name__, self.array) ../core/indexing.py:301: in __repr__ (type(self).__name__, self.array, self.key)) ../core/utils.py:377: in __repr__ return '%s(array=%r)' % (type(self).__name__, self.array) netCDF4/_netCDF4.pyx:2931: in netCDF4._netCDF4.Variable.__repr__ (netCDF4/_netCDF4.c:25068) ??? netCDF4/_netCDF4.pyx:2938: in netCDF4._netCDF4.Variable.__unicode__ (netCDF4/_netCDF4.c:25243) ??? netCDF4/_netCDF4.pyx:3059: in netCDF4._netCDF4.Variable.dimensions.__get__ (netCDF4/_netCDF4.c:27486) ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E RuntimeError: NetCDF: Not a valid ID netCDF4/_netCDF4.pyx:2994: RuntimeError ============================================== 1 failed in 0.50 seconds =============================================== ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,94328498 https://github.com/pydata/xarray/issues/463#issuecomment-142637232,https://api.github.com/repos/pydata/xarray/issues/463,142637232,MDEyOklzc3VlQ29tbWVudDE0MjYzNzIzMg==,380927,2015-09-23T15:19:36Z,2015-09-23T15:19:36Z,NONE,"I've run into the same problem and have been looking at the netCDF backend. A solution does not seem to be so easy as to open and close the file in the `__getitem__` method since this closes the file also for any other access e.g. attributes like `shape` or `dtype`. Short of decorating all the functions of the netCDF4 package I can not think of a workable solution to this. But maybe I'm overlooking something fundamental. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,94328498