html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/2954#issuecomment-509134252,https://api.github.com/repos/pydata/xarray/issues/2954,509134252,MDEyOklzc3VlQ29tbWVudDUwOTEzNDI1Mg==,500246,2019-07-08T08:39:01Z,2019-07-08T08:39:01Z,CONTRIBUTOR,And I can confirm that the problem I reported originally on May 10 is also gone with #3082.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-509132581,https://api.github.com/repos/pydata/xarray/issues/2954,509132581,MDEyOklzc3VlQ29tbWVudDUwOTEzMjU4MQ==,500246,2019-07-08T08:34:11Z,2019-07-08T08:34:38Z,CONTRIBUTOR,"@shoyer I checked out your branch and the latter test example runs successfully - no segmentation fault and no files left open.
I will test the former test example now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-508900470,https://api.github.com/repos/pydata/xarray/issues/2954,508900470,MDEyOklzc3VlQ29tbWVudDUwODkwMDQ3MA==,500246,2019-07-06T06:09:04Z,2019-07-06T06:09:04Z,CONTRIBUTOR,"There are some files triggering the problem at ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_Test-Data/FCI_L1C_24hr_Test_Data_for_Users/1.0/UNCOMPRESSED/
I will test the PR later (latest on Monday)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-508772044,https://api.github.com/repos/pydata/xarray/issues/2954,508772044,MDEyOklzc3VlQ29tbWVudDUwODc3MjA0NA==,500246,2019-07-05T14:13:20Z,2019-07-05T14:14:20Z,CONTRIBUTOR,"This triggers a segmentation fault (in the `.persist()` call) on my system, which may be related:
```python
import xarray
import os
import subprocess
xarray.set_options(file_cache_maxsize=1)
f = ""/path/to/netcdf/file.nc""
ds1 = xarray.open_dataset(f, ""/group1"", chunks=1024)
ds2 = xarray.open_dataset(f, ""/group2"", chunks=1024)
ds_cat = xarray.concat([ds1, ds2])
ds_cat.persist()
subprocess.run(fr""lsof | grep {os.getpid():d} | grep '\.nc$'"", shell=True)
```
But there's something with the specific netcdf file going on, for when I create artificial groups, it does not segfault.
```
Fatal Python error: Segmentation fault
Thread 0x00007f542bfff700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 470 in _handle_results
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f5448ff9700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 422 in _handle_tasks
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f54497fa700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 413 in _handle_workers
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f5449ffb700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f544a7fc700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f544affd700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f544b7fe700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f544bfff700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f5458a75700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f5459276700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Thread 0x00007f5459a77700 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py"", line 110 in worker
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 865 in run
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 917 in _bootstrap_inner
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py"", line 885 in _bootstrap
Current thread 0x00007f54731236c0 (most recent call first):
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py"", line 244 in _open_netcdf4_group
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/file_manager.py"", line 173 in acquire
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py"", line 56 in get_array
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py"", line 74 in _getitem
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py"", line 778 in explicit_indexing_adapter
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py"", line 64 in __getitem__
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py"", line 510 in __array__
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/numpy/core/numeric.py"", line 538 in asarray
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py"", line 604 in __array__
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/numpy/core/numeric.py"", line 538 in asarray
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py"", line 213 in _as_array_or_item
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py"", line 392 in values
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py"", line 297 in data
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py"", line 1204 in set_dims
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py"", line 298 in ensure_common_dims
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py"", line 2085 in concat
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py"", line 305 in _dataset_concat
File ""/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py"", line 120 in concat
File ""mwe13.py"", line 19 in
Segmentation fault (core dumped)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-508728959,https://api.github.com/repos/pydata/xarray/issues/2954,508728959,MDEyOklzc3VlQ29tbWVudDUwODcyODk1OQ==,500246,2019-07-05T11:29:50Z,2019-07-05T11:29:50Z,CONTRIBUTOR,"This can also be triggered by a `.persist(...)` call, although I don't yet understand the precise circumstances.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-491866549,https://api.github.com/repos/pydata/xarray/issues/2954,491866549,MDEyOklzc3VlQ29tbWVudDQ5MTg2NjU0OQ==,500246,2019-05-13T15:18:33Z,2019-05-13T15:18:33Z,CONTRIBUTOR,"In our code, this problem gets triggered because of xarrays lazy handling. If we have
```
with xr.open_dataset('file.nc') as ds:
val = ds[""field""]
return val
```
then when a caller tries to use `val`, xarray reopens the dataset and does not close it again. This means the context manager is actually useless: we're using the context manager to close the file as soon as we have accessed the value, but later the file gets opened again anyway. This is against the intention of the code.
We can avoid this by calling `val.load()` from within the context manager, as the linked satpy PR above does. But what is the intention of xarrays design here? Should lazy reading close the file after opening and reading the value? I would say it probably should do something like
```
if file_was_not_open:
open file
get value
close file # this step currently omitted
return value
else:
get value
return value
```
is not closing the file after it has been opened for retrieving a ""lazy"" file by design, or might this be considered a wart/bug?","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 2}",,442617907
https://github.com/pydata/xarray/issues/2954#issuecomment-491221266,https://api.github.com/repos/pydata/xarray/issues/2954,491221266,MDEyOklzc3VlQ29tbWVudDQ5MTIyMTI2Ng==,500246,2019-05-10T09:18:28Z,2019-05-10T09:18:28Z,CONTRIBUTOR,"Note that if I close every file neatly, there is no segmentation fault.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,442617907