issue_comments

Table actions

7 rows where issue = 442617907 and user = 500246 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • gerritholl · 7 ✖

issue 1

  • Segmentation fault reading many groups from many files · 7 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
509134252 https://github.com/pydata/xarray/issues/2954#issuecomment-509134252 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDUwOTEzNDI1Mg== gerritholl 500246 2019-07-08T08:39:01Z 2019-07-08T08:39:01Z CONTRIBUTOR

And I can confirm that the problem I reported originally on May 10 is also gone with #3082.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907
509132581 https://github.com/pydata/xarray/issues/2954#issuecomment-509132581 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDUwOTEzMjU4MQ== gerritholl 500246 2019-07-08T08:34:11Z 2019-07-08T08:34:38Z CONTRIBUTOR

@shoyer I checked out your branch and the latter test example runs successfully - no segmentation fault and no files left open.

I will test the former test example now.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907
508900470 https://github.com/pydata/xarray/issues/2954#issuecomment-508900470 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDUwODkwMDQ3MA== gerritholl 500246 2019-07-06T06:09:04Z 2019-07-06T06:09:04Z CONTRIBUTOR 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907
508772044 https://github.com/pydata/xarray/issues/2954#issuecomment-508772044 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDUwODc3MjA0NA== gerritholl 500246 2019-07-05T14:13:20Z 2019-07-05T14:14:20Z CONTRIBUTOR

This triggers a segmentation fault (in the .persist() call) on my system, which may be related:

python import xarray import os import subprocess xarray.set_options(file_cache_maxsize=1) f = "/path/to/netcdf/file.nc" ds1 = xarray.open_dataset(f, "/group1", chunks=1024) ds2 = xarray.open_dataset(f, "/group2", chunks=1024) ds_cat = xarray.concat([ds1, ds2]) ds_cat.persist() subprocess.run(fr"lsof | grep {os.getpid():d} | grep '\.nc$'", shell=True)

But there's something with the specific netcdf file going on, for when I create artificial groups, it does not segfault.

``` Fatal Python error: Segmentation fault

Thread 0x00007f542bfff700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 470 in _handle_results File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f5448ff9700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 422 in _handle_tasks File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f54497fa700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 413 in _handle_workers File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f5449ffb700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f544a7fc700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f544affd700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f544b7fe700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f544bfff700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f5458a75700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f5459276700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f5459a77700 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/multiprocessing/pool.py", line 110 in worker File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 865 in run File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 917 in _bootstrap_inner File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f54731236c0 (most recent call first): File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 244 in open_netcdf4_group File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 173 in acquire File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 56 in get_array File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 74 in getitem File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py", line 778 in explicit_indexing_adapter File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/backends/netCDF4.py", line 64 in getitem File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py", line 510 in array File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/numpy/core/numeric.py", line 538 in asarray File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/indexing.py", line 604 in array File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/numpy/core/numeric.py", line 538 in asarray File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py", line 213 in _as_array_or_item File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py", line 392 in values File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py", line 297 in data File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py", line 1204 in set_dims File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py", line 298 in ensure_common_dims File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/variable.py", line 2085 in concat File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py", line 305 in _dataset_concat File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/xarray/core/combine.py", line 120 in concat File "mwe13.py", line 19 in <module> Segmentation fault (core dumped) ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907
508728959 https://github.com/pydata/xarray/issues/2954#issuecomment-508728959 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDUwODcyODk1OQ== gerritholl 500246 2019-07-05T11:29:50Z 2019-07-05T11:29:50Z CONTRIBUTOR

This can also be triggered by a .persist(...) call, although I don't yet understand the precise circumstances.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907
491866549 https://github.com/pydata/xarray/issues/2954#issuecomment-491866549 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDQ5MTg2NjU0OQ== gerritholl 500246 2019-05-13T15:18:33Z 2019-05-13T15:18:33Z CONTRIBUTOR

In our code, this problem gets triggered because of xarrays lazy handling. If we have

with xr.open_dataset('file.nc') as ds: val = ds["field"] return val

then when a caller tries to use val, xarray reopens the dataset and does not close it again. This means the context manager is actually useless: we're using the context manager to close the file as soon as we have accessed the value, but later the file gets opened again anyway. This is against the intention of the code.

We can avoid this by calling val.load() from within the context manager, as the linked satpy PR above does. But what is the intention of xarrays design here? Should lazy reading close the file after opening and reading the value? I would say it probably should do something like

if file_was_not_open: open file get value close file # this step currently omitted return value else: get value return value

is not closing the file after it has been opened for retrieving a "lazy" file by design, or might this be considered a wart/bug?

 
{
    "total_count": 2,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 2
}
   Segmentation fault reading many groups from many files 442617907
491221266 https://github.com/pydata/xarray/issues/2954#issuecomment-491221266 https://api.github.com/repos/pydata/xarray/issues/2954 MDEyOklzc3VlQ29tbWVudDQ5MTIyMTI2Ng== gerritholl 500246 2019-05-10T09:18:28Z 2019-05-10T09:18:28Z CONTRIBUTOR

Note that if I close every file neatly, there is no segmentation fault.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Segmentation fault reading many groups from many files 442617907

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);