issue_comments

Table actions

2 rows where issue = 304589831 and user = 1197350 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

user 1

  • rabernat · 2 ✖

issue 1

  • Parallel open_mfdataset · 2 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
380121937 https://github.com/pydata/xarray/pull/1983#issuecomment-380121937 https://api.github.com/repos/pydata/xarray/issues/1983 MDEyOklzc3VlQ29tbWVudDM4MDEyMTkzNw== rabernat 1197350 2018-04-10T14:32:25Z 2018-04-10T14:32:25Z MEMBER

I recently tried this branch with my data server and got an error.

I opened a dataset this way ```python

works fine with parallel=False

ds = xr.open_mfdataset(os.path.join(ddir, 'V1_1.204.nc'), decode_cf=False, parallel=True) ```

and got the following error.

distributed.utils - ERROR - NetCDF: HDF error Traceback (most recent call last): File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/distributed/utils.py", line 237, in f result[0] = yield make_coro() File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "<string>", line 4, in raise_exc_info File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(*exc_info) File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/distributed/client.py", line 1356, in _gather traceback) File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/six.py", line 692, in reraise raise value.with_traceback(tb) File "/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/dask/compatibility.py", line 48, in apply return func(*args, **kwargs) File "/home/rpa/xarray/xarray/backends/api.py", line 318, in open_dataset return maybe_decode_store(store, lock) File "/home/rpa/xarray/xarray/backends/api.py", line 238, in maybe_decode_store drop_variables=drop_variables) File "/home/rpa/xarray/xarray/conventions.py", line 594, in decode_cf vars, attrs = obj.load() File "/home/rpa/xarray/xarray/backends/common.py", line 217, in load for k, v in self.get_variables().items()) File "/home/rpa/xarray/xarray/backends/netCDF4_.py", line 319, in get_variables iteritems(self.ds.variables)) File "/home/rpa/xarray/xarray/core/utils.py", line 308, in FrozenOrderedDict return Frozen(OrderedDict(*args, **kwargs)) File "/home/rpa/xarray/xarray/backends/netCDF4_.py", line 318, in <genexpr> for k, v in File "/home/rpa/xarray/xarray/backends/netCDF4_.py", line 311, in open_store_variable encoding['original_shape'] = var.shape File "netCDF4/_netCDF4.pyx", line 3381, in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:34388) File "netCDF4/_netCDF4.pyx", line 2759, in netCDF4._netCDF4.Dimension.__len__ (netCDF4/_netCDF4.c:27006) RuntimeError: NetCDF: HDF error

Without the distributed scheduler (but with parallel=True), I get no error, but the command never returns, and eventually I have to restart the kernel.

Any idea what could be going on? (Sorry for the non-reproducible bug report...I figured some trials "in the field" might be useful.)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Parallel open_mfdataset 304589831
379305062 https://github.com/pydata/xarray/pull/1983#issuecomment-379305062 https://api.github.com/repos/pydata/xarray/issues/1983 MDEyOklzc3VlQ29tbWVudDM3OTMwNTA2Mg== rabernat 1197350 2018-04-06T16:24:22Z 2018-04-06T16:24:22Z MEMBER

Can we imagine cases where it might actually degrade performance?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Parallel open_mfdataset 304589831

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);