issue_comments

Table actions

10 rows where issue = 379177627 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 3

issue 1

  • HDF Errors since xarray 0.11 · 10 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
437718958 https://github.com/pydata/xarray/issues/2551#issuecomment-437718958 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzcxODk1OA== dcherian 2448579 2018-11-12T00:12:46Z 2018-11-12T00:12:46Z MEMBER

"Perhaps we should raise our own error though instead of just passing things through to netCDF4"

This seems like a good idea

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437717439 https://github.com/pydata/xarray/issues/2551#issuecomment-437717439 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzcxNzQzOQ== shoyer 1217238 2018-11-11T23:52:34Z 2018-11-11T23:52:34Z MEMBER

Previously the indexing operation would sometimes return a NumPy array. Now it's always lazy, so accessing the dataset when the file is closed fails.

Perhaps we should raise our own error though instead of just passing things through to netCDF4.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437707951 https://github.com/pydata/xarray/issues/2551#issuecomment-437707951 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzcwNzk1MQ== fmaussion 10050469 2018-11-11T21:46:02Z 2018-11-11T21:46:02Z MEMBER

but now operates lazily.

If so, how does the dataset access the data if the file is closed? Shouldn't it raise an error instead?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437707332 https://github.com/pydata/xarray/issues/2551#issuecomment-437707332 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzcwNzMzMg== shoyer 1217238 2018-11-11T21:37:42Z 2018-11-11T21:37:42Z MEMBER

My best guess is that ds.thick.isel(x=('z', [1, 3, 4]), y=('z', [7, 2, 1])) previously loaded data into memory (and cached it) but now operates lazily.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437705060 https://github.com/pydata/xarray/issues/2551#issuecomment-437705060 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzcwNTA2MA== fmaussion 10050469 2018-11-11T21:08:13Z 2018-11-11T21:08:13Z MEMBER

Actually I have to ask: why is it possible to to an operation on a closed dataset? What is happening under the hood which creates this error and wasn't earlier?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437664983 https://github.com/pydata/xarray/issues/2551#issuecomment-437664983 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzY2NDk4Mw== fmaussion 10050469 2018-11-11T12:10:36Z 2018-11-11T12:10:36Z MEMBER

Yep, that looks a bug on your end. Were you using autoclose=True before?

No, the exact same piece of code works without error with xarray prev 0.11. Closing this now!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437618316 https://github.com/pydata/xarray/issues/2551#issuecomment-437618316 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzYxODMxNg== shoyer 1217238 2018-11-10T20:18:31Z 2018-11-10T20:18:31Z MEMBER

@fmaussion Yep, that looks a bug on your end. Were you using autoclose=True before? If so, that could have hidden the issue, because exiting the with block didn't use to actually close the file in that case.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437574599 https://github.com/pydata/xarray/issues/2551#issuecomment-437574599 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzU3NDU5OQ== fmaussion 10050469 2018-11-10T10:39:09Z 2018-11-10T10:40:35Z MEMBER

I am slightly regretting not doing a release candidate here,

I should be better placed than anyone to anticipate and test more frequently on master :roll_eyes:

So this reproduces the problem on my machine and travis. Note the misplaced computation out of the with block:

```python import xarray as xr import numpy as np import netCDF4 import os

try: os.remove('test.nc') except OSError: pass

thick = np.zeros((180, 200), np.float32)

ds = xr.Dataset() ds['thick'] = (('y', 'x'), thick) ds.to_netcdf('test.nc')

with xr.open_dataset('test.nc') as ds: dummy = ds.thick.isel(x=('z', [1, 3, 4]), y=('z', [7, 2, 1]))

dummy.min() # this is the issue

with netCDF4.Dataset('test.nc', 'a') as nc: nc.variables['thick'][:] = thick
```

I guess this was actually a bug in our code, but for some reason it worked fine with earlier xarray versions. I'm not sure if this requires action on the xarray side in the end...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437483895 https://github.com/pydata/xarray/issues/2551#issuecomment-437483895 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzQ4Mzg5NQ== shoyer 1217238 2018-11-09T20:22:03Z 2018-11-09T20:22:03Z MEMBER

I am slightly regretting not doing a release candidate here, but hopefully this should be straightforward to fix.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627
437471566 https://github.com/pydata/xarray/issues/2551#issuecomment-437471566 https://api.github.com/repos/pydata/xarray/issues/2551 MDEyOklzc3VlQ29tbWVudDQzNzQ3MTU2Ng== shoyer 1217238 2018-11-09T19:37:15Z 2018-11-09T19:37:15Z MEMBER

Oh my -- sorry about that!

Thinking about this a little more, I guess this should not be too surprising since I don't think we have any dask integration tests that cover appending to existing files. Maybe a good place to start would be adding one of those, e.g., adapted from this existing test: https://github.com/pydata/xarray/blob/575e97aef405c9b473508f5bc0e66332df4930f3/xarray/tests/test_distributed.py#L66

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   HDF Errors since xarray 0.11 379177627

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);