issue_comments

Table actions

4 rows where user = 2622379 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

user 1

  • jnhansen · 4 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
445524755 https://github.com/pydata/xarray/issues/2560#issuecomment-445524755 https://api.github.com/repos/pydata/xarray/issues/2560 MDEyOklzc3VlQ29tbWVudDQ0NTUyNDc1NQ== jnhansen 2622379 2018-12-09T10:06:24Z 2018-12-09T10:06:24Z NONE

I have done a bit more testing on this, and I believe the issue may not necessarily be with xarray but with rasterio (disclaimer: I haven't tested your pull request yet).

I can reproduce the following on Ubuntu and on Travis CI. On Mac OS none of these errors occur.

Minimum example:

python import xarray as xr import numpy as np import rasterio # The rasterio import makes the last line of this code fail. ds = xr.Dataset() ds['data'] = (('y', 'x'), np.ones((10, 10))) ds.to_netcdf('test.nc', engine='netcdf4')

I was able to fix the error by prepending a

python import netCDF4

at the very top of the script.

A very similar thing happens with engine='h5netcdf'. The same script works without rasterio, fails with rasterio, and can be fixed by inserting import h5netcdf at the top of the script.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Lock related problem in on travis-ci but not on local machine 383057458
445270674 https://github.com/pydata/xarray/issues/2560#issuecomment-445270674 https://api.github.com/repos/pydata/xarray/issues/2560 MDEyOklzc3VlQ29tbWVudDQ0NTI3MDY3NA== jnhansen 2622379 2018-12-07T15:39:05Z 2018-12-07T15:39:05Z NONE

I am having the exact same problem. Have you found a solution/workaround?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Lock related problem in on travis-ci but not on local machine 383057458
410361639 https://github.com/pydata/xarray/issues/2159#issuecomment-410361639 https://api.github.com/repos/pydata/xarray/issues/2159 MDEyOklzc3VlQ29tbWVudDQxMDM2MTYzOQ== jnhansen 2622379 2018-08-03T20:03:21Z 2018-08-03T20:14:17Z NONE

Yes, xarray should support that very easily -- assuming you have dask installed: python ds = auto_merge('*.nc') ds.to_netcdf('larger_than_memory.nc') auto_merge conserves the chunk sizes resulting from the individual files. If the single files are still too large to fit into memory individually you can rechunk to smaller chunk sizes. The same goes of course for the original xarray.open_mfdataset.

I tested it on a ~25 GB dataset (on a machine with less memory than that).

Note: ds = auto_merge('*.nc') actually runs in a matter of milliseconds, as it merely provides a view of the merged dataset. Only once you call ds.to_netcdf('larger_than_memory.nc') all the disk I/O happens.

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
   Concatenate across multiple dimensions with open_mfdataset 324350248
410348249 https://github.com/pydata/xarray/issues/2159#issuecomment-410348249 https://api.github.com/repos/pydata/xarray/issues/2159 MDEyOklzc3VlQ29tbWVudDQxMDM0ODI0OQ== jnhansen 2622379 2018-08-03T19:07:10Z 2018-08-03T19:12:21Z NONE

I just had the exact same problem, and while I didn't yet have time to dig into the source code of xarray.open_mfdataset, I wrote my own function to achieve this:

https://gist.github.com/jnhansen/fa474a536201561653f60ea33045f4e2

Maybe it's helpful to some of you.

Note that I make the following assumptions (which are reasonable for my use case): * the data variables in each part are identical * equality of the first element of two coordinate arrays is sufficient to assume equality of the two coordinate arrays

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Concatenate across multiple dimensions with open_mfdataset 324350248

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);