issue_comments

Table actions

11 rows where issue = 196541604 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

author_association 3

issue 1

  • Some queries · 11 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
453804715 https://github.com/pydata/xarray/issues/1173#issuecomment-453804715 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDQ1MzgwNDcxNQ== jhamman 2443309 2019-01-13T06:00:22Z 2019-01-13T06:00:22Z MEMBER

Closing this old issue. We've taken care of most of the issues discussed here through the various backend updates over the past two years.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268705335 https://github.com/pydata/xarray/issues/1173#issuecomment-268705335 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODcwNTMzNQ== shoyer 1217238 2016-12-22T02:51:29Z 2016-12-22T02:51:29Z MEMBER

1179 will make use of SerializableLock() for our default netCDF4/HDF5 lock.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268359031 https://github.com/pydata/xarray/issues/1173#issuecomment-268359031 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODM1OTAzMQ== pwolfram 4295853 2016-12-20T21:03:31Z 2016-12-20T21:03:31Z CONTRIBUTOR

@JoyMonteiro and @shoyer, as I've been thinking about this more and especially regarding #463, I was planning on building on opener from #1128 to essentially open, read, and then close a file each time a read get operation was needed on a newCDF file. My initial view was that output fundamentally would be serial but as @JoyMonteiro points out, there may be a benefit to making a provision for parallel output. However, we will probably run into the same netCDF limitation on the number of open files. Would we want similar functionality on opener for set as well as the get methods? I'm not sure how something like sync would work in this context and suspect this could lead to problems. Presumably we would be requiring writing each dimension, attribute, variable, etc at each call with its own associated open, write, and close. I obviously need to find the time to dig into this further...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268234430 https://github.com/pydata/xarray/issues/1173#issuecomment-268234430 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODIzNDQzMA== JoyMonteiro 7300413 2016-12-20T12:44:25Z 2016-12-20T12:44:25Z NONE

Playing around with things sounds like much more fun :) I can see how this will be useful, will start thinking of some test cases to code.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268120151 https://github.com/pydata/xarray/issues/1173#issuecomment-268120151 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODEyMDE1MQ== mrocklin 306380 2016-12-20T00:36:32Z 2016-12-20T00:36:32Z MEMBER

There have been some efforts and progress in using many NetCDF files on a distributed POSIX filesystem (NFS, gluster, not HDFS) but there is still some pain here. We should probably circle back up and figure out what still needs to be done (do you have a firm understanding of this @shoyer ?)

HDF5 on HDFS is, I suspect, sufficiently painful so that I would be more tempted to either avoid HDFS or to try other formats like ZArr (which I'm somewhat biased towards) (cc @alimanfoo). However my experience has been that most climate data lives on a POSIX file system, so experimentation here may not be high priority.

@JoyMonteiro if you have time then the first thing to do is to probably start using things and report where they're broken. I'm confident that small things will present themselves quickly :)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268119475 https://github.com/pydata/xarray/issues/1173#issuecomment-268119475 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODExOTQ3NQ== mrocklin 306380 2016-12-20T00:32:40Z 2016-12-20T00:32:40Z MEMBER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268113379 https://github.com/pydata/xarray/issues/1173#issuecomment-268113379 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODExMzM3OQ== JoyMonteiro 7300413 2016-12-19T23:54:13Z 2016-12-19T23:54:13Z NONE

Thanks. how big of an endeavour is this? I see some free time from 2-3rd week of Jan, and I could maybe contribute towards making this happen.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268105839 https://github.com/pydata/xarray/issues/1173#issuecomment-268105839 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODEwNTgzOQ== shoyer 1217238 2016-12-19T23:11:05Z 2016-12-19T23:11:05Z MEMBER

I don't know if anyone has tested writing netCDFs with dask.distributed yet. I suspect the only immediate issue would be errors from dask because threading.Lock() can't be pickled. We need to switch to using dask's SerializableLock() to make dask.distributed work smoothly.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268105389 https://github.com/pydata/xarray/issues/1173#issuecomment-268105389 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODEwNTM4OQ== JoyMonteiro 7300413 2016-12-19T23:08:38Z 2016-12-19T23:08:38Z NONE

@shoyer: does this also work with dask.distributed? The doc seems to only mention a thread pool.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268104755 https://github.com/pydata/xarray/issues/1173#issuecomment-268104755 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODEwNDc1NQ== JoyMonteiro 7300413 2016-12-19T23:05:16Z 2016-12-19T23:05:16Z NONE

did not know about that, thanks!!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604
268103602 https://github.com/pydata/xarray/issues/1173#issuecomment-268103602 https://api.github.com/repos/pydata/xarray/issues/1173 MDEyOklzc3VlQ29tbWVudDI2ODEwMzYwMg== shoyer 1217238 2016-12-19T22:59:22Z 2016-12-19T22:59:22Z MEMBER

For (1), take a look at save_mfdataset for saving to multiple files.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Some queries 196541604

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);