issue_comments
3 rows where issue = 345715825 and user = 1197350 sorted by updated_at descending
This data as json, CSV (advanced)
Suggested facets: created_at (date), updated_at (date)
issue 1
- Out-of-core processing with dask not working properly? · 3 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | performed_via_github_app | issue |
---|---|---|---|---|---|---|---|---|---|---|---|
408928221 | https://github.com/pydata/xarray/issues/2329#issuecomment-408928221 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwODkyODIyMQ== | rabernat 1197350 | 2018-07-30T16:37:05Z | 2018-07-30T16:37:23Z | MEMBER | Can you forget about zarr for a moment and just do a reduction on your dataset? For example:
Keep the same chunk arguments you are currently using. This will help us understand if the problem is with reading the files. Is it your intention to chunk the files contiguously in time? Depending on the underlying structure of the data within the netCDF file, this could amount to a complete transposition of the data, which could be very slow / expensive. This could have some parallels with #2004. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
408925488 | https://github.com/pydata/xarray/issues/2329#issuecomment-408925488 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwODkyNTQ4OA== | rabernat 1197350 | 2018-07-30T16:28:31Z | 2018-07-30T16:28:31Z | MEMBER |
Yes, this is what we want! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
408860643 | https://github.com/pydata/xarray/issues/2329#issuecomment-408860643 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwODg2MDY0Mw== | rabernat 1197350 | 2018-07-30T13:20:59Z | 2018-07-30T13:20:59Z | MEMBER | @lrntct - this sounds like a reasonable way to use zarr. We routinely do this sort of transcoding and it works reasonable well. Unfortunately something clearly isn't working right in your case. These things can be hard to debug, but we will try to help you. You might want to start by reviewing the guide I wrote for Pangeo on preparing zarr datasets. It would also be good to see a bit more detail. You posted a function If instead you have just one big netCDF file as in the example you posted above, I think I see you problem: you are calling More ideas:
- explicitly specify the chunks (rather than using Another useful piece of advice would be to use the dask distributed dashboard to monitor what is happening under the hood. You can do this by running
Hopefully these ideas can help you move forward. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [performed_via_github_app] TEXT, [issue] INTEGER REFERENCES [issues]([id]) ); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user 1