issue_comments
5 rows where author_association = "MEMBER", issue = 345715825 and user = 10050469 sorted by updated_at descending
This data as json, CSV (advanced)
Suggested facets: reactions, created_at (date), updated_at (date)
issue 1
- Out-of-core processing with dask not working properly? · 5 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | performed_via_github_app | issue |
---|---|---|---|---|---|---|---|---|---|---|---|
409238042 | https://github.com/pydata/xarray/issues/2329#issuecomment-409238042 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwOTIzODA0Mg== | fmaussion 10050469 | 2018-07-31T14:20:06Z | 2018-07-31T14:20:06Z | MEMBER | I updated my example above to show that the chunking over the last dimension is ridiculously slow. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
409172635 | https://github.com/pydata/xarray/issues/2329#issuecomment-409172635 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwOTE3MjYzNQ== | fmaussion 10050469 | 2018-07-31T10:25:16Z | 2018-07-31T14:18:29Z | MEMBER | Sorry for the confusion, I had an obvious mistake in my timing experiment above (forgot to do the actual computations...). The dimension order does make a difference: ```python import dask as da import xarray as xr d = xr.DataArray(da.array.zeros((1000, 721, 1440), chunks=(10, 721, 1440)), dims=('z', 'y', 'x')) d.to_netcdf('da.nc') # 8.3 Gb with xr.open_dataarray('da.nc', chunks={'z':10}) as d: %timeit d.sum().load() 3.94 s ± 95.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) with xr.open_dataarray('da.nc', chunks={'y':10}) as d: %timeit d.sum().load() 4.15 s ± 316 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) with xr.open_dataarray('da.nc', chunks={'x':10}) as d: %timeit d.sum().load() 1min 54s ± 1.43 s per loop (mean ± std. dev. of 7 runs, 1 loop each) with xr.open_dataarray('da.nc', chunks={'y':10, 'x':10}) as d: %timeit d.sum().load() 2min 23s ± 215 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
409168605 | https://github.com/pydata/xarray/issues/2329#issuecomment-409168605 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwOTE2ODYwNQ== | fmaussion 10050469 | 2018-07-31T10:09:36Z | 2018-07-31T13:21:34Z | MEMBER |
Can you still try to chunk along one dimension only? i.e. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
409165114 | https://github.com/pydata/xarray/issues/2329#issuecomment-409165114 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwOTE2NTExNA== | fmaussion 10050469 | 2018-07-31T09:56:54Z | 2018-07-31T10:20:32Z | MEMBER | [EDIT]: forgot the load ... <s> forget my comment about chunks - I thought this would make a difference but it's actually the opposite (to my surprise): </s> |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 | |
409159969 | https://github.com/pydata/xarray/issues/2329#issuecomment-409159969 | https://api.github.com/repos/pydata/xarray/issues/2329 | MDEyOklzc3VlQ29tbWVudDQwOTE1OTk2OQ== | fmaussion 10050469 | 2018-07-31T09:38:37Z | 2018-07-31T10:19:37Z | MEMBER | Out of curiosity:
- why do you chunk over lats and lons rather than time? The order of dimensions in your dataarray suggest that chunking over time could be more efficient
- can you show the output of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Out-of-core processing with dask not working properly? 345715825 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [performed_via_github_app] TEXT, [issue] INTEGER REFERENCES [issues]([id]) ); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user 1