issues: 1975574237
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1975574237 | I_kwDOAMm_X851wN7d | 8409 | Task graphs on `.map_blocks` with many chunks can be huge | 5635139 | closed | 0 | 6 | 2023-11-03T07:14:45Z | 2024-01-03T04:10:16Z | 2024-01-03T04:10:16Z | MEMBER | What happened?I'm getting task graphs > 1GB, I think possibly because the full indexes are being included in every task? What did you expect to happen?Only the relevant sections of the index would be included Minimal Complete Verifiable Example```Python da = xr.tutorial.load_dataset('air_temperature') Dropping the index doesn't generally matter that much...len(cloudpickle.dumps(da.chunk(lat=1, lon=1))) 15569320len(cloudpickle.dumps(da.chunk().drop_vars(da.indexes))) 15477313But with
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/8409/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | 13221727 | issue |