home / github / issue_comments

Menu
  • Search all tables
  • GraphQL API

issue_comments: 704530619

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4112#issuecomment-704530619 https://api.github.com/repos/pydata/xarray/issues/4112 704530619 MDEyOklzc3VlQ29tbWVudDcwNDUzMDYxOQ== 14314623 2020-10-06T20:20:34Z 2020-10-06T20:20:34Z CONTRIBUTOR

Just tried this with the newest dask version and can confirm that I do not get huge chunks anymore IF i specify dask.config.set({"array.slicing.split_large_chunks": True}). I also needed to modify the example to exceed the internal chunk size limitation: ```python import numpy as np import xarray as xr import dask dask.config.set({"array.slicing.split_large_chunks": True})

short_time = xr.cftime_range('2000', periods=12) long_time = xr.cftime_range('2000', periods=120)

data_short = np.random.rand(len(short_time)) data_long = np.random.rand(len(long_time)) n=1000 a = xr.DataArray(data_short, dims=['time'], coords={'time':short_time}).expand_dims(a=n, b=n).chunk({'time':3}) b = xr.DataArray(data_long, dims=['time'], coords={'time':long_time}).expand_dims(a=n, b=n).chunk({'time':3})

a,b = xr.align(a,b, join = 'outer') `` with the option turned on I get this fora`;

with the defaults, I still get one giant chunk.

Ill try this soon in a real world scenario described above. Just wanted to report back here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  627600168
Powered by Datasette · Queries took 0.888ms · About: xarray-datasette