home / github / issue_comments

Menu
  • Search all tables
  • GraphQL API

issue_comments: 1254873700

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/7059#issuecomment-1254873700 https://api.github.com/repos/pydata/xarray/issues/7059 1254873700 IC_kwDOAMm_X85Ky9pk 691772 2022-09-22T11:09:16Z 2022-09-22T11:09:16Z CONTRIBUTOR

I have managed to reduce the reproducing example (see "Minimal Complete Verifiable Example 2" above) and then also find a proper solution to fix this issue. I am still not sure whether this is a bug or intended behavior, so I'll won't close the issue for now.

Basically the issue occurs when a chunked NetCDF file is loaded from disk, passed to xarray.map_blocks() and is then used in .sel() as parameter to get a subset of some other xarray object which is not passed to the worker func(). I think the proper solution is to use the args parameter of map_blocks() instead of .sel():

``` --- run-broken.py 2022-09-22 13:00:41.095555961 +0200 +++ run.py 2022-09-22 13:01:14.452696511 +0200 @@ -30,17 +30,17 @@ def resample_annually(data): return data.sortby("time").resample(time="1A", label="left", loffset="1D").mean(dim="time")

  • def worker(data):
  • locations_chunk = locations.sel(locations=data.locations)
  • out_raw = data * locations_chunk
  • def worker(data, locations):
  • out_raw = data * locations out = resample_annually(out_raw) return out

    template = resample_annually(data)

    out = xr.map_blocks( - lambda data: worker(data).compute().chunk({"time": None}), + lambda data, locations: worker(data, locations).compute().chunk({"time": None}), data, + (locations,), template=template, ) ```

This seems to fix this issue and seems to be the proper solution anyway.

I still don't see why I am not allowed to use .sel() on shadowed objects in the worker func()ยด. Is this on purpose? If yes, should we add something to the documentation? Is this a specific behavior ofmap_blocks()`? Is it related to #6904?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  1379372915
Powered by Datasette · Queries took 0.848ms · About: xarray-datasette