home / github / issue_comments

Menu
  • Search all tables
  • GraphQL API

issue_comments: 358055204

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/1832#issuecomment-358055204 https://api.github.com/repos/pydata/xarray/issues/1832 358055204 MDEyOklzc3VlQ29tbWVudDM1ODA1NTIwNA== 306380 2018-01-16T18:15:03Z 2018-01-16T18:15:35Z MEMBER

This example is an interesting one that was adapted from something that @rabernat produced

```python import dask import xarray as xr import dask.array as da import pandas as pd from tornado import gen

from dask.distributed import Client client = Client(processes=False)

below I create a random dataset that is typical of high-res climate models

size of example can be adjusted up and down by changing shape

dims = ('time', 'depth', 'lat', 'lon') time = pd.date_range('1980-01-01', '1980-12-01', freq='1d') shape = (len(time), 5, 1800, 360)

what I consider to be a reasonable chunk size

chunks = (1, 1, 1800, 360) ds = xr.Dataset({k: (dims, da.random.random(shape, chunks=chunks)) for k in ['u', 'v', 'w']}, coords={'time': time})

create seasonal climatology

ds_clim = ds.groupby('time.week').mean(dim='time')

construct seasonal anomaly

ds_anom = ds.groupby('time.week') - ds_clim

compute variance of seasonal anomaly

ds_anom_var = (ds_anom**2).mean(dim='time') ds_anom_var.compute() ```

It works fine locally with processes=False and poorly with processes=True. If anyone has time to help on this issue I recommend investigating what is different in these two cases. If I had time I would start here by trying to improve our understanding with better visual diagnostics, although just poring over logs might also provide some insight.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  288785270
Powered by Datasette · Queries took 0.687ms · About: xarray-datasette