home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 270193755

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/pull/1179#issuecomment-270193755 https://api.github.com/repos/pydata/xarray/issues/1179 270193755 MDEyOklzc3VlQ29tbWVudDI3MDE5Mzc1NQ== 1197350 2017-01-03T18:59:10Z 2017-01-03T18:59:10Z MEMBER

Is there a clear fail case we can use as a test to demonstrate the value here?

I have found a fail case related to distributed: attempting to use to_netcdf() with a dask.distributed client fails because the threading.Lock() can't be serialized. A SerializableLock would overcome this problem.

Consider this example:

```python import dask.array as da from distributed import Client import xarray as xr

def create_and_store_dataset(): shape = (10000, 1000) chunks = (1000, 1000) data = da.zeros(shape, chunks=chunks) ds = xr.DataArray(data).to_dataset() ds.to_netcdf('test_dataset.nc') print("Success!")

create_and_store_dataset() client = Client() create_and_store_dataset() ```

The first call succeeds, while the second fails with TypeError: can't pickle thread.lock objects.

When using the distributed client, I can successfully call .store on the underlying dask array if I pass lock=SerializableLock().

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  197083082
Powered by Datasette · Queries took 0.685ms · About: xarray-datasette