html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/1128#issuecomment-265878280,https://api.github.com/repos/pydata/xarray/issues/1128,265878280,MDEyOklzc3VlQ29tbWVudDI2NTg3ODI4MA==,1217238,2016-12-08T22:44:12Z,2016-12-08T22:44:12Z,MEMBER,"@mangecoeur You still need to use `lock=False` (or `lock=dask.utils.SerializableLock()` with the dev version of dask) and use a spawning process pool (https://github.com/pydata/xarray/pull/1128#issuecomment-261936849).
The former should be updated internally, and the later should be a documentation note.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-264033283,https://api.github.com/repos/pydata/xarray/issues/1128,264033283,MDEyOklzc3VlQ29tbWVudDI2NDAzMzI4Mw==,1217238,2016-11-30T23:44:54Z,2016-11-30T23:44:54Z,MEMBER,"OK, in it goes!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-263927223,https://api.github.com/repos/pydata/xarray/issues/1128,263927223,MDEyOklzc3VlQ29tbWVudDI2MzkyNzIyMw==,1217238,2016-11-30T16:50:48Z,2016-11-30T16:50:48Z,MEMBER,@kynan @crusaderky Do you have concerns about merging this in the current state?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-263926969,https://api.github.com/repos/pydata/xarray/issues/1128,263926969,MDEyOklzc3VlQ29tbWVudDI2MzkyNjk2OQ==,1217238,2016-11-30T16:49:53Z,2016-11-30T16:49:53Z,MEMBER,"I decided that between the choices of not running these tests on Windows and leaking a few temp files, I would rather leak some temp files. So that's exactly what I've done in the latest commit, for explicitly whitelisted tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-263345757,https://api.github.com/repos/pydata/xarray/issues/1128,263345757,MDEyOklzc3VlQ29tbWVudDI2MzM0NTc1Nw==,1217238,2016-11-28T18:04:17Z,2016-11-28T18:04:17Z,MEMBER,"@mrocklin OK, so one option is to just ignore the permission errors and not remove the files on Windows. But is it really better to make the test suite leak temp files?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-263344764,https://api.github.com/repos/pydata/xarray/issues/1128,263344764,MDEyOklzc3VlQ29tbWVudDI2MzM0NDc2NA==,1217238,2016-11-28T18:00:38Z,2016-11-28T18:00:38Z,MEMBER,"OK, I'm ready to give up on the remaining test failures and merge this anyways (marking them as expected failures). They are specific to our test suite and for Windows only, due to the inability to delete files that are not closed.
If these manifest themselves as issues for real users, I am happy to revisit, especially if someone who uses Windows can help debug. The 5 minute feedback cycle of pushing a commit and then seeing what Appveyor says is too painful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-261980869,https://api.github.com/repos/pydata/xarray/issues/1128,261980869,MDEyOklzc3VlQ29tbWVudDI2MTk4MDg2OQ==,1217238,2016-11-21T16:04:14Z,2016-11-21T16:04:14Z,MEMBER,"> Does your failure work with the following spawning pool in Python 3?
Why, yes it does -- and it shows a nice speedup, as well! What was I missing here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-261841025,https://api.github.com/repos/pydata/xarray/issues/1128,261841025,MDEyOklzc3VlQ29tbWVudDI2MTg0MTAyNQ==,1217238,2016-11-21T04:36:02Z,2016-11-21T04:36:02Z,MEMBER,"This isn't yet working with dask multiprocessing for reading a netCDF4 file with in-memory compression. I'm not quite sure why:
```
In [5]: from multiprocessing.pool import Pool
In [7]: ds = xr.open_dataset('big-random.nc', lock=False, chunks={'x': 2500})
In [8]: dask.set_options(pool=Pool(4))
Out[8]:
In [9]: %time ds.sum().compute()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in ()
----> 1 get_ipython().magic('time ds.sum().compute()')
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py in magic(self, arg_s)
2156 magic_name, _, magic_arg_s = arg_s.partition(' ')
2157 magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2158 return self.run_line_magic(magic_name, magic_arg_s)
2159
2160 #-------------------------------------------------------------------------
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line)
2077 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2078 with self.builtin_trap:
-> 2079 result = fn(*args,**kwargs)
2080 return result
2081
in time(self, line, cell, local_ns)
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/IPython/core/magic.py in (f, *a, **k)
186 # but it's overkill for just that one bit of state.
187 def magic_deco(arg):
--> 188 call = lambda f, *a, **k: f(*a, **k)
189
190 if callable(arg):
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
1174 if mode=='eval':
1175 st = clock2()
-> 1176 out = eval(code, glob, local_ns)
1177 end = clock2()
1178 else:
in ()
/Users/shoyer/dev/xarray/xarray/core/dataset.py in compute(self)
348 """"""
349 new = self.copy(deep=False)
--> 350 return new.load()
351
352 @classmethod
/Users/shoyer/dev/xarray/xarray/core/dataset.py in load(self)
325
326 # evaluate all the dask arrays simultaneously
--> 327 evaluated_data = da.compute(*lazy_data.values())
328
329 for k, data in zip(lazy_data, evaluated_data):
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/base.py in compute(*args, **kwargs)
176 dsk = merge(var.dask for var in variables)
177 keys = [var._keys() for var in variables]
--> 178 results = get(dsk, keys, **kwargs)
179
180 results_iter = iter(results)
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, **kwargs)
67 results = get_async(pool.apply_async, len(pool._pool), dsk, result,
68 cache=cache, get_id=_thread_get_id,
---> 69 **kwargs)
70
71 # Cleanup pools associated to dead threads
/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, dumps, loads, **kwargs)
500 _execute_task(task, data) # Re-execute locally
501 else:
--> 502 raise(remote_exception(res, tb))
503 state['cache'][key] = res
504 finish_task(dsk, key, state, results, keyorder.get)
RuntimeError: NetCDF: HDF error
Traceback
---------
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 268, in execute_task
result = _execute_task(task, data)
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 248, in _execute_task
args2 = [_execute_task(a, cache) for a in args]
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 248, in
args2 = [_execute_task(a, cache) for a in args]
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 245, in _execute_task
return [_execute_task(a, cache) for a in arg]
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 245, in
return [_execute_task(a, cache) for a in arg]
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/async.py"", line 249, in _execute_task
return func(*args2)
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/dask/array/core.py"", line 51, in getarray
c = np.asarray(c)
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/numpy/core/numeric.py"", line 482, in asarray
return array(a, dtype, copy=False, order=order)
File ""/Users/shoyer/dev/xarray/xarray/core/indexing.py"", line 417, in __array__
return np.asarray(self.array, dtype=dtype)
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/numpy/core/numeric.py"", line 482, in asarray
return array(a, dtype, copy=False, order=order)
File ""/Users/shoyer/dev/xarray/xarray/core/indexing.py"", line 392, in __array__
return np.asarray(array[self.key], dtype=None)
File ""/Users/shoyer/conda/envs/xarray-dev/lib/python3.5/site-packages/numpy/core/numeric.py"", line 482, in asarray
return array(a, dtype, copy=False, order=order)
File ""/Users/shoyer/dev/xarray/xarray/core/indexing.py"", line 392, in __array__
return np.asarray(array[self.key], dtype=None)
File ""/Users/shoyer/dev/xarray/xarray/backends/netCDF4_.py"", line 56, in __getitem__
data = getitem(self.array, key)
File ""netCDF4/_netCDF4.pyx"", line 3695, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:37914)
File ""netCDF4/_netCDF4.pyx"", line 4376, in netCDF4._netCDF4.Variable._get (netCDF4/_netCDF4.c:47134)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-261837981,https://api.github.com/repos/pydata/xarray/issues/1128,261837981,MDEyOklzc3VlQ29tbWVudDI2MTgzNzk4MQ==,1217238,2016-11-21T04:08:22Z,2016-11-21T04:11:30Z,MEMBER,"I added pickle support to DataStores. This *should* solve the basic serialization issue for dask.distributed (#798), but does not yet resolve the ""too many open files"" issue.
@mrocklin this could use your review.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-261755551,https://api.github.com/repos/pydata/xarray/issues/1128,261755551,MDEyOklzc3VlQ29tbWVudDI2MTc1NTU1MQ==,1217238,2016-11-20T03:13:30Z,2016-11-20T03:13:30Z,MEMBER,"I removed the custom pickle override on `Dataset`/`DataArray` -- the issue I was working around was actually a indirect manifestation of bug on `IndexVariable.load()` (introduced in this PR).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033
https://github.com/pydata/xarray/pull/1128#issuecomment-261433336,https://api.github.com/repos/pydata/xarray/issues/1128,261433336,MDEyOklzc3VlQ29tbWVudDI2MTQzMzMzNg==,1217238,2016-11-18T02:36:21Z,2016-11-18T02:36:21Z,MEMBER,"> In the long run I think it would be more robust to check for attributes (duck type style) rather than types in the various places.
Indeed, in particular I'm not very happy with the `isinstance` check for `indexing.MemoryCachedArray` in `Variable.copy()` -- it's rather poor separation of concerns.
It exists so that `variable.compute()` does not cache data in-memory on `variable` but only on the computed variable. Otherwise, there's basically no point to the separate compute method: if you use `cache=True`, you are stuck with caching on the original object. Likewise, it ensures that `.copy()` creates an array with a new cache, which is consistent with the current behavior of `.copy()`.
As for type checking for dask arrays in `.data`: yes, it would be nice to have a well defined array interface layer that other array types could plug into. That would entail a significant amount of further work, however.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,189817033