html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/444#issuecomment-118373188,https://api.github.com/repos/pydata/xarray/issues/444,118373188,MDEyOklzc3VlQ29tbWVudDExODM3MzE4OA==,306380,2015-07-03T15:26:18Z,2015-07-03T15:26:18Z,MEMBER,"The library itself is not threadsafe?  What about on a per-file basis?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,91184107
https://github.com/pydata/xarray/issues/444#issuecomment-116182511,https://api.github.com/repos/pydata/xarray/issues/444,116182511,MDEyOklzc3VlQ29tbWVudDExNjE4MjUxMQ==,306380,2015-06-28T01:55:39Z,2015-06-28T01:55:39Z,MEMBER,"Oh, I didn't realize that that was built in already.  Sounds like you could
handle this easily on the xray side.
On Jun 27, 2015 4:40 PM, ""Stephan Hoyer"" notifications@github.com wrote:

> Of course, concurrent access to HDF5 files works fine on my laptop, using
> Anaconda's build of HDF5 (version 1.8.14). I have no idea what special
> flags they invoked when building it :).
> 
> That said, I have been unable to produce any benchmarks that show improved
> performance when simply doing multithreaded _reads_ without doing any
> computation (e.g., %time xray.open_dataset(..., chunks=...).load()). Even
> when I'm reading multiple independent chunks compressed on disk, CPU seems
> to be pegged at 100%, when using either netCDF4-python or h5py (via
> h5netcdf) to read the data. For non-compressed data, reads seem to be
> limited by disk speed, so CPU is also not relevant.
> 
> Given these considerations, it seems like we should use a lock when
> reading data into xray with dask. @mrocklin https://github.com/mrocklin
> we could just use lock=True with da.from_array, right? If we can find use
> cases for multi-threaded reads, we could also add an optional lock
> argument to open_dataset/open_mfdataset.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/xray/xray/issues/444#issuecomment-116165986.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,91184107
https://github.com/pydata/xarray/issues/444#issuecomment-116162351,https://api.github.com/repos/pydata/xarray/issues/444,116162351,MDEyOklzc3VlQ29tbWVudDExNjE2MjM1MQ==,306380,2015-06-27T22:12:37Z,2015-06-27T22:12:37Z,MEMBER,"There was a similar problem with PyTables, which didn't support concurrency
well.  This resulted in the from-hdf5 function in dask array which uses
explicit locks to avoid concurrent access.

We could repeat this treatment more generally without much trouble to force
single threaded access on access but still allow parallelism otherwise.
On Jun 27, 2015 2:33 PM, ""Răzvan Rădulescu"" notifications@github.com
wrote:

> So I just tried @mrocklin https://github.com/mrocklin's idea with using
> single-threaded stuff. This seems to fix the segmentation fault, but I am
> very curious as to why there's a problem with working in parallel. I tried
> two different hdf5 libraries (I think version 1.8.13 and 1.8.14) but I got
> the same segmentation fault. Anyway, working on a single thread is not a
> big deal, I'll just do that for the time being... I already tried gdb on
> python but I'm not experienced enough to make heads or tails of it... I
> have the gdb backtrace here
> https://gist.github.com/razvanc87/0986c4f7a591772e1778 but I don't know
> what to do with it...
> 
> @shoyer https://github.com/shoyer, the files are not the issue here,
> they're the same ones I provided in #443
> https://github.com/xray/xray/issues/443.
> 
> Question: does the hdf5 library need to be built with parallel support
> (mpi or something) maybe?... thanks guys
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/xray/xray/issues/444#issuecomment-116146897.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,91184107
https://github.com/pydata/xarray/issues/444#issuecomment-115930797,https://api.github.com/repos/pydata/xarray/issues/444,115930797,MDEyOklzc3VlQ29tbWVudDExNTkzMDc5Nw==,306380,2015-06-27T01:09:44Z,2015-06-27T01:09:44Z,MEMBER,"Alternatively can we try doing the operations that xray would do manually and see if one of them triggers something?  

One could also try

```
$ gdb python
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,91184107
https://github.com/pydata/xarray/issues/444#issuecomment-115930685,https://api.github.com/repos/pydata/xarray/issues/444,115930685,MDEyOklzc3VlQ29tbWVudDExNTkzMDY4NQ==,306380,2015-06-27T01:08:13Z,2015-06-27T01:08:13Z,MEMBER,"@shoyer asked me to chime in in case this is an issue with dask.  One thing to try would be to remove multi-threading from the equation.  I'm not sure how this would affect things but it's worth a shot.

``` python
>>> import dask
>>> from dask.async import get_sync
>>> dask.set_options(get=get_sync)  # use single-threaded scheduler by default
>>> ... do work as normal
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,91184107