html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1173#issuecomment-453804715,https://api.github.com/repos/pydata/xarray/issues/1173,453804715,MDEyOklzc3VlQ29tbWVudDQ1MzgwNDcxNQ==,2443309,2019-01-13T06:00:22Z,2019-01-13T06:00:22Z,MEMBER,Closing this old issue. We've taken care of most of the issues discussed here through the various backend updates over the past two years.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268705335,https://api.github.com/repos/pydata/xarray/issues/1173,268705335,MDEyOklzc3VlQ29tbWVudDI2ODcwNTMzNQ==,1217238,2016-12-22T02:51:29Z,2016-12-22T02:51:29Z,MEMBER,#1179 will make use of `SerializableLock()` for our default netCDF4/HDF5 lock.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268359031,https://api.github.com/repos/pydata/xarray/issues/1173,268359031,MDEyOklzc3VlQ29tbWVudDI2ODM1OTAzMQ==,4295853,2016-12-20T21:03:31Z,2016-12-20T21:03:31Z,CONTRIBUTOR,"@JoyMonteiro and @shoyer, as I've been thinking about this more and especially regarding #463, I was planning on building on opener from #1128 to essentially open, read, and then close a file each time a read `get` operation was needed on a newCDF file. My initial view was that output fundamentally would be serial but as @JoyMonteiro points out, there may be a benefit to making a provision for parallel output. However, we will probably run into the same netCDF limitation on the number of open files. Would we want similar functionality on `opener` for `set` as well as the `get` methods? I'm not sure how something like `sync` would work in this context and suspect this could lead to problems. Presumably we would be requiring writing each dimension, attribute, variable, etc at each call with its own associated open, write, and close. I obviously need to find the time to dig into this further...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268234430,https://api.github.com/repos/pydata/xarray/issues/1173,268234430,MDEyOklzc3VlQ29tbWVudDI2ODIzNDQzMA==,7300413,2016-12-20T12:44:25Z,2016-12-20T12:44:25Z,NONE,"Playing around with things sounds like much more fun :) I can see how this will be useful, will start thinking of some test cases to code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268120151,https://api.github.com/repos/pydata/xarray/issues/1173,268120151,MDEyOklzc3VlQ29tbWVudDI2ODEyMDE1MQ==,306380,2016-12-20T00:36:32Z,2016-12-20T00:36:32Z,MEMBER,"There have been some efforts and progress in using many NetCDF files on a distributed POSIX filesystem (NFS, gluster, not HDFS) but there is still some pain here. We should probably circle back up and figure out what still needs to be done (do you have a firm understanding of this @shoyer ?)
HDF5 on HDFS is, I suspect, sufficiently painful so that I would be more tempted to either avoid HDFS or to try other formats like ZArr (which I'm somewhat biased towards) (cc @alimanfoo). However my experience has been that most climate data lives on a POSIX file system, so experimentation here may not be high priority.
@JoyMonteiro if you have time then the first thing to do is to probably start using things and report where they're broken. I'm confident that small things will present themselves quickly :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268119475,https://api.github.com/repos/pydata/xarray/issues/1173,268119475,MDEyOklzc3VlQ29tbWVudDI2ODExOTQ3NQ==,306380,2016-12-20T00:32:40Z,2016-12-20T00:32:40Z,MEMBER,"Some related issues:
https://github.com/pydata/xarray/issues/798
https://github.com/dask/distributed/issues/629
Will write more shortly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268113379,https://api.github.com/repos/pydata/xarray/issues/1173,268113379,MDEyOklzc3VlQ29tbWVudDI2ODExMzM3OQ==,7300413,2016-12-19T23:54:13Z,2016-12-19T23:54:13Z,NONE,"Thanks. how big of an endeavour is this? I see some free time from 2-3rd week of Jan, and
I could maybe contribute towards making this happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268105839,https://api.github.com/repos/pydata/xarray/issues/1173,268105839,MDEyOklzc3VlQ29tbWVudDI2ODEwNTgzOQ==,1217238,2016-12-19T23:11:05Z,2016-12-19T23:11:05Z,MEMBER,I don't know if anyone has tested writing netCDFs with dask.distributed yet. I suspect the only immediate issue would be errors from dask because `threading.Lock()` can't be pickled. We need to switch to using dask's `SerializableLock()` to make dask.distributed work smoothly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268105389,https://api.github.com/repos/pydata/xarray/issues/1173,268105389,MDEyOklzc3VlQ29tbWVudDI2ODEwNTM4OQ==,7300413,2016-12-19T23:08:38Z,2016-12-19T23:08:38Z,NONE,@shoyer: does this also work with dask.distributed? The doc seems to only mention a thread pool.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268104755,https://api.github.com/repos/pydata/xarray/issues/1173,268104755,MDEyOklzc3VlQ29tbWVudDI2ODEwNDc1NQ==,7300413,2016-12-19T23:05:16Z,2016-12-19T23:05:16Z,NONE,"did not know about that, thanks!!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604
https://github.com/pydata/xarray/issues/1173#issuecomment-268103602,https://api.github.com/repos/pydata/xarray/issues/1173,268103602,MDEyOklzc3VlQ29tbWVudDI2ODEwMzYwMg==,1217238,2016-12-19T22:59:22Z,2016-12-19T22:59:22Z,MEMBER,"For (1), take a look at [save_mfdataset](http://xarray.pydata.org/en/stable/generated/xarray.save_mfdataset.html) for saving to multiple files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,196541604