html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/1198#issuecomment-288832565,https://api.github.com/repos/pydata/xarray/issues/1198,288832565,MDEyOklzc3VlQ29tbWVudDI4ODgzMjU2NQ==,4295853,2017-03-23T19:21:25Z,2017-03-23T19:21:25Z,CONTRIBUTOR,Thanks a bunch @shoyer!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288832196,https://api.github.com/repos/pydata/xarray/issues/1198,288832196,MDEyOklzc3VlQ29tbWVudDI4ODgzMjE5Ng==,1217238,2017-03-23T19:19:59Z,2017-03-23T19:19:59Z,MEMBER,"OK, in it goes!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288796995,https://api.github.com/repos/pydata/xarray/issues/1198,288796995,MDEyOklzc3VlQ29tbWVudDI4ODc5Njk5NQ==,4295853,2017-03-23T17:26:23Z,2017-03-23T17:26:23Z,CONTRIBUTOR,"@shoyer, all tests (including coveralls) passed. Please let me know if you have additional concerns and if we could merge fairly soon, e.g., because of https://github.com/MPAS-Dev/MPAS-Analysis/issues/151 I would really appreciate it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288795432,https://api.github.com/repos/pydata/xarray/issues/1198,288795432,MDEyOklzc3VlQ29tbWVudDI4ODc5NTQzMg==,4295853,2017-03-23T17:22:11Z,2017-03-23T17:22:11Z,CONTRIBUTOR,"@shoyer, this is ready for the final review now. Coveralls appears to have hung but other tests pass.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288791152,https://api.github.com/repos/pydata/xarray/issues/1198,288791152,MDEyOklzc3VlQ29tbWVudDI4ODc5MTE1Mg==,4295853,2017-03-23T17:08:41Z,2017-03-23T17:08:41Z,CONTRIBUTOR,"@shoyer, I had a minor bug that is now removed. The last caveat no longer applicable:
> The scipy backend can handle objects like BytesIO that really aren't file handles and there doesn't appear to be a clean way to close these types of objects. So, at present I'm explicitly setting _autoclose=False if they are encountered in the datastore. If this needs to be changed, particularly since it doesn't affect existing behavior, I'd prefer this be resolved in a separate issue / PR if possible.
I'll let you know when tests pass and this is ready for your final review.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288782907,https://api.github.com/repos/pydata/xarray/issues/1198,288782907,MDEyOklzc3VlQ29tbWVudDI4ODc4MjkwNw==,4295853,2017-03-23T16:44:44Z,2017-03-23T16:44:44Z,CONTRIBUTOR,"@shoyer, that subclass-based approach you outlined worked (fixture parameters really don't work with classes as far as I could tell). We now have more comprehensive, named testing. Note, there was one minor point that required more explicitly specification that arose from the more rigorous testing:
The scipy backend can handle objects like `BytesIO` that really aren't file handles and there doesn't appear to be a clean way to close these types of objects. So, at present I'm explicitly setting `_autoclose=False` if they are encountered in the datastore. If this needs to be changed, particularly since it doesn't affect existing behavior, I'd prefer this be resolved in a separate issue / PR if possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288559417,https://api.github.com/repos/pydata/xarray/issues/1198,288559417,MDEyOklzc3VlQ29tbWVudDI4ODU1OTQxNw==,1217238,2017-03-22T22:27:36Z,2017-03-22T22:27:36Z,MEMBER,"> @shoyer, if we generally cover test_backends for autoclose=True, then we should get the pickle testing for free
Agreed, that should do it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288559231,https://api.github.com/repos/pydata/xarray/issues/1198,288559231,MDEyOklzc3VlQ29tbWVudDI4ODU1OTIzMQ==,4295853,2017-03-22T22:26:40Z,2017-03-22T22:26:40Z,CONTRIBUTOR,"@shoyer, if we generally cover test_backends for `autoclose=True`, then we should get the pickle testing for free:
```
xarray/tests/test_backends.py:181: def test_pickle(self):
xarray/tests/test_backends.py:191: def test_pickle_dataarray(self):
xarray/tests/test_backends.py:792: def test_bytesio_pickle(self):
```
or was there some other test that is needed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288513495,https://api.github.com/repos/pydata/xarray/issues/1198,288513495,MDEyOklzc3VlQ29tbWVudDI4ODUxMzQ5NQ==,1217238,2017-03-22T19:32:53Z,2017-03-22T19:32:53Z,MEMBER,"Subclasses also work in place of fixtures in many cases (we use them in
much of the existing code).
On Wed, Mar 22, 2017 at 12:30 PM Phillip Wolfram
wrote:
> *@pwolfram* commented on this pull request.
> ------------------------------
>
> In xarray/tests/test_backends.py
> :
>
> >
> with self.assertRaisesRegexp(IOError, 'no files to open'):
> - open_mfdataset('foo-bar-baz-*.nc')
> + for close in [True, False]:
>
> I think I understand how to use fixtures for both of these arguments now...
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288416595,https://api.github.com/repos/pydata/xarray/issues/1198,288416595,MDEyOklzc3VlQ29tbWVudDI4ODQxNjU5NQ==,306380,2017-03-22T14:30:38Z,2017-03-22T14:30:38Z,MEMBER,Just seeing this now. I'm very glad to see progress here. Thanks @pwolfram ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288141361,https://api.github.com/repos/pydata/xarray/issues/1198,288141361,MDEyOklzc3VlQ29tbWVudDI4ODE0MTM2MQ==,4295853,2017-03-21T16:45:57Z,2017-03-21T16:45:57Z,CONTRIBUTOR,"Thanks @shoyer. Ok, so we pass checks following merge of #1311. Is there anything else that we need to do on this PR prior to merging?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288127957,https://api.github.com/repos/pydata/xarray/issues/1198,288127957,MDEyOklzc3VlQ29tbWVudDI4ODEyNzk1Nw==,1217238,2017-03-21T16:04:56Z,2017-03-21T16:04:56Z,MEMBER,"@pwolfram no worries, I am not concerned at all about credit for my commit :). It's possible to do gymnastics with git rebase to preserve history in cases like this but it's questionable if it's worth the trouble.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288102017,https://api.github.com/repos/pydata/xarray/issues/1198,288102017,MDEyOklzc3VlQ29tbWVudDI4ODEwMjAxNw==,4295853,2017-03-21T14:48:21Z,2017-03-21T14:48:21Z,CONTRIBUTOR,"@shoyer, we can roll back the squash if you want because for whatever reason my interactive rebase removed your name from tagged to the commit too, which is not what I expected. This is obviously suboptimal and the choice of having two commits versus one is up to you. I think the commit prior to the squash was `9c274eb`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-288101049,https://api.github.com/repos/pydata/xarray/issues/1198,288101049,MDEyOklzc3VlQ29tbWVudDI4ODEwMTA0OQ==,4295853,2017-03-21T14:45:28Z,2017-03-21T14:45:28Z,CONTRIBUTOR,The tests passed so I will squash the commits...,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287165679,https://api.github.com/repos/pydata/xarray/issues/1198,287165679,MDEyOklzc3VlQ29tbWVudDI4NzE2NTY3OQ==,4295853,2017-03-16T19:27:22Z,2017-03-16T19:27:22Z,CONTRIBUTOR,"@shoyer, all tests are passing now following your recommended edits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287163231,https://api.github.com/repos/pydata/xarray/issues/1198,287163231,MDEyOklzc3VlQ29tbWVudDI4NzE2MzIzMQ==,4295853,2017-03-16T19:17:44Z,2017-03-16T19:17:44Z,CONTRIBUTOR,@shoyer- can you please have another look following edits as you requested?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287142757,https://api.github.com/repos/pydata/xarray/issues/1198,287142757,MDEyOklzc3VlQ29tbWVudDI4NzE0Mjc1Nw==,4295853,2017-03-16T18:03:32Z,2017-03-16T18:03:32Z,CONTRIBUTOR,Note: tests pass before submitting changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287121290,https://api.github.com/repos/pydata/xarray/issues/1198,287121290,MDEyOklzc3VlQ29tbWVudDI4NzEyMTI5MA==,4295853,2017-03-16T16:54:19Z,2017-03-16T16:54:19Z,CONTRIBUTOR,They did- thank you! I'll take a quick look at your comments now so we can hopefully get this merged soon.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287113380,https://api.github.com/repos/pydata/xarray/issues/1198,287113380,MDEyOklzc3VlQ29tbWVudDI4NzExMzM4MA==,1217238,2017-03-16T16:28:58Z,2017-03-16T16:28:58Z,MEMBER,"I restarted all the timed out tests on Travis, hopefully they pass now!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287112491,https://api.github.com/repos/pydata/xarray/issues/1198,287112491,MDEyOklzc3VlQ29tbWVudDI4NzExMjQ5MQ==,4295853,2017-03-16T16:26:09Z,2017-03-16T16:26:09Z,CONTRIBUTOR,"@shoyer, do you mind restarting the Travis-CI?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287100278,https://api.github.com/repos/pydata/xarray/issues/1198,287100278,MDEyOklzc3VlQ29tbWVudDI4NzEwMDI3OA==,4295853,2017-03-16T15:47:53Z,2017-03-16T15:47:53Z,CONTRIBUTOR,"Awesome, thanks @vnoel for testing this for us all on your dataset!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287099757,https://api.github.com/repos/pydata/xarray/issues/1198,287099757,MDEyOklzc3VlQ29tbWVudDI4NzA5OTc1Nw==,731499,2017-03-16T15:46:14Z,2017-03-16T15:46:14Z,CONTRIBUTOR,"with that keyword the open_mfdataset goes through !
my code crashes later because the data is too big to fit in memory, but that's another problem ;-)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287090864,https://api.github.com/repos/pydata/xarray/issues/1198,287090864,MDEyOklzc3VlQ29tbWVudDI4NzA5MDg2NA==,4295853,2017-03-16T15:19:37Z,2017-03-16T15:19:37Z,CONTRIBUTOR,Is something wrong with travis-ci right now? It doesn't look like it is running the tests...,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287061697,https://api.github.com/repos/pydata/xarray/issues/1198,287061697,MDEyOklzc3VlQ29tbWVudDI4NzA2MTY5Nw==,4295853,2017-03-16T13:48:09Z,2017-03-16T13:48:09Z,CONTRIBUTOR,"Note, the force push was just to get travis-ci to ""reboot"". There was no real change to the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286637311,https://api.github.com/repos/pydata/xarray/issues/1198,286637311,MDEyOklzc3VlQ29tbWVudDI4NjYzNzMxMQ==,4295853,2017-03-15T04:22:28Z,2017-03-16T13:47:46Z,CONTRIBUTOR,"Note, if we include `482ef54` for resource-limited testing we will need to do something special for windows on appveyor. We could skip this in the short-term to make this merge and deal with these types of issues in #1309, which is what probably makes the most sense to help keep the scope of this PR more limited.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287061175,https://api.github.com/repos/pydata/xarray/issues/1198,287061175,MDEyOklzc3VlQ29tbWVudDI4NzA2MTE3NQ==,4295853,2017-03-16T13:46:18Z,2017-03-16T13:46:18Z,CONTRIBUTOR,"@vnoel, did you use `autoclose=True`? Can you please share the code snippet you used to instantiate this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-287058651,https://api.github.com/repos/pydata/xarray/issues/1198,287058651,MDEyOklzc3VlQ29tbWVudDI4NzA1ODY1MQ==,731499,2017-03-16T13:37:05Z,2017-03-16T13:37:05Z,CONTRIBUTOR,"Hey @pwolfram, I installed your branch and tried to `open_mfdataset` on 1474 hdf5 files.
I got the following:
```
File ""/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py"", line 524, in open_mfdataset
File ""/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py"", line 524, in
File ""/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py"", line 299, in open_dataset
File ""/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 203, in __init__
File ""/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 178, in _open_netcdf4_group
File ""netCDF4/_netCDF4.pyx"", line 1848, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:13992)
OSError: Too many open files
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286831879,https://api.github.com/repos/pydata/xarray/issues/1198,286831879,MDEyOklzc3VlQ29tbWVudDI4NjgzMTg3OQ==,4295853,2017-03-15T18:12:28Z,2017-03-15T18:12:28Z,CONTRIBUTOR,"@rabernat, I suspect you have some datasets that could be used to stress test this PR too... thanks for the help and advice yesterday.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286831672,https://api.github.com/repos/pydata/xarray/issues/1198,286831672,MDEyOklzc3VlQ29tbWVudDI4NjgzMTY3Mg==,4295853,2017-03-15T18:11:53Z,2017-03-15T18:11:53Z,CONTRIBUTOR,"@PeterDSteinberg and @vnoel, do you mind stress-testing this PR to make sure all works as expected? You can install easily via pip with
`pip -v install git+ssh://git@github.com/pwolfram/xarray@fix_too_many_open_files`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286824521,https://api.github.com/repos/pydata/xarray/issues/1198,286824521,MDEyOklzc3VlQ29tbWVudDI4NjgyNDUyMQ==,4295853,2017-03-15T17:48:42Z,2017-03-15T17:48:42Z,CONTRIBUTOR,"@shoyer, I've collapsed all the commits to a single commit. At this point I would say this meets the scope of this PR and it should be ready to merge once CI finishes its check and it passes. Please feel free to take another pass on the code for changes that you would like to see made.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286636632,https://api.github.com/repos/pydata/xarray/issues/1198,286636632,MDEyOklzc3VlQ29tbWVudDI4NjYzNjYzMg==,4295853,2017-03-15T04:17:01Z,2017-03-15T04:17:01Z,CONTRIBUTOR,"I obviously need to squash prior to the merge, however...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-286636605,https://api.github.com/repos/pydata/xarray/issues/1198,286636605,MDEyOklzc3VlQ29tbWVudDI4NjYzNjYwNQ==,4295853,2017-03-15T04:16:46Z,2017-03-15T04:16:46Z,CONTRIBUTOR,"@shoyer, this should essentially be ready to go now with the exception that I didn't come to a clean resolution on testing the too many open files issue via `resource`. Thoughts on this issue?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-280654955,https://api.github.com/repos/pydata/xarray/issues/1198,280654955,MDEyOklzc3VlQ29tbWVudDI4MDY1NDk1NQ==,4295853,2017-02-17T13:53:09Z,2017-02-17T13:53:09Z,CONTRIBUTOR,"@shoyer, I'd like to see this code integrated if possible. Should we just disable autoclose for h5netcdf and merge?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-278704180,https://api.github.com/repos/pydata/xarray/issues/1198,278704180,MDEyOklzc3VlQ29tbWVudDI3ODcwNDE4MA==,4295853,2017-02-09T17:00:50Z,2017-02-09T17:00:50Z,CONTRIBUTOR,"@shoyer, I'll leave the ball in your court for now but am happy to take another look before we go the disabling autoclose route for h5netcdf. Please let me know when/if you'd like me to take another look.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-278701754,https://api.github.com/repos/pydata/xarray/issues/1198,278701754,MDEyOklzc3VlQ29tbWVudDI3ODcwMTc1NA==,1217238,2017-02-09T16:52:53Z,2017-02-09T16:52:53Z,MEMBER,"I'm still working on it. A possible culprit is some sort of dangling reference that stops the file from being actually closed by h5py. If we can't figure it out in a few days, we can just disable autoclose for h5netcdf.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-278691935,https://api.github.com/repos/pydata/xarray/issues/1198,278691935,MDEyOklzc3VlQ29tbWVudDI3ODY5MTkzNQ==,4295853,2017-02-09T16:21:38Z,2017-02-09T16:21:38Z,CONTRIBUTOR,"@shoyer, it is quite possible the error is on my end because the error message implies that modification of the data prior to closing it may be the issue. However, I'm not getting that type of an error following a similar philosophy for the other backends, which is confusing.
Do you have advice on hunting this bug further? You obviously have more experience with `h5py` and `h5netcdf` than I do. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-278552122,https://api.github.com/repos/pydata/xarray/issues/1198,278552122,MDEyOklzc3VlQ29tbWVudDI3ODU1MjEyMg==,1217238,2017-02-09T05:36:41Z,2017-02-09T05:36:41Z,MEMBER,"I can reproduce issues with h5netcdf on my machine. Of course, I get slightly different error messages -- either an error about closing a closed file from h5py or a segfault.
So far, I've verified that it isn't a multi-threading issue -- I get the same errors when dask is run in single-threaded mode (`dask.set_options(get=dask.async.get_sync`). That's not encouraging, suggesting that this may be a legitimate h5netcdf or (perhaps more likely) h5py bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-277504185,https://api.github.com/repos/pydata/xarray/issues/1198,277504185,MDEyOklzc3VlQ29tbWVudDI3NzUwNDE4NQ==,1217238,2017-02-05T08:14:51Z,2017-02-05T08:14:51Z,MEMBER,"I'll take a look tomorrow. Getting all these backends to behave correctly
and consistently is a constant battle.
On Sat, Feb 4, 2017 at 9:20 PM Phillip Wolfram
wrote:
> @shoyer , the pushed code represents my
> progress. The initial PR had a bug-- essentially a calculation couldn't be
> performed following the load. This fixes that bug and provides a test to
> ensure that this doesn't happen. However, I'm having trouble with h5netcdf,
> which I'm not very familiar with compared to netcdf. This represents my
> current progress, I just need some more time (or even inspiration from you)
> to sort out this last key issue...
>
> I'm getting the following error:
>
> ================================================================================================================== FAILURES ==================================================================================================================
> ___________________________________________________________________________________________ OpenMFDatasetTest.test_4_open_large_num_files_h5netcdf ___________________________________________________________________________________________
>
> self =
>
> @requires_dask
> @requires_h5netcdf
> def test_4_open_large_num_files_h5netcdf(self):> self.validate_open_mfdataset_large_num_files(engine=['h5netcdf'])
>
> xarray/tests/test_backends.py:1040:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> xarray/tests/test_backends.py:1018: in validate_open_mfdataset_large_num_files
> self.assertClose(ds.foo.sum().values, np.sum(randdata))
> xarray/core/dataarray.py:400: in values
> return self.variable.values
> xarray/core/variable.py:306: in values
> return _as_array_or_item(self._data)
> xarray/core/variable.py:182: in _as_array_or_item
> data = np.asarray(data)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/numpy/core/numeric.py:482: in asarray
> return array(a, dtype, copy=False, order=order)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:1025: in __array__
> x = self.compute()
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:79: in compute
> return compute(self, **kwargs)[0]
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:179: in compute
> results = get(dsk, keys, **kwargs)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:537: in get_sync
> raise_on_exception=True, **kwargs)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:500: in get_async
> fire_task()
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:476: in fire_task
> callback=queue.put)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:525: in apply_sync
> res = func(*args, **kwds)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:268: in execute_task
> result = _execute_task(task, data)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in _execute_task
> args2 = [_execute_task(a, cache) for a in args]
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in
> args2 = [_execute_task(a, cache) for a in args]
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in _execute_task
> return [_execute_task(a, cache) for a in arg]
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in
> return [_execute_task(a, cache) for a in arg]
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:249: in _execute_task
> return func(*args2)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:52: in getarray
> c = a[b]
> xarray/core/indexing.py:401: in __getitem__
> return type(self)(self.array[key])
> xarray/core/indexing.py:376: in __getitem__
> return type(self)(self.array, self._updated_key(key))
> xarray/core/indexing.py:354: in _updated_key
> for size, k in zip(self.array.shape, self.key):
> xarray/core/indexing.py:364: in shape
> for size, k in zip(self.array.shape, self.key):
> xarray/core/utils.py:414: in shape
> return self.array.shape
> xarray/backends/netCDF4_.py:37: in __getattr__
> return getattr(self.datastore.ds.variables[self.var], attr)
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/contextlib.py:66: in __exit__
> next(self.gen)
> xarray/backends/h5netcdf_.py:105: in ensure_open
> self.close()
> xarray/backends/h5netcdf_.py:190: in close
> _close_ds(self.ds)
> xarray/backends/h5netcdf_.py:70: in _close_ds
> find_root(ds).close()
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5netcdf/core.py:458: in close
> self._h5file.close()
> ../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5py/_hl/files.py:302: in close
> self.id.close()
> h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
> ???
> h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
> ???
> h5py/h5f.pyx:282: in h5py.h5f.FileID.close (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/h5f.c:3905)
> ???
> h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
> ???
> h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
> ???
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> > ???
> E RuntimeError: dictionary changed size during iteration
>
> h5py/_objects.pyx:119: RuntimeError
> ============================================================================================ 1 failed, 1415 passed, 95 skipped in 116.54 seconds =============================================================================================
> Exception ignored in: .remove at 0x10f16e598>
> Traceback (most recent call last):
> File ""/Users/pwolfram/anaconda/envs/test_env_xarray35/lib/python3.5/weakref.py"", line 117, in remove
> TypeError: 'NoneType' object is not callable
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-277498382,https://api.github.com/repos/pydata/xarray/issues/1198,277498382,MDEyOklzc3VlQ29tbWVudDI3NzQ5ODM4Mg==,4295853,2017-02-05T05:20:22Z,2017-02-05T05:20:22Z,CONTRIBUTOR,"@shoyer, the pushed code represents my progress. The initial PR had a bug-- essentially a calculation couldn't be performed following the load. This fixes that bug and provides a test to ensure that this doesn't happen. However, I'm having trouble with h5netcdf, which I'm not very familiar with compared to netcdf. This represents my current progress, I just need some more time (or even inspiration from you) to sort out this last key issue...
I'm getting the following error:
```bash
================================================================================================================== FAILURES ==================================================================================================================
___________________________________________________________________________________________ OpenMFDatasetTest.test_4_open_large_num_files_h5netcdf ___________________________________________________________________________________________
self =
@requires_dask
@requires_h5netcdf
def test_4_open_large_num_files_h5netcdf(self):
> self.validate_open_mfdataset_large_num_files(engine=['h5netcdf'])
xarray/tests/test_backends.py:1040:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xarray/tests/test_backends.py:1018: in validate_open_mfdataset_large_num_files
self.assertClose(ds.foo.sum().values, np.sum(randdata))
xarray/core/dataarray.py:400: in values
return self.variable.values
xarray/core/variable.py:306: in values
return _as_array_or_item(self._data)
xarray/core/variable.py:182: in _as_array_or_item
data = np.asarray(data)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/numpy/core/numeric.py:482: in asarray
return array(a, dtype, copy=False, order=order)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:1025: in __array__
x = self.compute()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:79: in compute
return compute(self, **kwargs)[0]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:179: in compute
results = get(dsk, keys, **kwargs)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:537: in get_sync
raise_on_exception=True, **kwargs)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:500: in get_async
fire_task()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:476: in fire_task
callback=queue.put)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:525: in apply_sync
res = func(*args, **kwds)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:268: in execute_task
result = _execute_task(task, data)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in _execute_task
args2 = [_execute_task(a, cache) for a in args]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in
args2 = [_execute_task(a, cache) for a in args]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in _execute_task
return [_execute_task(a, cache) for a in arg]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in
return [_execute_task(a, cache) for a in arg]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:249: in _execute_task
return func(*args2)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:52: in getarray
c = a[b]
xarray/core/indexing.py:401: in __getitem__
return type(self)(self.array[key])
xarray/core/indexing.py:376: in __getitem__
return type(self)(self.array, self._updated_key(key))
xarray/core/indexing.py:354: in _updated_key
for size, k in zip(self.array.shape, self.key):
xarray/core/indexing.py:364: in shape
for size, k in zip(self.array.shape, self.key):
xarray/core/utils.py:414: in shape
return self.array.shape
xarray/backends/netCDF4_.py:37: in __getattr__
return getattr(self.datastore.ds.variables[self.var], attr)
../../anaconda/envs/test_env_xarray35/lib/python3.5/contextlib.py:66: in __exit__
next(self.gen)
xarray/backends/h5netcdf_.py:105: in ensure_open
self.close()
xarray/backends/h5netcdf_.py:190: in close
_close_ds(self.ds)
xarray/backends/h5netcdf_.py:70: in _close_ds
find_root(ds).close()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5netcdf/core.py:458: in close
self._h5file.close()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5py/_hl/files.py:302: in close
self.id.close()
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
???
h5py/h5f.pyx:282: in h5py.h5f.FileID.close (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/h5f.c:3905)
???
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E RuntimeError: dictionary changed size during iteration
h5py/_objects.pyx:119: RuntimeError
============================================================================================ 1 failed, 1415 passed, 95 skipped in 116.54 seconds =============================================================================================
Exception ignored in: .remove at 0x10f16e598>
Traceback (most recent call last):
File ""/Users/pwolfram/anaconda/envs/test_env_xarray35/lib/python3.5/weakref.py"", line 117, in remove
TypeError: 'NoneType' object is not callable
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-277400963,https://api.github.com/repos/pydata/xarray/issues/1198,277400963,MDEyOklzc3VlQ29tbWVudDI3NzQwMDk2Mw==,1217238,2017-02-04T00:36:15Z,2017-02-04T00:36:15Z,MEMBER,@pwolfram this looks pretty close to me now -- let me know when it's ready for review.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-277276242,https://api.github.com/repos/pydata/xarray/issues/1198,277276242,MDEyOklzc3VlQ29tbWVudDI3NzI3NjI0Mg==,731499,2017-02-03T15:27:04Z,2017-02-03T15:27:04Z,CONTRIBUTOR,"I'm just chiming in to signify my interest in seeing this issue solved. I have just hit ""OSError: Too many open files"". The data itself is not even huge, but it's scattered across many files and it's a PITA to revert to manual concatenation -- I've grown used to dask doing the work for me ;-)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-277028448,https://api.github.com/repos/pydata/xarray/issues/1198,277028448,MDEyOklzc3VlQ29tbWVudDI3NzAyODQ0OA==,4295853,2017-02-02T17:42:41Z,2017-02-02T17:42:41Z,CONTRIBUTOR,There are still a few more issues that need ironed out. I'll let you know when I've resolved them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-276474582,https://api.github.com/repos/pydata/xarray/issues/1198,276474582,MDEyOklzc3VlQ29tbWVudDI3NjQ3NDU4Mg==,4295853,2017-01-31T19:59:30Z,2017-01-31T19:59:30Z,CONTRIBUTOR,@shoyer and @PeterDSteinberg I've updated this PR to reflect requested changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-275254097,https://api.github.com/repos/pydata/xarray/issues/1198,275254097,MDEyOklzc3VlQ29tbWVudDI3NTI1NDA5Nw==,4295853,2017-01-25T22:32:17Z,2017-01-25T22:32:17Z,CONTRIBUTOR,"@PeterDSteinberg, did this PR fix the issue for you? I obviously need to update it but just wanted to confirm that the current branch resolved the too-many-open files error issue. Also, do you have any idea of the performance impact of these changes I'm proposing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-275251977,https://api.github.com/repos/pydata/xarray/issues/1198,275251977,MDEyOklzc3VlQ29tbWVudDI3NTI1MTk3Nw==,1445602,2017-01-25T22:22:52Z,2017-01-25T22:22:52Z,NONE,"I appreciate your work on this too-many-files-open error - I think your fixes will add a lot of value to the NetCDF multi-file functionality. In [this notebook using K-Means clustering on multi-file NetCDF data sets](https://github.com/ContinuumIO/elm-examples/blob/master/notebooks/temperature-PDFs-clustering.ipynb) I have repeatedly experienced the too-many-open files error, even with attempts to adjust via `ulimit`. I can test out the notebook again as this PR is finalized.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-272323006,https://api.github.com/repos/pydata/xarray/issues/1198,272323006,MDEyOklzc3VlQ29tbWVudDI3MjMyMzAwNg==,4295853,2017-01-13T00:03:24Z,2017-01-13T00:03:24Z,CONTRIBUTOR,"Thanks @shoyer. This makes sense. I think the path forward on the next round of edits should include making sure existing tests using `open_mfdataset` use both options to `autoclose`. If we do this we could future-proof ourselves against loss due to accidental breaking of this new functionality and avoid potentially contaminating existing workflows via performance concerns.
Documentation is also obviously required.
FYI as a heads up, I probably won't be able to get to this mid-week at the earliest but it appears we are close to a viable solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-272252623,https://api.github.com/repos/pydata/xarray/issues/1198,272252623,MDEyOklzc3VlQ29tbWVudDI3MjI1MjYyMw==,1217238,2017-01-12T19:08:33Z,2017-01-12T19:08:33Z,MEMBER,"This should be totally fine without performance or compatibility concerns as long as we set `autoclose=False` by the default.
In the long term, it would be nice to handle autoclosing automatically (invoking it when the number of open files exceeds some limit), but we should probably be a little more clever for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-272077913,https://api.github.com/repos/pydata/xarray/issues/1198,272077913,MDEyOklzc3VlQ29tbWVudDI3MjA3NzkxMw==,1217238,2017-01-12T05:12:53Z,2017-01-12T05:12:53Z,MEMBER,"Don't worry about #1087 -- I can rebase it.
On Wed, Jan 11, 2017 at 8:54 PM Phillip Wolfram
wrote:
> @shoyer , I just realized this might conflict
> with #1087 . Do you foresee
> this causing problems and what order do you plan to merge this PR and
> #1087 (which obviously
> predates this one...)? We are running into the snag with #463
> in our analysis and my
> personal preference would be to get some type of solution into place sooner
> than later. Thanks for considering this request.
>
> Also, I'm not sure exactly the best way to test performance either. Could
> we potentially use something like the ""toy"" test cases for this purpose?
> Ideally we would have a test case with O(100) files to gain a clearer
> picture of the performance cost of this PR.
>
> Please let me know what you want me to do with this PR-- should I clean it
> up in anticipation of a merge or just wait for now to see if there are
> extra things that need fixed via additional testing? Note I have the full
> scipy, h5netcdf and pynio implementations that can also be reviewed because
> they weren't available when you did your review yesterday.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-272075886,https://api.github.com/repos/pydata/xarray/issues/1198,272075886,MDEyOklzc3VlQ29tbWVudDI3MjA3NTg4Ng==,4295853,2017-01-12T04:54:27Z,2017-01-12T04:54:27Z,CONTRIBUTOR,"@shoyer, I just realized this might conflict with #1087. Do you foresee this causing problems and what order do you plan to merge this PR and #1087 (which obviously predates this one...)? We are running into the snag with #463 in our analysis and my personal preference would be to get some type of solution into place sooner than later. Thanks for considering this request.
Also, I'm not sure exactly the best way to test performance either. Could we potentially use something like the ""toy"" test cases for this purpose? Ideally we would have a test case with O(100) files to gain a clearer picture of the performance cost of this PR.
Please let me know what you want me to do with this PR-- should I clean it up in anticipation of a merge or just wait for now to see if there are extra things that need fixed via additional testing? Note I have the full scipy, h5netcdf and pynio implementations that can also be reviewed because they weren't available when you did your review yesterday.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-271996986,https://api.github.com/repos/pydata/xarray/issues/1198,271996986,MDEyOklzc3VlQ29tbWVudDI3MTk5Njk4Ng==,1217238,2017-01-11T21:16:35Z,2017-01-11T21:16:35Z,MEMBER,"> Does that mean if the checks pass the code is at least minimally correct in terms of not breaking previous design choices? E.g., does this imply that we are ok except for cleanup / implementation details on this PR?
If the checks pass, it means that it doesn't directly break anything that we have tests for. Which should cover *most* functionality. However, we'll still need to be careful not to introduce performance regressions -- we don't have any automated performance tests yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-271961921,https://api.github.com/repos/pydata/xarray/issues/1198,271961921,MDEyOklzc3VlQ29tbWVudDI3MTk2MTkyMQ==,4295853,2017-01-11T19:01:42Z,2017-01-11T19:01:42Z,CONTRIBUTOR,"Thanks @shoyer, Does that mean if the checks pass the code is at least minimally correct in terms of not breaking previous design choices? E.g., does this imply that we are ok except for cleanup / implementation details on this PR?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-271960859,https://api.github.com/repos/pydata/xarray/issues/1198,271960859,MDEyOklzc3VlQ29tbWVudDI3MTk2MDg1OQ==,1217238,2017-01-11T18:57:58Z,2017-01-11T18:57:58Z,MEMBER,"@pwolfram the allowed failures are pre-existing, not related to this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-271957327,https://api.github.com/repos/pydata/xarray/issues/1198,271957327,MDEyOklzc3VlQ29tbWVudDI3MTk1NzMyNw==,4295853,2017-01-11T18:44:51Z,2017-01-11T18:44:51Z,CONTRIBUTOR,"@shoyer, all the checks ""pass"" but there are still errors in the ""allowed"" list. If you get a change could you please provide me some perspective on whether these are errors on my end or not? I'm not exactly sure how to interpret them.
Once I know I have correctness in this code I plan to fix the inlines you graciously highlighted above. I think we are getting close here, assuming that I have enough testing to demonstrate we have accurately fixed the too many open file issue. Any additional ideas you have for tests would be really helpful too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056
https://github.com/pydata/xarray/pull/1198#issuecomment-271660072,https://api.github.com/repos/pydata/xarray/issues/1198,271660072,MDEyOklzc3VlQ29tbWVudDI3MTY2MDA3Mg==,4295853,2017-01-10T18:41:36Z,2017-01-10T18:41:36Z,CONTRIBUTOR,"The intent of this PR is to address (or at least partially address) the following issues:
* https://github.com/pydata/xarray/issues/463
* https://github.com/pydata/xarray/issues/798
* https://github.com/CCI-Tools/cate-core/issues/102
* https://github.com/MPAS-Dev/MPAS-Analysis/issues/49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,199900056