issue_comments

+1 from me as well.

+1 @fmaussion - Thanks!

Hi @darothen earthio is a recent experimental refactor of what was the elm.readers subpackage. elm - Ensemble Learning Models was developed with a Phase I NASA SBIR in 2016 and in part reflects our thinking in late 2015 when xarray was newer and we were planning the proposal. In the last ca. month we have started a Phase II of development on multi-model dask/xarray ML algorithms based on xarray, dask, scikit-learn and a Bokeh maps UI for tasks like land cover classification. I'll add you to elm and feel free to contact me at psteinberg [at] continuum [dot] io. We will do more promotion / blogs in the near term and also in about 12 months we will release a free/open collection of notebooks that form a "Machine Learning with Environmental Data" 3-day course.

Back to the subject matter of the thread.... You can assign the issue to me (can you add me also to xarray repo so I can assign myself things?).. I'll wait to get started until after @shoyer comments on @laliberte 's question:

(1) replicate serial coarsen into xarray or (2) point to dask coarsen methods?

Regridding is of interest to NASA and other clients of ours. It is important to them to be able to do broadcast operations between rasters that differ in resolution or are otherwise offset. We'll follow the XMap repo mentioned above ( @jhamman ) and see about building on that style. Our clients and open source tools like datashader for viz and elm for ML could use XMap and benefit from coordinate transformations and regridding. We have a meeting internally to discuss approaches to the coordinates' metadata and resampling / regridding and I'll be in touch further soon about how we can help here (see also the issues on this experimental earthio repo).

Hello @laliberte @shoyer @jhamman . I'm with Continuum and working on NASA funded Earth science ML (see ensemble learning models in github and its documentation here as well as earthio, an experimental repo we have discussed simplifying and transitioning to Xarray - earthio issue 12 and earthio issue 13). We (Continuum NASA, dask, and datashader team members) met with @rabernat this morning and discussed ideas for collaboration better with the Xarray team. I'll comment on more issues more regularly and make some experimental PRs over the next month. I'd like to keep most of the discussion on github issues so it is of general utility, but I'm happy to chat anytime if you want to talk further detail on longer term goals with Xarray.

We can submit a PR on this issue for dask's coarsen and the specs above for using block_reduce in some situations. We have a variety of tasks we are covering now and in a planning / architecture phase for NASA. If we are too slow to respond to this issue, feel free to ping me.

I appreciate your work on this too-many-files-open error - I think your fixes will add a lot of value to the NetCDF multi-file functionality. In this notebook using K-Means clustering on multi-file NetCDF data sets I have repeatedly experienced the too-many-open files error, even with attempts to adjust via ulimit. I can test out the notebook again as this PR is finalized.