html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/591#issuecomment-453800603,https://api.github.com/repos/pydata/xarray/issues/591,453800603,MDEyOklzc3VlQ29tbWVudDQ1MzgwMDYwMw==,2443309,2019-01-13T04:11:08Z,2019-01-13T04:11:08Z,MEMBER,Closing this old issue. The distrarray project seems to be done and we have more recent issues discussing flexible array interfaces.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,108623921
https://github.com/pydata/xarray/issues/591#issuecomment-144269142,https://api.github.com/repos/pydata/xarray/issues/591,144269142,MDEyOklzc3VlQ29tbWVudDE0NDI2OTE0Mg==,1217238,2015-09-30T03:04:18Z,2015-10-05T18:08:08Z,MEMBER,"Right now we have some custom logic in xray to support dask. This is OK, but ideally we would have generic hooks that would let other libraries supply ndarray objects suitable for putting in xray data structures that we don't know anything about. My hope is to resolve this upstream in NumPy (see https://github.com/numpy/numpy/issues/4164 and related issues). In particularly, I'd like to start with a proposal for `__array_concatenate__` and then add enough special methods that we can get ride of almost all the `isinstance` checks in `xray.core.ops`.

https://github.com/bolt-project/bolt/issues/58 provides an overview of the array API that we need to sensibly wrap another array library as an alternative to numpy/dask.

So let me make this a challenge: if anyone has an open source array library that checks off most of those boxes and that they want to use with xray, I will gladly help you wrap it. Even before we have a full ""duck array"" API in NumPy itself, if you can provide a compatibility module that makes your array library work like NumPy (e.g., like `dask.array`), I will do the necessary hacks to make it work in xray.

Distarray does look interesting, but it seems like it hasn't gotten much real world use (yet), so it's hard to say how well it will work. The project has tackled some pretty hard problems with representing and manipulating fully distributed arrays in a variety of underlying representations, but that means they've had less time to focus on building out the long tail of features necessary for a useful ndarray library. That said, I would love to be proven wrong -- if distarray can do most of the items on my list and a numpy compatible API, then it's worth thinking about what it would take to wrap it. Perhaps @bgrant can help clarify? (I enjoyed your talk at SciPy, by the way!)

Here are a few other projects that are also worth keeping an eye on:
- [Bolt](http://bolt-project.org/) (ndarrays on Spark, based on a tool @freeman-lab built for scalable neuroscience)
- [SciDB-Py](http://scidb-py.readthedocs.org/en/stable/) (a ""distributed array database"")
- [DyND](https://github.com/libdynd/libdynd) (a potential NumPy replacement, see https://github.com/libdynd/libdynd/issues/564)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,108623921
https://github.com/pydata/xarray/issues/591#issuecomment-144171481,https://api.github.com/repos/pydata/xarray/issues/591,144171481,MDEyOklzc3VlQ29tbWVudDE0NDE3MTQ4MQ==,2443309,2015-09-29T19:58:27Z,2015-09-29T19:58:27Z,MEMBER,"I think this is an interesting idea.  I was just asking @shoyer the other day if we could use `dask.distributed` with `xray` objects.

@shoyer will be able to speak more to the applicability and effort required to implement `distarray`.  I'm not sure how many array abstractions we want in xray.  Currently we have two (numpy and dask).  Fully distributed arrays may be the logical extension.

One major drawback I see is that we don't have a parallel netCDF library in python so I/O will be an issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,108623921