issue_comments

Closing this old issue. The distrarray project seems to be done and we have more recent issues discussing flexible array interfaces.

@shoyer I really like the "challenge" proposed here, and terrific summary of the state of things. It's definitely on our to-do list with Bolt and will keep you posted on progress, though we're currently knee-deep in the very related problem of context switching between different backend services (local, dask, distributed w/ spark, etc.)

Right now we have some custom logic in xray to support dask. This is OK, but ideally we would have generic hooks that would let other libraries supply ndarray objects suitable for putting in xray data structures that we don't know anything about. My hope is to resolve this upstream in NumPy (see https://github.com/numpy/numpy/issues/4164 and related issues). In particularly, I'd like to start with a proposal for __array_concatenate__ and then add enough special methods that we can get ride of almost all the isinstance checks in xray.core.ops.

https://github.com/bolt-project/bolt/issues/58 provides an overview of the array API that we need to sensibly wrap another array library as an alternative to numpy/dask.

So let me make this a challenge: if anyone has an open source array library that checks off most of those boxes and that they want to use with xray, I will gladly help you wrap it. Even before we have a full "duck array" API in NumPy itself, if you can provide a compatibility module that makes your array library work like NumPy (e.g., like dask.array), I will do the necessary hacks to make it work in xray.

Distarray does look interesting, but it seems like it hasn't gotten much real world use (yet), so it's hard to say how well it will work. The project has tackled some pretty hard problems with representing and manipulating fully distributed arrays in a variety of underlying representations, but that means they've had less time to focus on building out the long tail of features necessary for a useful ndarray library. That said, I would love to be proven wrong -- if distarray can do most of the items on my list and a numpy compatible API, then it's worth thinking about what it would take to wrap it. Perhaps @bgrant can help clarify? (I enjoyed your talk at SciPy, by the way!)

Here are a few other projects that are also worth keeping an eye on: - Bolt (ndarrays on Spark, based on a tool @freeman-lab built for scalable neuroscience) - SciDB-Py (a "distributed array database") - DyND (a potential NumPy replacement, see https://github.com/libdynd/libdynd/issues/564)

I think @shoyer's characterization of the current state of DistArray is pretty good. That said, I think this proposal is a good target for us to think about, and https://github.com/bolt-project/bolt/issues/58 is helpful. I'll try to look into this more if I can make some time.

I think this is an interesting idea. I was just asking @shoyer the other day if we could use dask.distributed with xray objects.

@shoyer will be able to speak more to the applicability and effort required to implement distarray. I'm not sure how many array abstractions we want in xray. Currently we have two (numpy and dask). Fully distributed arrays may be the logical extension.

One major drawback I see is that we don't have a parallel netCDF library in python so I/O will be an issue.