issue_comments

Just wanted to drop in and remind people interested in this that we hold a bi-weekly pangeo working group for distributed array computing, which is the perfect place to come and ask about any questions over zoom! I'll be there at 1pm EST today.

We'll fix the compatibility issues, but we first need to understand what the expectations on something like data.shape should be in these circumstances.

At a minimum, xarray expects .shape, .ndim and .dtype to always be defined. (And the number of dims to match the shape, which Joe's example above implies aesara doesn't do?) On top of that there are extra expectations about slicing and broadcasting changing shape in the same ways as it does for numpy arrays. (@keewis correct me if I've mis-stated this or missed something important here!)

For a shared variable, it's always possible to get a value for data.shape by referencing the underlying data, but the reason we don't do that by default is—in part—due to the fact that shared variables can be updated with values that have different shapes (but the same dtypes and number of dimensions).

This sounds a bit similar to discussions we have been having about wrapping ragged arrays in xarray, for which there are multiple ways you might choose to define the shape.

The simplest way to guarantee that aesara can be wrapped by xarray is if aesara conformed to the array API standard, and they have a test suite you can use to check that. We are also working on our own testing framework that duck-typed array libraries like aesara could import to quickly test integration with xarray.

(I was already typing this when Ryan posted so I'll finish anyway :sweat_smile:)

To clarify, what @jhamman is suggesting in this specific issue is xarray wrapping aesara, as opposed to aesara wrapping xarray.

Both of these goals would be great, but in xarray we would particularly love to be able to wrap aesara because: - We already have a large community of users who use xarray objects as their top-level interface, - It continues a larger project of xarray being generalized to wrap any numpy-like "duck array". - Our ultimate goal is for xarray users to be able to seamless switch out their computing backend and find which library gives them the best performance without changing the rest of their high-level code.

For xarray to wrap aesara aesara needs to provide a numpy-like API, ideally conforming to the python array api standard. If aesara already does this then we should try out the wrapping right now!

If you're interested in this topic I invite you to drop in to a meeting of the Pangeo working group on distributed arrays! We have so far had talks from distributed computing libraries including Arkouda, Ramba, and cubed, all of which we are hoping to support as compute backends.

If anyone is also separately interested in using xarray inside PyTensor / aesara then that's awesome, but we should try to track efforts in that direction on a different issue to keep this distinction clear. We plan to better support that direction of wrapping soon by fully exposing our (semi-private internal currently) lightweight Variable class.