issue_comments

If you haven't already, would be good if those running into issues here could look over the Array API. This is still something that is being worked on, but the goal is to standardize Array APIs. If there are things missing from that, it would be good to hear about them in a new issue.

There are also some relevant and very interesting PyTorch development discussions; there are a lot of Tensor subclasses in the making and thought being put into how those interact: - https://dev-discuss.pytorch.org/t/state-of-pytorch-core-september-2021-edition/332#alternative-tensors-5 - https://pytorch-dev-podcast.simplecast.com/episodes/tensor-subclasses-and-liskov-substitution-principle - https://dev-discuss.pytorch.org/t/functorch-levels-as-dynamically-allocated-classes/294

Thank you very much to everyone who came today (all 18 of us!). The notes from the meeting are here.

I've raised the ToDos we had as an issue in the new duck array discussion repo - anyone interested in this topic might want to watch the activity on that repository.

The plan now is to continue via asynchronous discussion on that new repo, referring back to issues in specific libraries when necessary. If we come to another sticking point that requires in-person discussion then we can organise another face-to-face meeting at that point, which I am happy to do. However we won't organise another meeting until there is a specific blockage to discuss. I will therefore keep this issue open so I can use it to alert anyone who might be interested in a follow-up meeting.

It looks like there are some other Dask folks participating. I'll step back and let them take over on our end.

On Wed, Sep 22, 2021 at 9:53 AM Hameer Abbasi @.***> wrote:

I would very much prefer not to be recorded.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/5648#issuecomment-925007468, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTAVX5WJDRARG4AOE43UDHUU5ANCNFSM5BHAMZ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Change of plan - use my personal zoom room instead (as I don't think I have permissions to start the xarray one):

https://columbiauniversity.zoom.us/j/97399260034?pwd=WS9GUnVrcG1YSWRPeXhITXRhdEJZUT09

Also unless anyone has a problem I'm going to record the meeting, just for note-taking purposes. (EDIT: I'll just take notes as the meeting goes on instead)

I would very much prefer not to be recorded.

I'd like to attend on behalf of QuTiP. I'll likely mostly listen -- QuTiP is not directly affected in the way that xarray, Dask, cupy, etc are, but we're users of potentially all of these array types (and already explicitly support numpy, CuPy, and our own sparse array format) and we are facing similar issues of our own (i.e. users of QuTiP are asking us to develop a __qutip_qobj__ style protocol similar to numpy's and we would like to learn the lessons of the last decade of numpy rather than repeat the steps ourselves in the coming one).

Will try to join.

I'd also like to attend, primarily just to learn.

TOMORROW: So there is just one slot when essentially everyone said they were free - but it's 11:00am EDT September 22nd, i.e. tomorrow morning.

I appreciate that's late notice - but could we can try for that and if we only get the super keen beans attending this time then we woud still be able to have a useful initial discussion about the exact problems that need resolving?

Alternatively if people could react to this comment with thumbs up / down for "that time is still good" / "no I need more notice" then that would be useful.

@mrocklin are you or anyone else who is able to speak for dask free? I noticed that @jacobtomlinson put down that you were free tomorrow also?

EDIT: @keewis I'm pinging you too

I'd like to attend too

Surprisingly I happen to be free tomorrow at exactly that time.

Great! I sent an invite to the few people whose emails I could easily find - let me know if that didn't arrive @mrocklin .

Maybe too soon to ask, but do we have a link to the video call?

I think we can probably just use the one we use for the xarray bi-weekly dev meetings:

https://us02web.zoom.us/j/88251613296?pwd=azZsSkU1UWJZTVFKNnhIUVdZcENUZz09

Maybe too soon to ask, but do we have a link to the video call?

I can be there! @jakirkham and @pentschev are also Dask maintainers so there should be good representation.

Surprisingly I happen to be free tomorrow at exactly that time. I've blocked it off. If you want to send a calendar invite to mrocklin at coiled that would be welcome.

On Tue, Sep 21, 2021 at 10:27 AM Tom Nicholas @.***> wrote:

TOMORROW: So there is just one slot when essentially everyone said they were free - but it's 11:00am EDT September 22nd, i.e. tomorrow morning.

I appreciate that's late notice - but could we can try for that and if we only get the super keen beans attending this time then we woud still be able to have a useful initial discussion about the exact problems that need resolving?

Alternatively if people could react to this comment with thumbs up / down for "that time is still good" / "no I need more notice" then that would be useful.

@mrocklin https://github.com/mrocklin are you or anyone else who is able to speak for dask free? I noticed that @jacobtomlinson https://github.com/jacobtomlinson put down that you were free tomorrow also?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/5648#issuecomment-924099831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTFTAT62OPPX7Z54MRDUDCP6NANCNFSM5BHAMZ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Apologies for the long delay, but I would like to try to organise this meeting for some time in the next few weeks. If you are interested could you please fill out this When2Meet meeting time poll. (If none of these times work for you because of time zone issues or something then please say so!) It would be great to have lots of libraries represented.

Agenda doc here - I will come back to this in more detail soon, but if you have specific things you would like to discuss then please add them.

👂🏾

* scipp (@SimonHeybrock, xref [NEP 18, physical units, uncertainties, and the scipp library? #3509](https://github.com/pydata/xarray/issues/3509))

Thanks! I am definitely interested.

Interesting - could you say a bit more? Looking at these two issues, it seemed more like the question was simply on hold until someone who wanted it badly enough came along?

There is a significant backwards compatibility break when a library adds __array_ufunc__ and __array_function__. JAX maintainers were not comfortable with that. @shoyer wrote https://numpy.org/neps/nep-0037-array-module.html as a follow-up largely because of that. That NEP is effectively superceded by the array API standard (https://data-apis.org/array-api/latest/ and NEP 47). PyTorch has decided to adopt that and implementation is well underway. Experimental support in NumPy will land next week (complete except for linalg). CuPy and MXNet plan to follow that NumPy implementation. JAX and Dask not yet decided I believe, but likely to follow NumPy as well.

Canonical/minimal API of a "duck array" and how to detect it (though may be superseded by NEPs 30 and 47 among others)

This is basically what the array API standard provides (most functions follow NumPy, with deviations mostly where other libraries were already deviating because they could implement something on GPU, with a JIT, or with a non-strided memory model). __array_function__ has worked quite well for CuPy and Dask, because they follow the NumPy API almost to the letter, with only a couple of exceptions (e.g. 0-D array instead of array scalars in CuPy). JAX, PyTorch and MXNet all deviate much more, and since the NumPy API is not very well-defined (there's 1500+ public objects plus more semi-public ones), you'd have no guarantees about what works and what doesn't.

That said, __array_ufunc__ and __array_function__ aren't going anywhere. The RAPIDS ecosystem is invested in it and I believe largely happy with it. So adding Xarray and Pint to the mix sounds potentially interesting.

• Awkward? (@jpivarski)

I'm interested. Let us know when the time will be or if there's a poll for picking a time. Thanks!

Happy to hop on a call for this as well, thanks for organizing all of this @TomNicholas!

I'm also happy to join the meeting. Thanks @TomNicholas for the initiative here and @jacobtomlinson for tagging me.

NumPy masked arrays (??)

Tagging @greglucas because of his work on https://github.com/numpy/numpy/pull/16022#issuecomment-888495182 - at least I think a very common desired use case of multi-nested arrays would be xarray(pint(dask(np.masked)).

• Awkward? (@jpivarski)

Count me in for the meeting!

Here are a few suggestions about possible topics to add to the agenda (based on linked issues/discussions), if we can fit it all in:

• Canonical/minimal API of a "duck array" and how to detect it (though may be superseded by NEPs 30 and 47 among others)
• Consistency of type deferral (i.e., between construction, binary ops, __array_ufunc__, __array_function__, and array modules...for example, these are uniform in Pint, but construction and array module functions are deliberately different from the others for Dask arrays)
• API for inter-type casting and changing what types are used in a nested array (e.g. #3245 and #5568)
• How to handle unknown duck arrays
• Nested array reprs (both short and full)
• Best practices for "carrying through" operations belonging to wrapped types (i.e., doing Dask-related things to a Pint Quantity or xarray DataArray that contains a Dask array), even if multiple layers deep

Also, tagging a few other array type libraries and maintainers/contributors who may be interested (please ping the relevant folks if you know them):

• unyt (@ngoldbaum)
• astropy.units (??)
• NumPy masked arrays (??)
• scipp (@SimonHeybrock, xref https://github.com/pydata/xarray/issues/3509)

(interesting side note is the first three of these are all ndarray subclasses right now...perhaps discussing the interplay between array subclassing and wrapping is in order too?)

Thanks to everyone ho has expressed interest already! I'll give some more time for responses before starting to think about potential meeting times.

There probably isn't much to say though - PyTorch is unlikely to adopt array_function at this point, just like JAX.

Interesting - could you say a bit more? Looking at these two issues, it seemed more like the question was simply on hold until someone who wanted it badly enough came along?

https://github.com/google/jax/issues/1565

https://github.com/pytorch/pytorch/issues/22402

(I'm interested because I'm in the set of people who have an ambition to use these libraries (wrapped in xarray) at some point further down the line, but that might remain just an ambition :sweat_smile:)

The key thing here seems to be Dask <-> Xarray <-> Pint, unless I'm missing something?

You're right, that was the primary motivation for this issue, but it also seemed like a good opportunity to get as many libraries talking (and their relationships codified) as possible, especially if one of the outcomes ends up being a type casting hierarchy resource/NEP.

Happy to attend. It might also be useful to have @pentschev involved too.

I'm happy to join, seems interesting. And yes, I can say something about PyTorch. There probably isn't much to say though - PyTorch is unlikely to adopt __array_function__ at this point, just like JAX. And it doesn't seem critical for this hierarchy anyway - the fundamental array objects (PyTorch/CuPy/NumPy/Sparse/JAX arrays or tensors) do not have or need a class hierarchy, they are all at the bottom and should not be mixed.

The key thing here seems to be Dask <-> Xarray <-> Pint, unless I'm missing something?

I'd also be happy to attend. Keep in mind I'm in the CET timezone.

I would be happy to attend and look forward to what I'm sure will be a vigorous discussion :) Thank you for providing convenient links to reading materials ahead of time.

As a warning, my responsiveness to github comments these days is not what it used to be. If I miss something here then please forgive me.

cc @hameerabbasi for sparse, and @amcnicho might also be interested