issue_comments

@jacobtomlinson Really glad someone's working on this! I'd be glad to help if I can (although I've never contributed to xarray and I don't know much about GPUs). I have some questions though.

Do you have a specific purpose in mind for this? I ask because most other discussions I see related to this really just wanna do ML. However, there's a large user base (myself included) that would benefit immensely from just doing regular (non-machine-learning) operations with a GPU backend.

Also, what's the status on the development? I see no comments after July 2020 and I'm hoping I can help get this back on track if needed!

While I'm sure some compatibility code has gone into xarray

actually, I have been able to get by without compatibility code, the code changes outside of test_units are mostly code refactoring (see #3706). This is probably because pint behaves (or tries to behave) like whatever it is wrapping, so that doesn't have to apply to other duck arrays.

While adding support for cupy as a 1st party might be less complicated, I think we should aim for 3rd party because that might be more maintainable, and adding support for similar duck arrays would become easier (I might be missing something, though).

While there are parts where interaction between numpy.ndarray and cupy.ndarray fails (e.g. #4231) – I still don't have any idea how to fix that in a general way – there are also several methods where we want to explicitly convert to numpy (e.g. when plotting), but without using numpy.asarray since that is commonly used to ensure objects are array-like.

Ideally, to make those work we'd have a standard on how to explicitly get the data of a duck array as a numpy array (obj.to_numpy()?). Right now, we have several: for sparse it is named dense() (or something like that), for cupy it is get(), and for pint we have the magnitude property (and I'm sure there are many more).

I've written this comment a few times to try and not come across as confrontational. I'm not intending to be at all, so please don't take it that way 😅. Tone is hard in comments! I'm just trying to figure out how to proceed quickly.

I've noticed a diverging theme that seems to be coming up in various conversations (see #3234 and #3245) around API design for alternative array implementations.

It seems to boil down to whether an array implementation has 1st party or 3rd party support within xarray.

For numpy and Dask they appear to be 1st party. They influence the main API of xarray and xarray contains baked in logic to create and work with them.

The work on pint so far points towards it being 3rd party. While I'm sure some compatibility code has gone into xarray much of the logic lives out in an accessor library. Given that pint is extending the numpy API this makes sense.

I initially started this work assuming that cupy would be added as 1st party type, given that it attempts to replicate the numpy API without addition. However I'm not sure this is the right stance.

There are a few questions such as "should .plot coerce cupy arrays to numpy?" (with the conversation in #3234 pointing towards no) which are making me think that perhaps it should be more 3rd party.

I think it would help with API design and speed here if a decision were to be made about cupy (and sparse) being 1st or 3rd party.

Perhaps some core maintainers could weigh in here?

@jacobtomlinson Any idea how this would play with the work that's been going on for units here; I'm specifically wondering if xarray ( pint ( cupy )) would/could work.

As far as I'd see it, the pieces to get this working are

• xarray( pint ) (practically ready, just working out indexes and corner cases: #3594)
• pint( cupy ) (https://github.com/hgrecco/pint/issues/964)
• xarray( cupy ) (this PR)

and then finally testing xarray( pint( cupy )) works automatically from there. https://github.com/hgrecco/pint/issues/964 was deferred due to CI/testing concerns, so it will be great to see what @jacobtomlinson can come up with here for xarray, since hopefully at some point it would be transferable over to pint as well.

@jacobtomlinson Any idea how this would play with the work that's been going on for units here; I'm specifically wondering if xarray ( pint ( cupy )) would/could work.

This PR for adding pint support is a useful reference. https://github.com/pydata/xarray/pull/3238

@jacobtomlinson, thank you for getting this started. I'll be monitoring closely this issue. Let me know if I can help in any way.