issue_comments

Since we have been making a lot of progress using __array_function__ (which does not exclude subclasses but is much more flexible), this approach is outdated and I'm closing this and #1118.

We could add the experimental option for allowing subclasses for testing purposes, but ultimately I think we want to insist that objects we use inside data implement __array_function__, regardless of whether they are a numpy array subclass or duck array. Otherwise there's no way complicated operations like concat are going to work properly.

@shoyer: What kind of testing would that be? If this is something that would be removed again in the future and has no actual value, feel free to close (but don't forget the original PR). Otherwise, I'd be happy to continue working on this.

then I would like to finish this one and modify the tests in a new one.

You are welcome to continue here or in a new PR, whichever you prefer. I don't have a strong opinion, please do whichever seems easier for you.

I think the best examples of complex operations to check for compatibility can be found in xarray/tests/test_sparse.py and xarray/tests/test_dask.py

I think I would like to tackle this, but I definitely will need some help with finding examples for some of the more advanced operations.

@shoyer should I add these in a new PR or should I repurpose this one?

Do you have a rough idea of which operations these tests should check? The ones currently implemented are mean() and an xfailing concat. There is also the sel(), but as it only tries to use values with the matching unit, it works even though it should probably fail.

However, the question of whether or not this PR is obsolete now still remains: if one were to replace quantities with pint, the test checking that this does not work without the proposed changes enabled is the only one that fails.

Does pint do __array_function__ yet? It looks like the answer is "not yet" but there's a branch with experimental support: https://github.com/hgrecco/pint/pull/764

We could add the experimental option for allowing subclasses for testing purposes, but ultimately I think we want to insist that objects we use inside data implement __array_function__, regardless of whether they are a numpy array subclass or duck array. Otherwise there's no way complicated operations like concat are going to work properly.

I'd love to add tests that verify that xarray can properly wrap arrays with units as soon as possible. We could even merge tests that are all marked "xfail" and that currently only pass when using the experimental version of pint from https://github.com/hgrecco/pint/pull/764.

As we saw with sparse array support, this will be useful for two reasons: 1. It will help us identify remaining issues that need to be solved either in xarray or in an array-with-units library like pint. 2. There are also likely some subtle tweaks we can do inside xarray to better support arrays with units, e.g., displaying units in the xarray.Dataset repr.

I'm not too sure about putting _asarray into npcompat.py, but apart from that I would say it is ready to be merged.

However, the question of whether or not this PR is obsolete now still remains: if one were to replace quantities with pint, the test checking that this does not work without the proposed changes enabled is the only one that fails.

now that #3117 has been merged, what is the status on this? From what I can tell from the few I looked at, most ndarray subclasses either already use NEP18 or will do so in the future. I guess what I want to know is: should I continue to work on it or does it overlap too much with that PR?

Looks like a glitch related to downloading an example file. Ignore it.

I added the option, though the documentation is minimal. It gets used by a function to decide between asarray and asanyarray, and this function is then called in as_compatible_data and in the Variable.data property. Also, .mean() did not work before because duck_array_ops.asarray used np.asarray, which I replaced with that switching function. Should I test more of these reduce functions (if so, which)?

The matrix class gets treated similar to MaskedArray, and the unittests prove that enabling subclass support does not work for these two.

Is the failure of the docs related to any of the changes I made?

This looks like a good start to me. So we can start making progress on this, I think it would make sense to hide this behavior behind an option, e.g., xarray.set_options(enable_experimental_ndarray_subclass_support=True).

I would consider writing some tests with two of the ndarray subclasses that ship with NumPy: - np.matrix - np.ma.MaskedArray

Both of these subclasses violate some important principles of subclassing, namely they aren't substitutable with objects of their parent class. Thus, I think it would be reasonable to explicitly "blacklist" them for use inside in xarray.DataArray. I think we already explicitly convert MaskedArray objects, but perhaps we should also add a check for np.matrix objects.

In terms of operations, one big class that comes to mind are aggregations, e.g., .mean(). It would be good to make sure that these work.

Most of the items above are implemented in unit-support, even support for array wrappers with units like pint (if my tests are sufficient for deciding that: pint still complains/warns about stripped units). The calls to asarray and asanyarray are skipped based on whether any of the __array_*__ methods and the attributes unit or units exist.

I did not push these commits to this PR because most of it does not have anything to do with subclasses. I did not open a new PR for the branch either because it builds on this one and because I'd like to receive feedback on any of this first.

Combining this with dask arrays should still fail, though.

Looks great! Thanks @keewis

I don't know this area that well, so I'll defer to someone else on a proper review