issue_comments

Finally had some time to play around with the accessors, I have opened a PR adding them: https://github.com/arviz-devs/xarray-einstats/pull/51

I would be happy to integrate einops functionality better with xarray. Also feedback and contributions for xarray-einstats will be very welcome.

That being said, I haven't worked with accessors yet, so I would need some help to try and add this.

I am not sure I completely understand option 2, but option 1 seems a better fit to what we are doing at ArviZ (so far we are managing quite well with the InferenceData mentioned above which is a collection of independent xarray datasets). In our case, well defined selection for multiple variables at the same time (i.e. at the dataset level) is very useful.

I was also wondering what changes (if any) would each option imply when using apply_ufunc

I have created a library for easier linear algebra (plus others) with xarray: https://xarray-einstats.readthedocs.io/en/latest/.

It currently has: * wrappers for many numpy.linalg functions * wrappers for scipy.stats distributions plus a few functions * wrappers for einops functions * a numba.guvectorize-decorated version of numpy.histogram for dataarrays

I have mostly added wrappers for things I personally use, also trying to not overlap with xr-scipy nor xarray-extras, so it might not make much sense as a group, but the modules are completely independent between them and could be reorganized into independent packages or merged into existing ones.

Feedback very welcome!

Great, thanks for the offer :smile: I think I'll start with a minimal repo back at ArviZ and see how much of that can be used more generally. I probably should have named the example above arviz_dot instead. I definitely want to be able to do arviz_dot(a, b) and have it work automatically because in most ArviZ cases we do have all the information we need for this to be possible. We'll therefore definitely need a higher ArviZ layer, but I think an xarray-linalg would be a great base on which to build and we can keep both in different files/modules so it can be split easily.

I also commented here as I thought it was the most related issue, many of the functions I have in mind are related to linalg, but we'll probably have some other functions too.

Would there be any interest in an "xarray linalg library"? Some kind of library of that sort would be really useful to ArviZ and our users, I'll probably write some very basic functions myself on a minimal library or add them to ArviZ directly, but if there are other people interested we can try to find synergies and get something more general.

In our case, 99% of the time we want to "batch" over the chain and draw dimensions. So things as simple as:

def xarray_dot(a, b, dim=None): if dim is None: a_dims = set(a.dims) b_dims = set(b.dims) dim_set = a_dims.intersection(b_dims) - {"chain", "draw"} if len(dim_set) == 1: dim = dim_set.pop() else: ValueError if not isinstance(dim, str): raise ValueError return xr.apply_ufunc( np.einsum, "...i,...i", a, b, input_core_dims=[[], [dim], [dim]] )

are already extremely useful.

Being able to invert, cholesky... without using apply_ufunc would be convenient, even more so if we define some conventions. i.e. I generally use dim and dim bis for covariance matrices so they have shape chain, draw, ..., dim, dim bis, so I can invert those independently of their number and order of dimensions without even needing to say which are the "matrix" dimensions.

In our case, being able to ds.transpose("dim1", "dim2") and have it ignore chain, draw would also be convenient, which is probably a very ArviZ specific situation, but maybe there are other libraries/people that commonly have some batch dimensions, even if they use different names for them or if the user chooses those names freely.

Here are some biomedical papers that are using ArviZ and therefore xarray even if most don't cite xarray and some don't cite ArviZ either. Topics are quite disperse: covid, psychology, biomolecules, oncology...

Not really sure if there is anything we can do from ArviZ to help with that, if there is let us know and we'll do our best cc @percygautam

Yep, I hope I'll have time for more to come :)

Fixed tests. Now single index coordinates will keep their attributes when converted to non indexing coordinates.

I think changes in code would also make multi index keep their attributes, but I don't think multiindex can have attributes so it does not make any difference. I was wondering is this would be enough to close the original issue or if there is extra work to be done with multiindex.

Thanks @dcherian! If there is anything I can do to help please say so. I don't really know where to start searching for this new error but I can run some tests or look into it if given some pointers. Whatever means less work for you.

Regarding current test, I can modify it so it does not trigger the bug and open an issue for this second bug. Is this ok or do yo prefer to tackle both in this PR?

It is an issue of incompatibilities with sphinx_rtd_theme and sphinx2, and I am not sure when will it be solved, I have looked a little into it but for now I don't know how. There is one small problem with xarray doc generation behaviour. In the doc/environment.yml the sphinx version is fixed to 1.8, however, in the rtd build, the doc/enviroment.yml is overridden and sphinx 2 is used. I generated the docs locally with the environment in doc/ (right) to compared with the online docs (left):

I have no clear preference between sphinx1 or sphinx2, but both online and local docs should be the same. If you prefer sphinx2, I have no problem modifying doc/environment.yml, otherwise, I don't think a pull request can solve it (the build script in in the rtd website does not look like the one on github).

@max-sixty Not at all, whatever is best. I actually opened the issue without being 100% it was one.

I'm trying to think whether there would be any performance cost there - i.e. are there any arrays where preemptive broadcasting would be both expensive and unnecessary?

Even if there were a performance cost (compared to the actual behaviour), it could be easily avoided by using all dims as input_core_dims couldn't it? IIUC, all dims should be broadcasted unless they are in input core dims, so it broadcasting could still be avoided without problem.

Then shouldn't a in the first example keep its original shape?