issue_comments

I have added several test cases, almost all pass. The ones that don't pass are related to dim="time" and incompletely implemented pandas_cov and pandas_corr, that currently don't have a dim parameter.

I had a look at np.cov, np.corrcoef, pd.Series.cov, pd.frame.cov, pd.Series.corr, and pd.frame.corr: Computation of cov and corr is implemented in many different ways.

xr.cov and xr.corr now imitate the behavior of the respective pd.Series functions.

I actually prefer the numpy behavior, resulting in covariance and correlation matrices. However I feel that efficient implementation of this behavior is above my current understanding of xarray. So, I would highly appreciate your support on this implementation!

Otherwise, I added the ddof parameter to xr.cov to imitate the behavior of the respective pd.Series functions. I now get the results that pd.Series produces. However, I had to set ddof=1 for xr.cov and ddof=0 for xr.corr. Does that make sense?

For xarray.corr(array, array, dim='y'), I think we'd expect an array of the same dimensionality as below, with one point as the correlation of 1.0 (because {1,2} is 100% correlated to {1,2}) and another NaN. Does that make sense?

On the one side, I am with you in terms of "What You Put In Is What You Get Out", on the other hand pandas.Series.cov and np.cov have other behaviors that both seem plausible:

So for the moment, we might stick with pandas.Series.cov? Or (future PR?) we rather want the np.cov behavior, this would require more effort and the repr should probably be given as xarray instead of np.array, right?

I would start with having the code in the function; i.e. don't abstract too early unless we're really confident on the eventual state. When we find ourselves something multiple times, then we can start pulling it out into more abstractions (e.g. fixtures).

Alright! I found myself having very similar tests and therefore abstracted it to test_func. I completely agree with not abstracting too early and but when having the same structure several times.

Either way, it would be good to have a clear list of da_a and da_b and respective expected results. I am a bit confused about what we actually would expect for xarray.DataArray([[1, 2], [np.nan, np.nan]], dims=['x', 'y']).

Maybe, as a first step some type checking would be enough, to identify such special cases as the one above and give a ValueError or NotImplementedError?

@shoyer good spot, thanks

@r-beer we could add that as a test case

Yes, I will add it.

Thinking about test arrays in general: Wouldn't it be good to define some data array fixtures in the conftest.py which can then be used in all different test_function.py tests?

In view of the additional test I started to restructure the test_cov and test_corr to a single test_func. I am now asking myself how much of the data preprocessing shall go to the parametrization and how much remains in the test function...

Any guidelines or suggestions?

Either way, it would be good to have a clear list of da_a and da_b and respective expected results. I am a bit confused about what we actually would expect for xarray.DataArray([[1, 2], [np.nan, np.nan]], dims=['x', 'y']).

Turns out the assert np.allclose statements actually failed for cov, but not for corr: ```bash actual = xr.cov(ts1, ts2)

  assert np.allclose(expected, actual)

E assert False E + where False = <function allclose at 0x7ff5dbaf0e18>(0.004632253406505826, <xarray.DataArray ()>\narray(0.004323)) E + where <function allclose at 0x7ff5dbaf0e18> = np.allclose ``` 0.00463 vs. 0.00432

PS: I personally would like to use cov and corr also with dask. Does it also make sense to put this in a future PR, together with the above-mentioned other improvements?

Further PRs:

* Add `dot`, `cov` and `corr` to `computation.rst`
  > PS: It looks like `xr.dot` is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add `dot`, `cov` and `corr` to `computation.rst` in a future PR.

* Test (and if necessary implement) `dask` compatibility to `cov` and `corr`

* Improve documentation on `How to contribute to the xarray documentation?`
  > I think it already gives a good overview, but then it might be unclear how to proceed.
  > Maybe it might make sense to add a subsection `How to contribute to the xarray documentation?` in which the different options are shortly described examplary:
  > ```
  > 1. change docstring, e.g. ....
  > 2. update api.rst, e.g. ....
  > 3. update whats-new.rst, e.g. ...
  > ```
  > 
  > 
  > Then, underline that building the documentation locally is optional:
  > ```
  > Optionally, you can build the docs locally as described in the next section.
  > ```

Further PRs:

• Add dot, cov and corr to computation.rst

  PS: It looks like xr.dot is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add dot, cov and corr to computation.rst in a future PR.
  

• Test (and if necessary implement) dask compatibility to cov and corr

• Improve documentation on How to contribute to the xarray documentation?

  I think it already gives a good overview, but then it might be unclear how to proceed.
  
  Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary:
  
  1. change docstring, e.g. ....
  2. update api.rst, e.g. ....
  3. update whats-new.rst, e.g. ...
  
  Then, underline that building the documentation locally is optional:
  
  Optionally, you can build the docs locally as described in the next section.
  

• Extend xarray functionality, such that pairwise correlations and covariances can be easily calculated. Either write a wrapper function or extend xr.corr and xr.cov, such that the following computation is supported with ease: python pairwise_dim = 'space' for region in da_a[pairwise_dim]: print(xr.corr(da_a.sel(space=region.item()), da_a, dim='time').values) Target output would be an xarray with pairwise correlations, e.g.: bash [ 1. -0.43942827 -0.22425502] [-0.43942827 1. 0.30940081] [-0.22425502 0.30940081 1. ]

@dcherian, @max-sixty, @keewis, @Hoeze, second try: any more improvements required?

I'm not sure what's going on with the formatting. If you run black --version and black . from the root of the repo, what do you get?

black --version:

shell black, version 19.10b0

black .:

shell reformatted ...\xarray\xarray\core\dataarray.py All done! ✨ �🍰✨✨ 1 file reformatted, 143 files left unchanged.

Now, I additionally get errors from pre-commit claiming that GIT: unstaged files detected.

Guess, the pre-commit installation was not completely correct?

I'm not sure what's going on with the formatting. If you run black --version and black . from the root of the repo, what do you get?

black --version: bash black, version 19.10b0 black .: bash reformatted ...\xarray\xarray\core\dataarray.py All done! ✨ �🍰✨✨ 1 file reformatted, 143 files left unchanged.

PS: I personally would like to use cov and corr also with dask. Does it also make sense to put this in a future PR, together with the above-mentioned other improvements?

Further PRs: - Add dot, cov and corr to computation.rst

PS: It looks like xr.dot is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add dot, cov and corr to computation.rst in a future PR. - Test (and if necessary implement) dask compatibility to cov and corr - Improve documentation on How to contribute to the xarray documentation? I think it already gives a good overview, but then it might be unclear how to proceed.

Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary:

1. change docstring, e.g. .... 2. update api.rst, e.g. .... 3. update whats-new.rst, e.g. ...

Then, underline that building the documentation locally is optional:

Optionally, you can build the docs locally as described in the next section.

@dcherian, @max-sixty, @keewis, @Hoeze, this is it, is it?

Let me know if there are some adjustments to be made. 🙂

And I still get trailing whitespace errors, despite having set up pre-commit to run black and flake8 automatically. Do I have to set some parameters to delete the trailing whitespaces?

Computation? Where is xr.dot documented?

xr.dot is implemented in computation, what do you mean by "documented"?

PS: accidently closed and commented instead of commenting

@r-beer let's go with @shoyer 's suggestion to change this from a method to a function xr.cov & xr.corr. Is that OK?

@shoyer, @max-sixty, @keewis, where should these functions be placed?

my first guesses: - computation - dataarray

(lmk if the docs were unclear and we can update)

I think it already gives a good overview, but then it might be unclear how to proceed.

Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary: 1. change docstring, e.g. .... 2. update api.rst, e.g. .... 3. update whats-new.rst, e.g. ... Then, underline that building the documentation locally is optional:

Optionally, you can build the docs locally as described in the next section.

For the sake of completeness (and to already learn the procedure) I just added some information to whats-new.rst and api.rst for the methods da.cov and da.corr.

I would be happy for feedback!

Tomorrow, I will rewrite the methods to functions and go through all the steps again. Good night from Germany!

@r-beer let's go with @shoyer 's suggestion to change this from a method to a function xr.cov & xr.corr. Is that OK?

Sure! I actually asked myself a similar software design question in another project.

Shouldn't be too much effort I guess? I will tackle it tomorrow. 😴

I am now looking into the generation of the documentation.

What do you mean by this? You shouldn't have to generate anything locally?

Oh, ok. I thought I have to create xarray-docs to update whats-new.rst and api.rst?

PS: Environment creation worked flawlessly. It's a pleasure to get into xarray contribution, thanks to you guys! 😃

@r-beer That's strange, could it be a browser cache issue? I took the screenshot from the stable version of the docs (http://xarray.pydata.org/en/stable/contributing.html#requirements).

Indeed, I deleted the cache and the documentation refreshed. Thanks!

I think this was already updated in #3410? Here's what I see in the contributing guidelines:

Ok, thanks for the information. I followed the official guidelines, but will double-check with the dev version from now on.

@r-beer your current test case looks fine, no stress on the fixtures!

Alright!

I am now looking into the generation of the documentation. However, it seems like the contributing guidelines are not up-to-date, as it's not possible to create the xarray-docs environment (see #3556).

@max-sixty, I have found the da(request) fixture. Do I correctly understand that the tests you mentioned above can be implemented using this fixture and parametrize decorators as in @pytest.mark.parametrize("da", (1, 2), indirect=True)?

I have tried to wrap my head around the elegant use of fixtures and parametrize but didn't get there yet. If you have some time to show a more elegant solution, I would be happy to learn! If not, I guess this should work, too.

Are the examples OK like this?

I'm not sure about the examples in the examples folder, I think those are real-world examples?

What I meant was examples like those in DataArray.integrate (source) that simply demonstrate how to use that function.

OK! That simplifies the tasks quite a bit 🙂

I will implement this and the special cases for the tests, I might add Jupyter Notebook examples later to the documentation, when the dataset.corr functionality is also implemented.

We could also add some variants which test along different dimensions / all NaNs / other special cases.

@max-sixty, I have found the da(request) fixture. Do I correctly understand that the tests you mentioned above can be implemented using this fixture and parametrize decorators as in @pytest.mark.parametrize("da", (1, 2), indirect=True)?

Concerning the examples, I have started to create a Jupyter Notebook in the examples folder. Is this the right way to create the examples?

I have seen that the existing examples contain html code. Why is that? Why not only markdown?

Test looks good. Let's make those adjustments you put in the TODO and add one for the other function?

@max-sixty, do you mean to add a TODO to the function mentioned in this commit: https://github.com/pydata/xarray/pull/3550/commits/d57eb7cfaf1407179080f0ca95e7951e5622b493?

Here, I think the @requires_scipy_or_netCDF4 is actually necessary: python @requires_scipy_or_netCDF4 @requires_dask @pytest.mark.filterwarnings("ignore:use make_scale(name) instead") def test_open_mfdataset_manyfiles(

Great start!

Thanks for the review @keewis

I think it's fine to leave Dataset for another PR; it'd be great to get this in, and given it's your first PR (welcome!) let's not bloat it

Test looks good. Let's make those adjustments you put in the TODO and add one for the other function? We could also add some variants which test along different dimensions / all NaNs / other special cases.

Thank you two, @max-sixty and @keewis, for the constructive feedback that made the start easier! I would also be happy to go once through the whole process and then implement the PR for the ds separately. This will probably cut the learning curve and accelerate the process.

Additionally, I wonder whether I have to configure black for the spaces between colons?

I had autoformatting running with autopep8 previously but since one of the last VS code updates something broke so I will set this up properly again on Friday or tomorrow.

since the APIs of DataArray and Dataset are usually kept in sync as much as possible, it might be good to add the new methods to Dataset, too. This doesn't have to be in this PR, though, and I'd wait for confirmation first.

Yes, the functionality that I would be very interested in would be to calculate corr() also between different variables of a dataset as in pandas.DataFrame.corr as well as correlations between different locations within the same DataArray (e.g. correlations of time series with different latitudes).

for the new commit, yes. The older commits could be rewritten if you don't like the other email address being public (I guide you through the process if you want) but since PRs get squashed after the merge you don't have to.

OK, no need for the extra effort at this point, the squashing at the end of PR is fine for me. Thank you for the offer, anyway! 🙂

Are any other changes necessary for the PR?

ideas: - examples for corr and cov usage - whats-new.rst

There seems to be a problem with your email address (github can't match it to your account). Does it match the one you used for signing up on github?

Dear @keewis, I just set my real email address also as public email address and set my local git user name and email address accordingly. Does this solve the problem?

@max-sixty, all three checks fail because test_open_mfdataset_manyfiles requires either netcdf4 or scipy.

Shall I, therefore, add the @requires_scipy_or_netCDF4 decorator?

Thanks for the fast response @crusaderky!

Alright, I have done so and changed basestrings into str, now the tests run (mostly) through locally. bash 1 failed, 6539 passed, 1952 skipped, 37 xfailed, 31 warnings in 86.34s A general question concerning collaboration on existing PRs: Should I push the changes to my fork and then create a new PR that replaces @hrishikeshac's PR or shall I push to @hrishikeshac's fork and see whether it works to update the existing PR?

Or is there another option?

PS: Permission to push to hrishikeshac:corr is denied for me.

Alright, I only got two merge conflicts in dataarray.py:

minor merge conflict concerning imports: 1. accessors -> accessors_td 2. broadcast has been dropped in master?

Secondly, some bigger merge conflicts concerning some of dataarray's methods, but they seem to be not in conflict with each other: 1. integrate(), unify_chunks() and map_blocks added in master 2. cov() and corr() added in corr branch 3. It seems like str = property(StringAccessor) should be at the end of dataarray's definition, for mypy reasons...

Can you please comment my suggested changes (accepting either changes from master or both, if no conflicts).

@max-sixty, thanks for the fast response!

Yeah, I get the traceback and already started diving into it. However, I assumed that @hrishikeshac's branch "corr" wasn't up-to-date.

Shall I merge changes from master or develop into corr, before looking further into the tests?

I read http://xarray.pydata.org/en/stable/contributing.html, is this identical to contributing.rst? Following those guidelines, I would use the following commands to "retrieve the changes from the master branch": git fetch upstream git rebase upstream/master

Where upstream = https://github.com/pydata/xarray.git?

@max-sixty, thanks for the fast response!

Yeah, I get the traceback and already started diving into it. However, I assumed that @hrishikeshac's branch "corr" wasn't up-to-date.

Shall I merge changes from master or develop into corr, before looking further into the tests?

Dear @Hoeze, I will (try to) finalize this merge request, as I am also very interested in this functionality.

I am new to xarray and contribution. I downloaded @hrishikeshac's code and ran the pytest tests locally.

I get 17 failed, 2088 passed, 2896 skipped, 7 xfailed, 754 warnings in 49.95s.

Is there an elegant way to share "which tests failed where" in order to avoid that I try to fix tests, that might already have been fixed in other branches?

I will already start to get a better understanding of why the tests fail and try to fix them in the meantime.

@r-beer would be great to finish this off! I think this would be a popular feature. You could take @hrishikeshac 's code (which is close!) and make the final changes.

OK, that means to make #2652 pass, right?

I downloaded the respective branch from @hrishikeshac, and ran the tests locally.

See respective discussion in #2652.

I am also highly interested in this function and in contributing to xarray in general!

If I understand correctly, https://github.com/pydata/xarray/pull/2350 and https://github.com/pydata/xarray/pull/2652 do not solve this PR, do they?

How can I help you finishing these PRs?