issue_comments

@r-beer in case you come back to see this: thank you for taking it so far; your code was helpful to eventually getting this feature in. And we'd of course appreciate any additional contributions.

Hello @r-beer! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file xarray/tests/test_computation.py:

Line 817:1: E302 expected 2 blank lines, found 1

Comment last updated at 2019-11-25 20:47:29 UTC

I have added several test cases, almost all pass. The ones that don't pass are related to dim="time" and incompletely implemented pandas_cov and pandas_corr, that currently don't have a dim parameter.

I had a look at np.cov, np.corrcoef, pd.Series.cov, pd.frame.cov, pd.Series.corr, and pd.frame.corr: Computation of cov and corr is implemented in many different ways.

xr.cov and xr.corr now imitate the behavior of the respective pd.Series functions.

I actually prefer the numpy behavior, resulting in covariance and correlation matrices. However I feel that efficient implementation of this behavior is above my current understanding of xarray. So, I would highly appreciate your support on this implementation!

Otherwise, I added the ddof parameter to xr.cov to imitate the behavior of the respective pd.Series functions. I now get the results that pd.Series produces. However, I had to set ddof=1 for xr.cov and ddof=0 for xr.corr. Does that make sense?

For xarray.corr(array, array, dim='y'), I think we'd expect an array of the same dimensionality as below, with one point as the correlation of 1.0 (because {1,2} is 100% correlated to {1,2}) and another NaN. Does that make sense?

On the one side, I am with you in terms of "What You Put In Is What You Get Out", on the other hand pandas.Series.cov and np.cov have other behaviors that both seem plausible:

So for the moment, we might stick with pandas.Series.cov? Or (future PR?) we rather want the np.cov behavior, this would require more effort and the repr should probably be given as xarray instead of np.array, right?

I would start with having the code in the function; i.e. don't abstract too early unless we're really confident on the eventual state. When we find ourselves something multiple times, then we can start pulling it out into more abstractions (e.g. fixtures).

Alright! I found myself having very similar tests and therefore abstracted it to test_func. I completely agree with not abstracting too early and but when having the same structure several times.

Either way, it would be good to have a clear list of da_a and da_b and respective expected results. I am a bit confused about what we actually would expect for xarray.DataArray([[1, 2], [np.nan, np.nan]], dims=['x', 'y']).

For xarray.corr(array, array, dim='y'), I think we'd expect an array of the same dimensionality as below, with one point as the correlation of 1.0 (because {1,2} is 100% correlated to {1,2}) and another NaN. Does that make sense?

In [17]: xr.dot(da,da, dims='y') Out[17]: <xarray.DataArray (x: 2)> array([ 5., nan]) Dimensions without coordinates: x

@shoyer good spot, thanks @r-beer we could add that as a test case

Yes, I will add it.

Thinking about test arrays in general: Wouldn't it be good to define some data array fixtures in the conftest.py which can then be used in all different test_function.py tests?

I think we could do more of this, yes. It's a tradeoff between (a) better sharing & performance vs. (b) customization for each function & localization of effects. We have def create_test_data in a few files, before we used pytest, so could convert some of the usages to fixtures. @shoyer and I discussed this briefly with @keewis 's work on tests for units. I think @shoyer is keener on having fixture-like code close to the test functions.

In view of the additional test I started to restructure the test_cov and test_corr to a single test_func. I am now asking myself how much of the data preprocessing shall go to the parametrization and how much remains in the test function...

Any guidelines or suggestions?

I would start with having the code in the function; i.e. don't abstract too early unless we're really confident on the eventual state. When we find ourselves something multiple times, then we can start pulling it out into more abstractions (e.g. fixtures).

Either way, it would be good to have a clear list of da_a and da_b and respective expected results. I am a bit confused about what we actually would expect for xarray.DataArray([[1, 2], [np.nan, np.nan]], dims=['x', 'y']).

Maybe, as a first step some type checking would be enough, to identify such special cases as the one above and give a ValueError or NotImplementedError?

@shoyer good spot, thanks

@r-beer we could add that as a test case

Yes, I will add it.

Thinking about test arrays in general: Wouldn't it be good to define some data array fixtures in the conftest.py which can then be used in all different test_function.py tests?

In view of the additional test I started to restructure the test_cov and test_corr to a single test_func. I am now asking myself how much of the data preprocessing shall go to the parametrization and how much remains in the test function...

Any guidelines or suggestions?

Either way, it would be good to have a clear list of da_a and da_b and respective expected results. I am a bit confused about what we actually would expect for xarray.DataArray([[1, 2], [np.nan, np.nan]], dims=['x', 'y']).

@shoyer good spot, thanks

@r-beer we could add that as a test case

Turns out the assert np.allclose statements actually failed for cov, but not for corr: ```bash actual = xr.cov(ts1, ts2)

  assert np.allclose(expected, actual)

E assert False E + where False = <function allclose at 0x7ff5dbaf0e18>(0.004632253406505826, <xarray.DataArray ()>\narray(0.004323)) E + where <function allclose at 0x7ff5dbaf0e18> = np.allclose ``` 0.00463 vs. 0.00432

@r-beer great list of future PRs!

@shoyer take a glance if you get a second. I'm deliberately pushing this through even though https://github.com/pydata/xarray/pull/3550/files#diff-293280a1ed0154c9fb35b842f69e7e33R1112 isn't resolved, since I think these are important features which have dragged for years, and we can iterate on the performance (and @r-beer has done a great job on his first PR!)

PS: I personally would like to use cov and corr also with dask. Does it also make sense to put this in a future PR, together with the above-mentioned other improvements?

Further PRs:

* Add `dot`, `cov` and `corr` to `computation.rst`
  > PS: It looks like `xr.dot` is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add `dot`, `cov` and `corr` to `computation.rst` in a future PR.

* Test (and if necessary implement) `dask` compatibility to `cov` and `corr`

* Improve documentation on `How to contribute to the xarray documentation?`
  > I think it already gives a good overview, but then it might be unclear how to proceed.
  > Maybe it might make sense to add a subsection `How to contribute to the xarray documentation?` in which the different options are shortly described examplary:
  > ```
  > 1. change docstring, e.g. ....
  > 2. update api.rst, e.g. ....
  > 3. update whats-new.rst, e.g. ...
  > ```
  > 
  > 
  > Then, underline that building the documentation locally is optional:
  > ```
  > Optionally, you can build the docs locally as described in the next section.
  > ```

Further PRs:

• Add dot, cov and corr to computation.rst

  PS: It looks like xr.dot is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add dot, cov and corr to computation.rst in a future PR.
  

• Test (and if necessary implement) dask compatibility to cov and corr

• Improve documentation on How to contribute to the xarray documentation?

  I think it already gives a good overview, but then it might be unclear how to proceed.
  
  Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary:
  
  1. change docstring, e.g. ....
  2. update api.rst, e.g. ....
  3. update whats-new.rst, e.g. ...
  
  Then, underline that building the documentation locally is optional:
  
  Optionally, you can build the docs locally as described in the next section.
  

• Extend xarray functionality, such that pairwise correlations and covariances can be easily calculated. Either write a wrapper function or extend xr.corr and xr.cov, such that the following computation is supported with ease: python pairwise_dim = 'space' for region in da_a[pairwise_dim]: print(xr.corr(da_a.sel(space=region.item()), da_a, dim='time').values) Target output would be an xarray with pairwise correlations, e.g.: bash [ 1. -0.43942827 -0.22425502] [-0.43942827 1. 0.30940081] [-0.22425502 0.30940081 1. ]

@dcherian, @max-sixty, @keewis, @Hoeze, second try: any more improvements required?

I'm not sure what's going on with the formatting. If you run black --version and black . from the root of the repo, what do you get?

black --version:

shell black, version 19.10b0

black .:

shell reformatted ...\xarray\xarray\core\dataarray.py All done! ✨ �🍰✨✨ 1 file reformatted, 143 files left unchanged.

Now, I additionally get errors from pre-commit claiming that GIT: unstaged files detected.

Guess, the pre-commit installation was not completely correct?

I'm not sure what's going on with the formatting. If you run black --version and black . from the root of the repo, what do you get?

black --version: bash black, version 19.10b0 black .: bash reformatted ...\xarray\xarray\core\dataarray.py All done! ✨ �🍰✨✨ 1 file reformatted, 143 files left unchanged.

PS: I personally would like to use cov and corr also with dask. Does it also make sense to put this in a future PR, together with the above-mentioned other improvements?

Further PRs: - Add dot, cov and corr to computation.rst

PS: It looks like xr.dot is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add dot, cov and corr to computation.rst in a future PR. - Test (and if necessary implement) dask compatibility to cov and corr - Improve documentation on How to contribute to the xarray documentation? I think it already gives a good overview, but then it might be unclear how to proceed.

Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary:

1. change docstring, e.g. .... 2. update api.rst, e.g. .... 3. update whats-new.rst, e.g. ...

Then, underline that building the documentation locally is optional:

Optionally, you can build the docs locally as described in the next section.

@dcherian, @max-sixty, @keewis, @Hoeze, this is it, is it?

Let me know if there are some adjustments to be made. 🙂

And I still get trailing whitespace errors, despite having set up pre-commit to run black and flake8 automatically. Do I have to set some parameters to delete the trailing whitespaces?

where should these functions be placed?

Sorry, I misunderstood this question. I thought you were asking about documentation but I see that you're asking about which .py file to stick these functions in. Let's do computation just like xr.dot.

PS: It looks like xr.dot is only present in the "API reference" and not in https://xarray.pydata.org/en/stable/computation.html. We should add dot, cov and corr to computation.rst in a future PR.

Computation? Where is xr.dot documented?

xr.dot is implemented in computation, what do you mean by "documented"?

Computation? Where is xr.dot documented?

Awesome, thanks a lot @r-beer!

PS: accidently closed and commented instead of commenting

@r-beer let's go with @shoyer 's suggestion to change this from a method to a function xr.cov & xr.corr. Is that OK?

@shoyer, @max-sixty, @keewis, where should these functions be placed?

my first guesses: - computation - dataarray

(lmk if the docs were unclear and we can update)

I think it already gives a good overview, but then it might be unclear how to proceed.

Maybe it might make sense to add a subsection How to contribute to the xarray documentation? in which the different options are shortly described examplary: 1. change docstring, e.g. .... 2. update api.rst, e.g. .... 3. update whats-new.rst, e.g. ... Then, underline that building the documentation locally is optional:

Optionally, you can build the docs locally as described in the next section.

For the sake of completeness (and to already learn the procedure) I just added some information to whats-new.rst and api.rst for the methods da.cov and da.corr.

I would be happy for feedback!

Tomorrow, I will rewrite the methods to functions and go through all the steps again. Good night from Germany!

@r-beer let's go with @shoyer 's suggestion to change this from a method to a function xr.cov & xr.corr. Is that OK?

Sure! I actually asked myself a similar software design question in another project.

Shouldn't be too much effort I guess? I will tackle it tomorrow. 😴

Oh, ok. I thought I have to create xarray-docs to update whats-new.rst and api.rst?

You can just edit the files directly! No need to build all the docs. It should be much simpler than the work you've already done!

(lmk if the docs were unclear and we can update)

I am now looking into the generation of the documentation.

What do you mean by this? You shouldn't have to generate anything locally?

Oh, ok. I thought I have to create xarray-docs to update whats-new.rst and api.rst?

@r-beer let's go with @shoyer 's suggestion to change this from a method to a function xr.cov & xr.corr. Is that OK?

I am now looking into the generation of the documentation.

What do you mean by this? You shouldn't have to generate anything locally?

@r-beer your current test case looks fine, no stress on the fixtures!

Alright!

I am now looking into the generation of the documentation. However, it seems like the contributing guidelines are not up-to-date, as it's not possible to create the xarray-docs environment (see #3556).

One question we started discussing on the old PR which we should resolve ASAP given @r-beer 's work: do we want xr.cov(da1, da2) or da1.cov(da2)?

In my opinion the function form is clearer, because cor and corr are symmetric in both arguments.

@r-beer your current test case looks fine, no stress on the fixtures!

One question we started discussing on the old PR which we should resolve ASAP given @r-beer 's work: do we want xr.cov(da1, da2) or da1.cov(da2)?

CC @fujiisoup @shoyer

@max-sixty, I have found the da(request) fixture. Do I correctly understand that the tests you mentioned above can be implemented using this fixture and parametrize decorators as in @pytest.mark.parametrize("da", (1, 2), indirect=True)?

I have tried to wrap my head around the elegant use of fixtures and parametrize but didn't get there yet. If you have some time to show a more elegant solution, I would be happy to learn! If not, I guess this should work, too.

Are the examples OK like this?

I'm not sure about the examples in the examples folder, I think those are real-world examples?

What I meant was examples like those in DataArray.integrate (source) that simply demonstrate how to use that function.

OK! That simplifies the tasks quite a bit 🙂

I will implement this and the special cases for the tests, I might add Jupyter Notebook examples later to the documentation, when the dataset.corr functionality is also implemented.

I'm not sure about the examples in the examples folder, I think those are real-world examples?

What I meant was examples like those in DataArray.integrate (source) that simply demonstrate how to use that function.

We could also add some variants which test along different dimensions / all NaNs / other special cases.

@max-sixty, I have found the da(request) fixture. Do I correctly understand that the tests you mentioned above can be implemented using this fixture and parametrize decorators as in @pytest.mark.parametrize("da", (1, 2), indirect=True)?

Concerning the examples, I have started to create a Jupyter Notebook in the examples folder. Is this the right way to create the examples?

I have seen that the existing examples contain html code. Why is that? Why not only markdown?

Test looks good. Let's make those adjustments you put in the TODO and add one for the other function?

@max-sixty, do you mean to add a TODO to the function mentioned in this commit: https://github.com/pydata/xarray/pull/3550/commits/d57eb7cfaf1407179080f0ca95e7951e5622b493?

Here, I think the @requires_scipy_or_netCDF4 is actually necessary: python @requires_scipy_or_netCDF4 @requires_dask @pytest.mark.filterwarnings("ignore:use make_scale(name) instead") def test_open_mfdataset_manyfiles(

Great start!

Thanks for the review @keewis

I think it's fine to leave Dataset for another PR; it'd be great to get this in, and given it's your first PR (welcome!) let's not bloat it

Test looks good. Let's make those adjustments you put in the TODO and add one for the other function? We could also add some variants which test along different dimensions / all NaNs / other special cases.

Thank you two, @max-sixty and @keewis, for the constructive feedback that made the start easier! I would also be happy to go once through the whole process and then implement the PR for the ds separately. This will probably cut the learning curve and accelerate the process.

Additionally, I wonder whether I have to configure black for the spaces between colons?

I had autoformatting running with autopep8 previously but since one of the last VS code updates something broke so I will set this up properly again on Friday or tomorrow.

Great start!

Thanks for the review @keewis

I think it's fine to leave Dataset for another PR; it'd be great to get this in, and given it's your first PR (welcome!) let's not bloat it

Test looks good. Let's make those adjustments you put in the TODO and add one for the other function? We could also add some variants which test along different dimensions / all NaNs / other special cases.

since the APIs of DataArray and Dataset are usually kept in sync as much as possible, it might be good to add the new methods to Dataset, too. This doesn't have to be in this PR, though, and I'd wait for confirmation first.

Yes, the functionality that I would be very interested in would be to calculate corr() also between different variables of a dataset as in pandas.DataFrame.corr as well as correlations between different locations within the same DataArray (e.g. correlations of time series with different latitudes).

since the APIs of DataArray and Dataset are usually kept in sync as much as possible, it might be good to add the new methods to Dataset, too. This doesn't have to be in this PR, though, and I'd wait for confirmation first.

for the new commit, yes. The older commits could be rewritten if you don't like the other email address being public (I can guide you through the process if you want) but since PRs get squashed during the merge you don't have to.

yes, whats-new.rst and api.rst will have to eventually be updated before the merge. Also, examples would be really good.

for the new commit, yes. The older commits could be rewritten if you don't like the other email address being public (I guide you through the process if you want) but since PRs get squashed after the merge you don't have to.

OK, no need for the extra effort at this point, the squashing at the end of PR is fine for me. Thank you for the offer, anyway! 🙂

Are any other changes necessary for the PR?

ideas: - examples for corr and cov usage - whats-new.rst

There seems to be a problem with your email address (github can't match it to your account). Does it match the one you used for signing up on github?

Dear @keewis, I just set my real email address also as public email address and set my local git user name and email address accordingly. Does this solve the problem?

There seems to be a problem with your email address (github can't match it to your account). Does it match the one you used for signing up on github?

@max-sixty, all three checks fail because test_open_mfdataset_manyfiles requires either netcdf4 or scipy.

Shall I, therefore, add the @requires_scipy_or_netCDF4 decorator?