html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/pull/5284#issuecomment-849811814,https://api.github.com/repos/pydata/xarray/issues/5284,849811814,MDEyOklzc3VlQ29tbWVudDg0OTgxMTgxNA==,5635139,2021-05-27T17:31:02Z,2021-05-27T17:31:02Z,MEMBER,Thank you very much @AndrewWilliams3142 !,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-849169576,https://api.github.com/repos/pydata/xarray/issues/5284,849169576,MDEyOklzc3VlQ29tbWVudDg0OTE2OTU3Ng==,5635139,2021-05-26T22:46:43Z,2021-05-26T22:46:43Z,MEMBER,Please feel free to add a whatsnew @AndrewWilliams3142 ,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-848612330,https://api.github.com/repos/pydata/xarray/issues/5284,848612330,MDEyOklzc3VlQ29tbWVudDg0ODYxMjMzMA==,56925856,2021-05-26T09:19:50Z,2021-05-26T09:19:50Z,CONTRIBUTOR,"Hey both, I've added a test to check that dask doesn't compute when calling either `xr.corr()` or `xr.cov()`, and also that the end result is still a `dask ` array. Let me know if there's anything I've missed though! thanks for the help :) @dcherian, regarding the `apply_ufunc` approach, I might leave that for now but as you said it can always be a future PR","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-838231568,https://api.github.com/repos/pydata/xarray/issues/5284,838231568,MDEyOklzc3VlQ29tbWVudDgzODIzMTU2OA==,56925856,2021-05-11T10:28:08Z,2021-05-12T20:45:00Z,CONTRIBUTOR,"Thanks for that @dcherian ! I didn't know you could use print debugging on chunked operations like this! One thing actually: If I change `da = da.where(missing_vals)` to `da = da.where(~missing_vals)` then we get the results we'd expect. Do you think this fixes the problem? ``` def _get_valid_values(da, other): da1, da2 = xr.align(da, other, join=""outer"", copy=False) # 2. Ignore the nans missing_vals = np.logical_or(da1.isnull(), da2.isnull()) if missing_vals.any(): da = da.where(~missing_vals) return da else: return da ``` ``` print(da_a.map_blocks(_get_valid_values, args=[da_b]).compute()) array([[1. , 2. , 3. , 4. ], [1. , 0.1, 0.2, 0.3], [2. , 3.2, nan, 1.8]]) Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04 * space (space) object 'IA' 'IL' 'IN' * ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-837097308,https://api.github.com/repos/pydata/xarray/issues/5284,837097308,MDEyOklzc3VlQ29tbWVudDgzNzA5NzMwOA==,2448579,2021-05-10T18:25:29Z,2021-05-10T18:25:29Z,MEMBER,"Well that was confusing! `if missing_vals.any():` will not be triggered if all the values in a block are valid. With `.chunk({""time"": 1})`, some of the blocks are all valid. I discovered this with `print` debugging and the single-threaded scheduler which loops over blocks in a for-loop ``` # Define function to use in map_blocks def _get_valid_values(da, other): da1, da2 = xr.align(da, other, join=""inner"", copy=False) # 2. Ignore the nans missing_vals = np.logical_or(da1.isnull(), da2.isnull()) print(missing_vals) if missing_vals.any(): da = da.where(missing_vals) return da da_a.map_blocks(_get_valid_values, args=[da_b]).compute(scheduler=""sync"") ``` For better performance, we should try a `dask.array.map_blocks` approach with `duck_array_ops.isnull` ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-837045902,https://api.github.com/repos/pydata/xarray/issues/5284,837045902,MDEyOklzc3VlQ29tbWVudDgzNzA0NTkwMg==,14808389,2021-05-10T17:53:05Z,2021-05-10T17:53:05Z,MEMBER,"#4559 and #4668 fixed a similar issue using `map_blocks`, maybe you can use that as a reference?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-837032429,https://api.github.com/repos/pydata/xarray/issues/5284,837032429,MDEyOklzc3VlQ29tbWVudDgzNzAzMjQyOQ==,56925856,2021-05-10T17:44:29Z,2021-05-10T17:44:29Z,CONTRIBUTOR,"Hi @dcherian , just thinking about your suggestion for using `map_blocks` on the actual `valid_values` check. I've tested this and was wondering if you could maybe point to where I'm going wrong? It does mask out some of the values in a lazy way, but not the *correct* ones. ```python3 da_a = xr.DataArray( np.array([[1, 2, 3, 4], [1, 0.1, 0.2, 0.3], [2, 3.2, 0.6, 1.8]]), dims=(""space"", ""time""), coords=[ (""space"", [""IA"", ""IL"", ""IN""]), (""time"", pd.date_range(""2000-01-01"", freq=""1D"", periods=4)), ], ).chunk({'time':1}) da_b = xr.DataArray( np.array([[0.2, 0.4, 0.6, 2], [15, 10, 5, 1], [1, 3.2, np.nan, 1.8]]), dims=(""space"", ""time""), coords=[ (""space"", [""IA"", ""IL"", ""IN""]), (""time"", pd.date_range(""2000-01-01"", freq=""1D"", periods=4)), ], ).chunk({'time':1}) print(da_a) >>> array([[1. , 2. , 3. , 4. ], [1. , 0.1, 0.2, 0.3], [2. , 3.2, 0.6, 1.8]]) Coordinates: * space (space) >> array([[ 0.2, 0.4, 0.6, 2. ], [15. , 10. , 5. , 1. ], [ 1. , 3.2, nan, 1.8]]) Coordinates: * space (space) >> array([[1. , 2. , nan, 4. ], [1. , 0.1, nan, 0.3], [2. , 3.2, 0.6, 1.8]]) Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04 * space (space) object 'IA' 'IL' 'IN' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804 https://github.com/pydata/xarray/pull/5284#issuecomment-835907226,https://api.github.com/repos/pydata/xarray/issues/5284,835907226,MDEyOklzc3VlQ29tbWVudDgzNTkwNzIyNg==,5635139,2021-05-09T22:15:25Z,2021-05-09T22:15:25Z,MEMBER,"Hi @AndrewWilliams3142 , thanks for another PR! This looks good. Could we add a test like https://github.com/pydata/xarray/pull/4559/files#diff-74d2dc289aa601b2de094fb3a3b687fd65963401b51b95cc5e0afcd06cc4cb82R45? And maybe reference this or #4559 as an explanation of what's going on.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,882876804