html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/3066#issuecomment-508158344,https://api.github.com/repos/pydata/xarray/issues/3066,508158344,MDEyOklzc3VlQ29tbWVudDUwODE1ODM0NA==,1217238,2019-07-03T16:10:39Z,2019-07-03T16:11:17Z,MEMBER,"@mrezak Thanks for the report and the clear example! Certainly this is an annoying inconsistency. I'm trying to figure out whether this is also a bug or not. I think the difference comes down to how pandas and xarray pass data into the `custom` function. Pandas passes individual slices, trimming out values outside the window. Xarray passes an N+1 dimensional view of the array data with extra dimension added for the ""window offset"", with values outside the window filled with NaN: ```python import numpy as np import pandas as pd import xarray def custom(x, axis=0): print(x) return np.mean(x, axis) print('pandas example') d = pd.DataFrame(np.random.rand(11,3)) r = d.rolling(10, min_periods=5).apply(custom) print(r.iloc[0:10,:]) print('\nxarray example') xd = d.to_xarray().to_array() r = xd.rolling(index=10, min_periods=5).reduce(custom) print(r[:,0:10]) ``` Output:
``` pandas example [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209 0.08384912] [0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209 0.08384912 0.23068875] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 ] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566 0.18844817] [0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566 0.18844817 0.51895952] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 ] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 0.4677236 ] [0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 0.4677236 0.7083373 ] 0 1 2 0 NaN NaN NaN 1 NaN NaN NaN 2 NaN NaN NaN 3 NaN NaN NaN 4 0.468570 0.643508 0.588603 5 0.427758 0.602433 0.630744 6 0.400894 0.545281 0.603162 7 0.468679 0.502798 0.638438 8 0.441866 0.488832 0.652639 9 0.406064 0.458794 0.634147 xarray example [[[ nan nan nan nan nan nan nan nan nan 0.06130714] [ nan nan nan nan nan nan nan nan 0.06130714 0.86751339] [ nan nan nan nan nan nan nan 0.06130714 0.86751339 0.06688379] [ nan nan nan nan nan nan 0.06130714 0.86751339 0.06688379 0.45866121] [ nan nan nan nan nan 0.06130714 0.86751339 0.06688379 0.45866121 0.88848511] [ nan nan nan nan 0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799] [ nan nan nan 0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828] [ nan nan 0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625] [ nan 0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209] [0.06130714 0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209 0.08384912] [0.86751339 0.06688379 0.45866121 0.88848511 0.22369799 0.23970828 0.94317625 0.22736209 0.08384912 0.23068875]] [[ nan nan nan nan nan nan nan nan nan 0.87929068] [ nan nan nan nan nan nan nan nan 0.87929068 0.81303738] [ nan nan nan nan nan nan nan 0.87929068 0.81303738 0.62778023] [ nan nan nan nan nan nan 0.87929068 0.81303738 0.62778023 0.34381748] [ nan nan nan nan nan 0.87929068 0.81303738 0.62778023 0.34381748 0.55361603] [ nan nan nan nan 0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802] [ nan nan nan 0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 ] [ nan nan 0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754] [ nan 0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566] [0.87929068 0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566 0.18844817] [0.81303738 0.62778023 0.34381748 0.55361603 0.39705802 0.2023665 0.20541754 0.37710566 0.18844817 0.51895952]] [[ nan nan nan nan nan nan nan nan nan 0.33501081] [ nan nan nan nan nan nan nan nan 0.33501081 0.67972562] [ nan nan nan nan nan nan nan 0.33501081 0.67972562 0.08622488] [ nan nan nan nan nan nan 0.33501081 0.67972562 0.08622488 0.89673242] [ nan nan nan nan nan 0.33501081 0.67972562 0.08622488 0.89673242 0.94532091] [ nan nan nan nan 0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888] [ nan nan nan 0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841] [ nan nan 0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995] [ nan 0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 ] [0.33501081 0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 0.4677236 ] [0.67972562 0.08622488 0.89673242 0.94532091 0.84144888 0.43766841 0.88536995 0.7662462 0.4677236 0.7083373 ]]] array([[ nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.406064], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.458794], [ nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.634147]]) Coordinates: * index (index) int64 0 1 2 3 4 5 6 7 8 9 * variable (variable) int64 0 1 2 ```
Xarray's version is certainly going to be way faster, but it has the downside of treating windows differently. One way to work around this would be to use `np.nanmean` inside `custom` instead of `np.mean`. cc @jhamman @fujiisoup who worked on this and may have ideas","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,462424005