{"database": "github", "table": "issues", "is_view": false, "human_description_en": "where repo = 13221727, state = \"closed\" and user = 25071375 sorted by updated_at descending", "rows": [[2215890029, "I_kwDOAMm_X86EE8xt", 8894, "Rolling reduction with a custom function generates an excesive use of memory that kills the workers", 25071375, "closed", 0, null, null, 8, "2024-03-29T19:15:28Z", "2024-04-01T20:57:59Z", "2024-03-30T01:49:17Z", "CONTRIBUTOR", null, null, null, "### What happened?\r\n\r\nHi, I have been trying to use a custom function on the rolling reduction method, the original function tries to filter the nan values (any numpy function that I have used that handles nans generates the same problem) to later apply some simple aggregate functions, but it is killing all my workers even when the data is very small (I have 7 workers and all of them have 3 Gb of RAM).\r\n\r\n### What did you expect to happen?\r\n\r\nI would expect less use of memory taking into account the size of the rolling window, the simplicity of the function and the amount of data used on the example.\r\n\r\n### Minimal Complete Verifiable Example\r\n\r\n```Python\r\nimport numpy as np\r\nimport dask.array as da\r\nimport xarray as xr\r\nimport dask\r\n\r\n\r\ndef f(x, axis):\r\n    # If I replace np.nansum by np.sum everything works perfectly and the amount of memory used is very small\r\n    return np.nansum(x, axis=axis)\r\n\r\narr = xr.DataArray(\r\n    dask.array.zeros(\r\n        shape=(300, 30000),\r\n        dtype=float,\r\n        chunks=(30, 6000)\r\n    ),\r\n    dims=[\"a\", \"b\"],\r\n    coords={\"a\": list(range(300)), \"b\": list(range(30000))}\r\n)\r\n\r\narr.rolling(a=252).reduce(f).chunk({\"a\": 252}).to_zarr(\"/data/test/test_write\", mode=\"w\")\r\n```\r\n\r\n\r\n### MVCE confirmation\r\n\r\n- [X] Minimal example \u2014 the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.\r\n- [X] Complete example \u2014 the example is self-contained, including all data and the text of any traceback.\r\n- [X] Verifiable example \u2014 the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.\r\n- [X] New issue \u2014 a search of GitHub Issues suggests this is not a duplicate.\r\n- [ ] Recent environment \u2014 the issue occurs with the latest version of xarray and its dependencies.\r\n\r\n### Relevant log output\r\n\r\n```Python\r\nKilledWorker: Attempted to run task ('nansum-overlap-sum-aggregate-sum-aggregate-e732de6ad917d5f4084b05192ca671c4', 0, 0) on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://172.18.0.2:39937. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.\r\n```\r\n\r\n\r\n### Anything else we need to know?\r\n\r\n_No response_\r\n\r\n### Environment\r\n\r\n<details>\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0]\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 4.14.275-207.503.amzn2.x86_64\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: en_US.UTF-8\r\nLANG: en_US.UTF-8\r\nLOCALE: ('en_US', 'UTF-8')\r\nlibhdf5: 1.14.3\r\nlibnetcdf: None\r\n\r\nxarray: 2024.1.0\r\npandas: 2.2.1\r\nnumpy: 1.26.3\r\nscipy: 1.11.4\r\nnetCDF4: None\r\npydap: None\r\nh5netcdf: None\r\nh5py: 3.10.0\r\nNio: None\r\nzarr: 2.16.1\r\ncftime: None\r\nnc_time_axis: None\r\niris: None\r\nbottleneck: 1.3.7\r\ndask: 2024.1.0\r\ndistributed: 2024.1.0\r\nmatplotlib: 3.8.2\r\ncartopy: None\r\nseaborn: 0.13.1\r\nnumbagg: 0.7.0\r\nfsspec: 2023.12.2\r\ncupy: None\r\npint: None\r\nsparse: 0.15.1\r\nflox: 0.8.9\r\nnumpy_groupies: 0.10.2\r\nsetuptools: 69.0.3\r\npip: 23.3.2\r\nconda: 23.11.0\r\npytest: 7.4.4\r\nmypy: None\r\nIPython: 8.20.0\r\nsphinx: None\r\n\r\n</details>\r\n", "{\"url\": \"https://api.github.com/repos/pydata/xarray/issues/8894/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", null, "completed", 13221727, "issue"], [865206283, "MDU6SXNzdWU4NjUyMDYyODM=", 5210, "Probable error using zarr process synchronizer", 25071375, "closed", 0, null, null, 1, "2021-04-22T17:05:10Z", "2023-10-14T20:36:18Z", "2023-10-14T20:36:18Z", "CONTRIBUTOR", null, null, null, "Hi I was trying to use Xarray open_zarr with the Zarr ProcessSynchronizer class and it produces a set of errors, I don't know if those errors are produced because I don't understand the logic of the ProcessSynchronizer or is a simple bug. I have a small code which reproduces the problems, basically, if I put a different path in the Zarr ProcessSynchronizer class all the error disappear but it creates a new folder.\r\n\r\n```python\r\nimport xarray \r\nimport zarr\r\nimport numpy as np\r\n\r\n\r\narr = xarray.DataArray(\r\n    data=np.array([\r\n        [1, 2, 7, 4, 5],\r\n        [np.nan, 3, 5, 5, 6],\r\n        [3, 3, np.nan, 5, 6],\r\n        [np.nan, 3, 10, 5, 6],\r\n        [np.nan, 7, 8, 5, 6],\r\n    ], dtype=float),\r\n    dims=['index', 'columns'],\r\n    coords={'index': [0, 1, 2, 3, 4], 'columns': [0, 1, 2, 3, 4]},\r\n)\r\n\r\n# If the synchronizer is created using another path the code will work without any error but it creates a new folder,\r\n# that is the correct way to use the process synchronizer?\r\n# synchronizer = zarr.ProcessSynchronizer('dummy_array.sync')\r\n\r\n# Using the original path produce a set of weird problems\r\nsynchronizer = zarr.ProcessSynchronizer('dummy_array')\r\n\r\n# Executing the commented code I obtain: PermissionError: [WinError 5].\r\n# arr.to_dataset(name='data').to_zarr('dummy_array', mode='w', synchronizer=synchronizer, compute=True)\r\n\r\narr.to_dataset(name='data').to_zarr('dummy_array', mode='w', compute=True)\r\n\r\n# If this section of the code is uncommented It will throw a different error when xarray.open_zarr being executed\r\n# a = zarr.open_array('dummy_array/data', synchronizer=synchronizer, mode='r')\r\n\r\n# PermissionError: [Errno 13] Permission denied\r\nxarray.open_zarr('dummy_array', synchronizer=synchronizer)\r\n```\r\n\r\n\r\n<details><summary>Output of <tt>xr.show_versions()</tt></summary>\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]\r\npython-bits: 64\r\nOS: Windows\r\nOS-release: 10\r\nmachine: AMD64\r\nprocessor: Intel64 Family 6 Model 165 Stepping 2, GenuineIntel\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: None\r\nLOCALE: es_ES.cp1252\r\nlibhdf5: 1.10.4\r\nlibnetcdf: None\r\n\r\nxarray: 0.17.0\r\npandas: 1.1.3\r\nnumpy: 1.19.2\r\nscipy: 1.5.2\r\nnetCDF4: None\r\npydap: None\r\nh5netcdf: None\r\nh5py: 2.10.0\r\nNio: None\r\nzarr: 2.7.1\r\ncftime: None\r\nnc_time_axis: None\r\nPseudoNetCDF: None\r\nrasterio: None\r\ncfgrib: None\r\niris: None\r\nbottleneck: 1.3.2\r\ndask: 2.30.0\r\ndistributed: 2.30.1\r\nmatplotlib: 3.3.2\r\ncartopy: None\r\nseaborn: 0.11.0\r\nnumbagg: None\r\npint: None\r\nsetuptools: 50.3.1.post20201107\r\npip: 21.0.1\r\nconda: 4.10.0\r\npytest: 6.2.3\r\nIPython: 7.19.0\r\nsphinx: 3.2.1\r\n\r\n\r\n</details>", "{\"url\": \"https://api.github.com/repos/pydata/xarray/issues/5210/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", null, "completed", 13221727, "issue"], [1089504942, "PR_kwDOAMm_X84wUTRX", 6118, "New algorithm for forward filling", 25071375, "closed", 0, null, null, 3, "2021-12-27T22:36:37Z", "2022-01-03T23:06:06Z", "2022-01-03T17:55:59Z", "CONTRIBUTOR", null, 0, "pydata/xarray/pulls/6118", "<!-- Feel free to remove check-list items aren't relevant to your change -->\r\n\r\n- [x] Closes #6112\r\n- [x] Tests added\r\n- [x] Passes `pre-commit run --all-files`\r\n- [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst`\r\n- [x] New functions/methods are listed in `api.rst`\r\n", "{\"url\": \"https://api.github.com/repos/pydata/xarray/issues/6118/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", null, null, 13221727, "pull"], [1088893989, "I_kwDOAMm_X85A5zQl", 6112, "Forward Fill not working when there are all-NaN chunks", 25071375, "closed", 0, null, null, 6, "2021-12-27T01:27:05Z", "2022-01-03T17:55:59Z", "2022-01-03T17:55:59Z", "CONTRIBUTOR", null, null, null, "<!-- Please include a self-contained copy-pastable example that generates the issue if possible.\r\n\r\nPlease be concise with code posted. See guidelines below on how to provide a good bug report:\r\n\r\n- Craft Minimal Bug Reports: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports\r\n- Minimal Complete Verifiable Examples: https://stackoverflow.com/help/mcve\r\n\r\nBug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.\r\n-->\r\n\r\n**What happened**: I'm working with a report dataset that only has data on some specific periods of time, the problem is that when I use the forward fill method it returns me many nans even on the last cells (it's a forward fill without limit).\r\n\r\n**What you expected to happen**: The array should not have nans in the last cells if it has data in any other cell or there should be a warning somewhere.\r\n\r\n**Minimal Complete Verifiable Example**:\r\n\r\n```python\r\nimport xarray as xr\r\n\r\nxr.DataArray(\r\n    [1, 2, np.nan, np.nan, np.nan, np.nan],\r\n    dims=['a']\r\n).chunk(\r\n    2\r\n).ffill(\r\n    'a'\r\n).compute()\r\n```\r\noutput: array([ 1.,  2.,  2.,  2., nan, nan])\r\n\r\n**Anything else we need to know?**: I check a little bit the internal code of Xarray for forward filling when it use dask and I think that the problem is that the algorithm makes an initial forward fill on all the blocks and then it makes a map_overlap for forward filling between chunks which in case that there is an empty chunk will not work due that it is going to take the last value of the empty chunk which is nan (hope this help).\r\n**Environment**:\r\n\r\n<details><summary>Output of <tt>xr.show_versions()</tt></summary>\r\n\r\n<!-- Paste the output here xr.show_versions() here -->\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.10.0 | packaged by conda-forge | (default, Nov 20 2021, 02:25:18) [GCC 9.4.0]\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 5.4.0-1025-aws\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: en_US.UTF-8\r\nLANG: en_US.UTF-8\r\nLOCALE: ('en_US', 'UTF-8')\r\nlibhdf5: None\r\nlibnetcdf: None\r\n\r\nxarray: 0.20.2\r\npandas: 1.3.5\r\nnumpy: 1.21.4\r\nscipy: None\r\nnetCDF4: None\r\npydap: None\r\nh5netcdf: None\r\nh5py: None\r\nNio: None\r\nzarr: 2.10.3\r\ncftime: None\r\nnc_time_axis: None\r\nPseudoNetCDF: None\r\nrasterio: None\r\ncfgrib: None\r\niris: None\r\nbottleneck: 1.3.2\r\ndask: 2021.12.0\r\ndistributed: 2021.12.0\r\nmatplotlib: None\r\ncartopy: None\r\nseaborn: None\r\nnumbagg: None\r\nfsspec: 2021.11.1\r\ncupy: None\r\npint: None\r\nsparse: None\r\nsetuptools: 59.4.0\r\npip: 21.3.1\r\nconda: None\r\npytest: 6.2.5\r\nIPython: 7.30.1\r\nsphinx: None\r\n\r\n</details>\r\n", "{\"url\": \"https://api.github.com/repos/pydata/xarray/issues/6112/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", null, "completed", 13221727, "issue"]], "truncated": false, "filtered_table_rows_count": 4, "expanded_columns": [], "expandable_columns": [[{"column": "repo", "other_table": "repos", "other_column": "id"}, "name"], [{"column": "milestone", "other_table": "milestones", "other_column": "id"}, "title"], [{"column": "assignee", "other_table": "users", "other_column": "id"}, "login"], [{"column": "user", "other_table": "users", "other_column": "id"}, "login"]], "columns": ["id", "node_id", "number", "title", "user", "state", "locked", "assignee", "milestone", "comments", "created_at", "updated_at", "closed_at", "author_association", "active_lock_reason", "draft", "pull_request", "body", "reactions", "performed_via_github_app", "state_reason", "repo", "type"], "primary_keys": ["id"], "units": {}, "query": {"sql": "select id, node_id, number, title, user, state, locked, assignee, milestone, comments, created_at, updated_at, closed_at, author_association, active_lock_reason, draft, pull_request, body, reactions, performed_via_github_app, state_reason, repo, type from issues where \"repo\" = :p0 and \"state\" = :p1 and \"user\" = :p2 order by updated_at desc limit 101", "params": {"p0": "13221727", "p1": "closed", "p2": "25071375"}}, "facet_results": {"state": {"name": "state", "type": "column", "hideable": false, "toggle_url": "/github/issues.json?repo=13221727&state=closed&user=25071375", "results": [{"value": "closed", "label": "closed", "count": 4, "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&user=25071375", "selected": true}], "truncated": false}, "repo": {"name": "repo", "type": "column", "hideable": false, "toggle_url": "/github/issues.json?repo=13221727&state=closed&user=25071375", "results": [{"value": 13221727, "label": "xarray", "count": 4, "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?state=closed&user=25071375", "selected": true}], "truncated": false}, "type": {"name": "type", "type": "column", "hideable": false, "toggle_url": "/github/issues.json?repo=13221727&state=closed&user=25071375", "results": [{"value": "issue", "label": "issue", "count": 3, "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&type=issue", "selected": false}, {"value": "pull", "label": "pull", "count": 1, "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&type=pull", "selected": false}], "truncated": false}}, "suggested_facets": [{"name": "closed_at", "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&_facet=closed_at"}, {"name": "created_at", "type": "date", "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&_facet_date=created_at"}, {"name": "updated_at", "type": "date", "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&_facet_date=updated_at"}, {"name": "closed_at", "type": "date", "toggle_url": "http://xarray-datasette.fly.dev/github/issues.json?repo=13221727&state=closed&user=25071375&_facet_date=closed_at"}], "next": null, "next_url": null, "private": false, "allow_execute_sql": true, "query_ms": 3676.557041995693}