id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 2162954553,I_kwDOAMm_X86A7BE5,8800,unload python object from memory,14925067,closed,0,,,5,2024-03-01T09:14:30Z,2024-03-11T14:38:16Z,2024-03-11T14:38:15Z,NONE,,,,"### Is your feature request related to a problem? The problem related to the feature I am looking for is the fact that `xarray` have a `.compute()` mechanism that allows to load in memory python object (such as numpy matrix) but I don't manage to ""free"" or ""unload"" a numpy matrix assigned in a `xarray.Dataset()` . I think this question has been partially opened in this ticket : https://github.com/pydata/xarray/issues/4243 ### Describe the solution you'd like I would like that such snippet gives constant memory footprint: ```python import xarray as xr import numpy as np import psutil import os n_iteration = 20 x = np.arange(6000) y = np.arange(3000) ds = xr.Dataset() def get_memory(): pid = os.getpid() python_process = psutil.Process(pid) memoryUse = ( python_process.memory_info()[0] / 2.0 ** 30 ) # memory use in GB...I think print(""memory use:"", memoryUse) for n in range(n_iteration): matrix_numpy = np.random.rand(y.size,x.size) ds['var'+str(n)] = xr.DataArray(matrix_numpy,dims=['y','x'],coords={'x':x,'y':y}) # here I don't need this matrix anymore, I just want it to be stored in the xr.Dataset del matrix_numpy # I suppose that the memory should be free, but obviously not... get_memory() print(ds) ``` currently the memory footprint is growing at each iteration: ```python memory use: 0.23286819458007812 memory use: 0.3671150207519531 memory use: 0.501251220703125 memory use: 0.6353721618652344 memory use: 0.7694854736328125 memory use: 0.9036026000976562 memory use: 1.0377159118652344 memory use: 1.1718292236328125 memory use: 1.3059463500976562 memory use: 1.4401473999023438 memory use: 1.5741729736328125 memory use: 1.7082862854003906 memory use: 1.8424034118652344 memory use: 1.9765167236328125 memory use: 2.110637664794922 memory use: 2.244762420654297 memory use: 2.378875732421875 memory use: 2.51300048828125 memory use: 2.647113800048828 memory use: 2.7812347412109375 Dimensions: (x: 6000, y: 3000) Coordinates: * x (x) float64 -5.0 -4.998 -4.997 -4.995 ... 4.995 4.997 4.998 5.0 * y (y) float64 -5.0 -4.997 -4.993 -4.99 ... 4.99 4.993 4.997 5.0 Data variables: (12/20) var0 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var1 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var2 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var3 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var4 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var5 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 ... ... var14 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var15 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var16 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var17 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var18 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var19 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 ``` ### Describe alternatives you've considered _No response_ ### Additional context # context: xarray==2023.11.0 ubuntu 22.04 execution in python (not ipython not notebook) ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8800/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 925533850,MDExOlB1bGxSZXF1ZXN0NjczOTg0NjQz,5501,fix dtype complex for rasterio backend,14925067,closed,0,,,9,2021-06-20T06:49:54Z,2023-03-26T20:04:33Z,2023-03-26T20:04:32Z,NONE,,0,pydata/xarray/pulls/5501,"rasterio backend support for the `complex_int16` dtype - [x] Closes #5491 - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5501/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 924722006,MDU6SXNzdWU5MjQ3MjIwMDY=,5491,rasterio backend not supporting complex_int16 type,14925067,closed,0,,,3,2021-06-18T09:29:27Z,2023-02-28T21:48:22Z,2023-02-28T21:48:21Z,NONE,,,,"Dear xarray team, problem : `rasterio` xarray's backend doesn't support `complex_int16` dtype. cmd : `xarray.open_rasterio(tiff)` error : ```python ----> 4 xarray.open_rasterio(tiff) /home1/datawork/agrouaze/conda_envs2/envs/xsar_gdal3.3_v2/lib/python3.8/site-packages/xarray/backends/rasterio_.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock) 371 attrs[k] = v 372 --> 373 data = indexing.LazilyIndexedArray(RasterioArrayWrapper(manager, lock, vrt_params)) 374 375 # this lets you write arrays loaded with rasterio /home1/datawork/agrouaze/conda_envs2/envs/xsar_gdal3.3_v2/lib/python3.8/site-packages/xarray/backends/rasterio_.py in __init__(self, manager, lock, vrt_params) 40 if not np.all(np.asarray(dtypes) == dtypes[0]): 41 raise ValueError(""All bands should have the same dtype"") ---> 42 self._dtype = np.dtype(dtypes[0]) 43 44 @property TypeError: data type 'complex_int16' not understood ``` versions : rasterio == 1.2.4 gdal == 3.3.0 numpy == 1.20.4 xarray == 0.18.2 system : Linux Ubuntu 16.04.7 Python 3.8.10 possible patch in `xarray/backends/rasterio_.py` line 42: ```python if dtypes[0]!='complex_int16':#patch agrouaze June 2021 self._dtype = np.dtype(dtypes[0]) else: self._dtype = np.complex ``` Thanks for your help! ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5491/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue