home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 2162954553

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
2162954553 I_kwDOAMm_X86A7BE5 8800 unload python object from memory 14925067 closed 0     5 2024-03-01T09:14:30Z 2024-03-11T14:38:16Z 2024-03-11T14:38:15Z NONE      

Is your feature request related to a problem?

The problem related to the feature I am looking for is the fact that xarray have a .compute() mechanism that allows to load in memory python object (such as numpy matrix) but I don't manage to "free" or "unload" a numpy matrix assigned in a xarray.Dataset() . I think this question has been partially opened in this ticket : https://github.com/pydata/xarray/issues/4243

Describe the solution you'd like

I would like that such snippet gives constant memory footprint: ```python import xarray as xr import numpy as np import psutil import os

n_iteration = 20 x = np.arange(6000) y = np.arange(3000) ds = xr.Dataset() def get_memory(): pid = os.getpid() python_process = psutil.Process(pid) memoryUse = ( python_process.memory_info()[0] / 2.0 ** 30 ) # memory use in GB...I think print("memory use:", memoryUse)

for n in range(n_iteration): matrix_numpy = np.random.rand(y.size,x.size) ds['var'+str(n)] = xr.DataArray(matrix_numpy,dims=['y','x'],coords={'x':x,'y':y}) # here I don't need this matrix anymore, I just want it to be stored in the xr.Dataset del matrix_numpy # I suppose that the memory should be free, but obviously not... get_memory() print(ds) ```

currently the memory footprint is growing at each iteration: python memory use: 0.23286819458007812 memory use: 0.3671150207519531 memory use: 0.501251220703125 memory use: 0.6353721618652344 memory use: 0.7694854736328125 memory use: 0.9036026000976562 memory use: 1.0377159118652344 memory use: 1.1718292236328125 memory use: 1.3059463500976562 memory use: 1.4401473999023438 memory use: 1.5741729736328125 memory use: 1.7082862854003906 memory use: 1.8424034118652344 memory use: 1.9765167236328125 memory use: 2.110637664794922 memory use: 2.244762420654297 memory use: 2.378875732421875 memory use: 2.51300048828125 memory use: 2.647113800048828 memory use: 2.7812347412109375 <xarray.Dataset> Dimensions: (x: 6000, y: 3000) Coordinates: * x (x) float64 -5.0 -4.998 -4.997 -4.995 ... 4.995 4.997 4.998 5.0 * y (y) float64 -5.0 -4.997 -4.993 -4.99 ... 4.99 4.993 4.997 5.0 Data variables: (12/20) var0 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var1 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var2 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var3 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var4 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var5 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 ... ... var14 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var15 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var16 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var17 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var18 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12 var19 (y, x) float64 2.21e-12 2.229e-12 2.247e-12 ... 2.229e-12 2.21e-12

Describe alternatives you've considered

No response

Additional context

context:

xarray==2023.11.0 ubuntu 22.04 execution in python (not ipython not notebook)

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/8800/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 1 row from issues_id in issues_labels
  • 0 rows from issue in issue_comments
Powered by Datasette · Queries took 1.197ms · About: xarray-datasette