issues: 293293632
This data as json
| id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 293293632 | MDU6SXNzdWUyOTMyOTM2MzI= | 1874 | running out of memory trying to write SQL | 1794116 | closed | 0 | 3 | 2018-01-31T20:05:39Z | 2019-02-04T04:29:09Z | 2019-02-04T04:29:08Z | NONE | Python version:3 xarray version: 0.9.6 I am using xarray to read very large NetCDF files (~47G). Then I need to write the data to a postgres DB. I have tried parsing the array and using an INSERT for every row, but this is taking a very long time (weeks). I have read that bulk insert would be a lot faster, so I am looking for a solution along those lines. I also saw that Pandas has a DataFrame.to_sql() function and xarray has Dataset.to_dataframe() function, so I was trying out this approach. However, when trying to convert my xarray Dataset to a Pandas Dataframe, I ran out of memory quickly. Is this expected behavior? If so can you suggest another solution to this problem? |
{
"url": "https://api.github.com/repos/pydata/xarray/issues/1874/reactions",
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
completed | 13221727 | issue |