issues: 506205450
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
506205450 | MDU6SXNzdWU1MDYyMDU0NTA= | 3394 | Update a small slice of a large netcdf file without overwriting the entire file. | 15239248 | open | 0 | 1 | 2019-10-12T16:06:18Z | 2021-07-04T03:32:00Z | NONE | MCVE Code Sample```python Your code hereorig = '/tmp/orig.h5' ii = 100000 data = xr.Dataset( { 'x':('t',np.random.randn(ii)), 'y':('t',np.random.randn(ii)) }, coords={'t':range(ii)} ) function to save the large file usnig chunksizesdef save(ds,path,kwargs): dvars = ds.variables chunksize = 100 var_dic = {} for var in dvars: var_dic[var]={ 'chunksizes': (chunksize,) } delayed =ds.to_netcdf(path,encoding=var_dic,kwargs) save(data,orig) data.close() open the file, using daskdata_1 = xr.open_mfdataset([orig],chunks={'t':100}) Change variable xdata_1['x']=data_1['x']+20 data_1.close() update only variable x. This works!data_1['x'].to_netcdf(orig,mode='a') try the same but now update only a slice of the x variableopen the file, using daskdata_1 = xr.open_mfdataset(orig,chunks={'t':100}) Change variable xdata_1['x']=data_1['x']+20 data_1.close() update only variable x. this doesnt work!data_1['x'][{'t':slice(0,10)}].to_netcdf(orig,mode='a') ``` Expected OutputProblem DescriptionHi, I have a large dataset that does not fit in memory. Lets say i only want to update a small portion of it. Is there any way to update this small portion without having to rewrite the entire file. I was fiddling around and found a way to update one variable at a time, but i want to be able to update only a subsection of this variable Output of
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/3394/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
13221727 | issue |