Related to #4035, just using 'region' might be the answer for me.

Within my system, I reprocess a subset of an already written dataset (written to zarr) that I would then like to write back to zarr, overwriting the stored data. It seems like the only way to do that currently is to load the zarr array in memory, replace the changed bit, and then write the full array back with mode='w'.

I have a hacky way of doing this (outside of to_zarr) that kind of aligns with how append_dim works with to_zarr. I specify the overwrite_dim, match the incoming data to the part of the written zarr array that is of name=overwrite_dim and that has the same values, and use numpy syntax to overwrite just that part of the array that matches the incoming data.

```

initialize zarr array

ds = xr.Dataset({'arr': (('time', 'data_dim'), np.ones((10,3)))}, coords={'time': np.arange(100,110,1), 'data_dim': np.arange(3)}) ans = ds.to_zarr(r'C:\collab\dasktest\data_dir\test', mode='w') ans.get_variables()

this would be the result of the first write

Frozen({'arr': <xarray.Variable (time: 10, data_dim: 3)> array([[1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.]])

what I'd like to use to_zarr to do, just overwrite part of the first write, leaving the rest intact

import zarr overwrite_dim = 'time' rg = zarr.open(r'C:\collab\dasktest\data_dir\test') overwrite_subset_ds = xr.Dataset({'arr': (('time', 'data_dim'), np.full((2,3), 2))}, coords={'time': np.array([103, 104]), 'data_dim': np.arange(3)}) overwrite_index = np.isin(rg[overwrite_dim], overwrite_subset_ds[overwrite_dim].values)

if overwrite_index.any(): for darray in overwrite_subset_ds: if overwrite_dim in overwrite_subset_ds[darray].dims: d_dims_loc = d_dims.index('time') msk = np.zeros_like(rg['arr'], dtype=bool) msk[overwrite_index] = True rg[darray].set_mask_selection(msk, overwrite_subset_ds[darray].stack({'zwrite':d_dims}))

rg[darray][:]

array([[1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [2., 2., 2.], [2., 2., 2.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.], [1., 1., 1.]]) ```

So the old zarr array remains except for the portion i wanted changed. It would be nice if I could just do:

overwrite_subset_ds.to_zarr(r'C:\collab\dasktest\data_dir\test', mode='w', overwrite_dim='time')

And it would just do it.

Would this be in the scope of to_zarr? It seems almost necessary if you are going to use xarray/zarr as a working system, with data changing as it is processed.

Or maybe this is already possible and I need to RTFM?

What happened:

High memory usage seen when sorting after loading from disk. Loading from disk took about 150MB, where after the sort I saw a usage of about 1.5 GB. I believe this is due to the reindexing that requires the data to be loaded into memory during sort. So I guess I am not surprised, but I wanted to submit this as a possible issue just to make sure that my reasoning is good. For my use case, I will have to abandon sortby and ensure data is sorted prior to writing to disk.

I am afraid my MVCE relies on data on disk that I have. If this is an actual issue that needs more looking into, I can provide an example that anyone can run. Otherwise I can close.

Minimal Complete Verifiable Example:

```python import xarray as xr from psutil import virtual_memory startmem = virtual_memory().used data = xr.open_zarr(r"D:\falkor\FK181005_processed\em302_105_10_06_2018\attitude.zarr", synchronizer=None, mask_and_scale=False, decode_coords=False, decode_times=False, decode_cf=False, concat_characters=False) afterload_mem = virtual_memory().used - startmem ans = data.sortby('time') aftersort_mem = virtual_memory().used - startmem print('Without sort: {}'.format(afterload_mem)) print('With sort: {}'.format(aftersort_mem))

Out: Without sort: 149241856 Out: With sort: 1657593856 ```

Environment: