issues

MCVE Code Sample

```python import multiprocessing import xarray as xr import numpy as np from s3fs import S3FileSystem, S3Map from time import sleep

from main import set_aws_credentials set_aws_credentials()

def download_file(lead_time): return 'path_to_your_file'

def make_xarray_dataset(file_path, lead_time): var1 = np.random.rand(1, 721, 1440, 22) var2 = np.random.rand(1, 721, 1440, 22) lat = np.linspace(-90, 90, 721) lon = np.linspace(0, 360, 1440) height = range(22) ds = xr.Dataset({'var1': (['lead_time', 'lat', 'lon', 'height'], var1), 'var2': (['lead_time', 'lat', 'lon', 'height'], var2)}, coords={'lat': lat, 'lon': lon, 'height': height, 'lead_time': [lead_time]}) return ds

def upload_to_s3(dataset, append): s3 = S3FileSystem() s3map = S3Map('S3_path_to_your_zarr', s3=s3) # If we are appending to an already existing dataset if append: dataset.to_zarr(store=s3map, mode='a', append_dim='lead_time') else: dataset.to_zarr(store=s3map, mode='w')

def lead_time_worker(lead_time, append=True): file_path = download_file(lead_time) dataset = make_xarray_dataset(file_path, lead_time) upload_to_s3(dataset, append=True) return 0

if name == 'main': lead_times = range(10) first_lead_time = True processes = [] for lead_time in lead_times: if first_lead_time: process = multiprocessing.Process(target=lead_time_worker, args=(lead_time, False)) process.start() process.join() first_lead_time = False else: process = multiprocessing.Process(target=lead_time_worker, args=(lead_time,)) process.start() processes.append(process) sleep(5) # Sleep in order to shift the different processes so that they don't begin at the same time for p in processes: p.join() ``` will raise

ValueError: conflicting sizes for dimension 'lead_time': length X on 'Var1' and length Y on 'Var2'

Problem Description

First of all, thanks a lot to the community for the PR #2706, I was really looking forward to it. I already experienced using the new append parameter, and got some problems trying to do it in a parallel way.

I want to upload a very big zarr (global numerical weather prediction, output of GFS model that you can check out here) on a S3 bucket. In order to speed this up, as each single file of the source contains the data for one lead time(length of time between the issuance of a forecast and the occurrence of the phenomena that were predicted) and I want to concatenate them all, I tried to have one process per lead time and all of them to append to the same data store using Dataset.to_zarr() with append=True.

However, when doing that, I get the error described above. Indeed, the processes are appending simultaneously, so the data is not necessarily consistent when a new process tries to append, some variables will already have the values of one lead time and some will not because the process is not finished, which will lead to calculate_dimensions() raising this error.

I wonder if there is a way I haven't found to work around this using simply a synchronizer? If not, do you think it would be possible (and reasonable) to implement a parameter allowing to bypass this check on the append dimension in an 'eventually consistent' approach?

Output of xr.show_versions()

MCVE Code Sample

```python

import xarray as xr import numpy as np from s3fs import S3FileSystem, S3Map

s3 = S3FileSystem() bucket_name = 'your-bucket-name' s3_path = bucket_name + 'some_path.zarr' store = S3Map(s3_path, s3=s3) for i in range(6): if i%2 == 0: group = 'Group1' else: group = 'Group2' lead_time = i//2 var1 = np.random.rand(1) ds = xr.Dataset({'var1': (['lead_time'], var1)}, coords={'lead_time': [lead_time]}) ds.to_zarr(store=store, mode='a', append_dim='lead_time', group=group)

```

Output

This code returns the following error:

The KeyError can happen on a variable name as well as on a dimension name, it depends on the runs.

Problem Description

I am trying to use the append mode introduced in the PR #2706 on zarr groups. This raises a KeyError as you can see in the trace above. Is it a bug or a feature that is not supported (yet)?

Output of xr.show_versions()