html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/6069#issuecomment-1059400265,https://api.github.com/repos/pydata/xarray/issues/6069,1059400265,IC_kwDOAMm_X84_JSpJ,9576982,2022-03-04T18:09:44Z,2022-03-04T18:10:49Z,NONE,"@d70-t we can try to branch it to the CF related issue yes.
The `del` method is the one I tried and when doing it on my files I had very weird things happening so I would not recommend it as a proper workaround. as I wrote before it was not appending the file as it should have.
I have now a run functioning with the `region` method but I had to simulate my whole file which was a bit challenging and is actually pretty easy to break as I need to use the geometry of a single variable to generate my temporal and spatial coordinates for the whole archive. Going through the whole variables is a bit of a no-go.
The initialisation with both methods is really a challenge I find.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1059274384,https://api.github.com/repos/pydata/xarray/issues/6069,1059274384,IC_kwDOAMm_X84_Iz6Q,9576982,2022-03-04T15:42:36Z,2022-03-04T15:42:36Z,NONE,"I have tried to specify the chunk before writing the dataset and I have had some really strange behaviour with data written into the same chunks, the time dimension never went over 5, growing and reducing through the processing...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1059121536,https://api.github.com/repos/pydata/xarray/issues/6069,1059121536,IC_kwDOAMm_X84_IOmA,9576982,2022-03-04T12:30:01Z,2022-03-04T12:30:01Z,NONE,"Effectively I have unstable results with sometimes errors with timesteps refusing to write
I systematically have this warning
``` python
/opt/conda/lib/python3.7/site-packages/xarray/core/dataset.py:2050: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs
safe_chunks=safe_chunks,
```
the crashes are related to dimension of time itself but time is always of size 1, so it is hard to understand
``` python
/tmp/ipykernel_1629/1269180709.py in aggregate_with_time(farm_name, resolution_M, canvas, W, H, master_raster_coordinates)
39 raster.drop(
40 ['x','y']).to_zarr(
---> 41 uri, mode='a', append_dim='time')
42 #except:
43 #print('something went wrong')
/opt/conda/lib/python3.7/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
2048 append_dim=append_dim,
2049 region=region,
-> 2050 safe_chunks=safe_chunks,
2051 )
2052
/opt/conda/lib/python3.7/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
1406 _validate_datatypes_for_zarr_append(dataset)
1407 if append_dim is not None:
-> 1408 existing_dims = zstore.get_dimensions()
1409 if append_dim not in existing_dims:
1410 raise ValueError(
/opt/conda/lib/python3.7/site-packages/xarray/backends/zarr.py in get_dimensions(self)
450 if d in dimensions and dimensions[d] != s:
451 raise ValueError(
--> 452 f""found conflicting lengths for dimension {d} ""
453 f""({s} != {dimensions[d]})""
454 )
ValueError: found conflicting lengths for dimension time (2 != 1)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1059078276,https://api.github.com/repos/pydata/xarray/issues/6069,1059078276,IC_kwDOAMm_X84_IECE,9576982,2022-03-04T11:26:04Z,2022-03-04T11:26:04Z,NONE,"In my case I specify _fillvalue in the reprojection so I would not think this is an issue to overwrite it.
I just don't know how to do it","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1059052257,https://api.github.com/repos/pydata/xarray/issues/6069,1059052257,IC_kwDOAMm_X84_H9rh,9576982,2022-03-04T10:50:09Z,2022-03-04T10:50:09Z,NONE,"OK that's not exactly the same error message, I could not even start the appending. But that's basically one example that could be tested. A model would want to compute each of these variables step by step and variable by variable and save them for each single iteration. There is no need of concurrent writing as most of the resources are focused on the modelling.
``` python
import xarray as xr
from rasterio.enums import Resampling
import numpy as np
ds = xr.tutorial.open_dataset('air_temperature').isel(time=0)
ds = ds.rio.write_crs('EPSG:4326')
dst = ds.rio.reproject('EPSG:3857', shape=(250, 250), resampling=Resampling.bilinear, nodata=np.nan)
dst.to_zarr('test.zarr')
```
Returns
> ---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> /opt/conda/lib/python3.7/site-packages/zarr/util.py in normalize_fill_value(fill_value, dtype)
> 277 else:
> --> 278 fill_value = np.array(fill_value, dtype=dtype)[()]
> 279
>
> ValueError: cannot convert float NaN to integer
>
> During handling of the above exception, another exception occurred:
>
> ValueError Traceback (most recent call last)
> /tmp/ipykernel_2604/3259577033.py in
> ----> 1 dst.to_zarr('test.zarr')
>
> /opt/conda/lib/python3.7/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
> 2048 append_dim=append_dim,
> 2049 region=region,
> -> 2050 safe_chunks=safe_chunks,
> 2051 )
> 2052
>
> /opt/conda/lib/python3.7/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
> 1429 writer = ArrayWriter()
> 1430 # TODO: figure out how to properly handle unlimited_dims
> -> 1431 dump_to_store(dataset, zstore, writer, encoding=encoding)
> 1432 writes = writer.sync(compute=compute)
> 1433
>
> /opt/conda/lib/python3.7/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
> 1117 variables, attrs = encoder(variables, attrs)
> 1118
> -> 1119 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
> 1120
> 1121
>
> /opt/conda/lib/python3.7/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
> 549
> 550 self.set_variables(
> --> 551 variables_encoded, check_encoding_set, writer, unlimited_dims=unlimited_dims
> 552 )
> 553 if self._consolidate_on_close:
>
> /opt/conda/lib/python3.7/site-packages/xarray/backends/zarr.py in set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
> 607 dtype = str
> 608 zarr_array = self.zarr_group.create(
> --> 609 name, shape=shape, dtype=dtype, fill_value=fill_value, **encoding
> 610 )
> 611 zarr_array.attrs.put(encoded_attrs)
>
> /opt/conda/lib/python3.7/site-packages/zarr/hierarchy.py in create(self, name, **kwargs)
> 889 """"""Create an array. Keyword arguments as per
> 890 :func:`zarr.creation.create`.""""""
> --> 891 return self._write_op(self._create_nosync, name, **kwargs)
> 892
> 893 def _create_nosync(self, name, **kwargs):
>
> /opt/conda/lib/python3.7/site-packages/zarr/hierarchy.py in _write_op(self, f, *args, **kwargs)
> 659
> 660 with lock:
> --> 661 return f(*args, **kwargs)
> 662
> 663 def create_group(self, name, overwrite=False):
>
> /opt/conda/lib/python3.7/site-packages/zarr/hierarchy.py in _create_nosync(self, name, **kwargs)
> 896 kwargs.setdefault('cache_attrs', self.attrs.cache)
> 897 return create(store=self._store, path=path, chunk_store=self._chunk_store,
> --> 898 **kwargs)
> 899
> 900 def empty(self, name, **kwargs):
>
> /opt/conda/lib/python3.7/site-packages/zarr/creation.py in create(shape, chunks, dtype, compressor, fill_value, order, store, synchronizer, overwrite, path, chunk_store, filters, cache_metadata, cache_attrs, read_only, object_codec, dimension_separator, **kwargs)
> 139 fill_value=fill_value, order=order, overwrite=overwrite, path=path,
> 140 chunk_store=chunk_store, filters=filters, object_codec=object_codec,
> --> 141 dimension_separator=dimension_separator)
> 142
> 143 # instantiate array
>
> /opt/conda/lib/python3.7/site-packages/zarr/storage.py in init_array(store, shape, chunks, dtype, compressor, fill_value, order, overwrite, path, chunk_store, filters, object_codec, dimension_separator)
> 356 chunk_store=chunk_store, filters=filters,
> 357 object_codec=object_codec,
> --> 358 dimension_separator=dimension_separator)
> 359
> 360
>
> /opt/conda/lib/python3.7/site-packages/zarr/storage.py in _init_array_metadata(store, shape, chunks, dtype, compressor, fill_value, order, overwrite, path, chunk_store, filters, object_codec, dimension_separator)
> 392 chunks = normalize_chunks(chunks, shape, dtype.itemsize)
> 393 order = normalize_order(order)
> --> 394 fill_value = normalize_fill_value(fill_value, dtype)
> 395
> 396 # optional array metadata
>
> /opt/conda/lib/python3.7/site-packages/zarr/util.py in normalize_fill_value(fill_value, dtype)
> 281 # re-raise with our own error message to be helpful
> 282 raise ValueError('fill_value {!r} is not valid for dtype {}; nested '
> --> 283 'exception: {}'.format(fill_value, dtype, e))
> 284
> 285 return fill_value
>
> ValueError: fill_value nan is not valid for dtype int16; nested exception: cannot convert float NaN to integer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1059022639,https://api.github.com/repos/pydata/xarray/issues/6069,1059022639,IC_kwDOAMm_X84_H2cv,9576982,2022-03-04T10:10:08Z,2022-03-04T10:10:08Z,NONE,"The _FillValue is always the same (np.nan) and specified when I reproject with rioxarray.
so I don't understand the first error then.
The thing is that the _fillvalue is attached to a variable not the whole dataset. But it never change. Not too sure what to do","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1058323632,https://api.github.com/repos/pydata/xarray/issues/6069,1058323632,IC_kwDOAMm_X84_FLyw,9576982,2022-03-03T17:54:27Z,2022-03-03T17:54:27Z,NONE,"I did make
ds.attrs={}
but at each appending I get a warning
```
/opt/conda/lib/python3.7/site-packages/xarray/core/dataset.py:2050: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs
safe_chunks=safe_chunks,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1058315108,https://api.github.com/repos/pydata/xarray/issues/6069,1058315108,IC_kwDOAMm_X84_FJtk,9576982,2022-03-03T17:45:15Z,2022-03-03T17:45:15Z,NONE,"I have looked at these examples and I still don't manage to make it work in the real world.
I find append the most logical but I have attributes attached to a dataset that I don't seem to be able to drop before appending. This generates this error:
`ValueError: failed to prevent overwriting existing key _FillValue in attrs. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.`
However, I cannot find a way of getting rid of this attribute","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1034678675,https://api.github.com/repos/pydata/xarray/issues/6069,1034678675,IC_kwDOAMm_X849q_GT,9576982,2022-02-10T09:18:47Z,2022-02-10T09:18:47Z,NONE,"If Xarray/zarr is to replace netcdf, appending by time step is really an important feature
Most (all?) numerical models will output results per time step onto a multidimensional grid with different variables
Said grid will also have other parameters that will help rebuild the geometry or follow standards, like CF and Ugrid (The things that you are supposed to drop). The geometry of the grid is computed at the initialisation of the model. It is a bit counter intuitive to get rid of it for incremental backups especially that each write will not concern this part of the file.
What I do at the moment is that I create a first dataset at the final dimension based on dummy dask arrays
Export it `to_zarr` with` compute = False`
With a buffer system, I create a new dataset for **each** buffer with the right data at the right place meaning only the time interval concerned and I write `to_zarr` with the region attribute
I flush the buffer dataset after it being written.
At the end I write all the parameters before closing the main dataset.
To my knowledge, that's the only method which works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208
https://github.com/pydata/xarray/issues/6069#issuecomment-1032480933,https://api.github.com/repos/pydata/xarray/issues/6069,1032480933,IC_kwDOAMm_X849imil,9576982,2022-02-08T11:01:21Z,2022-02-08T11:01:21Z,NONE,"I don't get the second crash. It is not true that these variables are not in common, they are the coordinates of each of the variables. They are all made the same. This is a typical example of an unstructured grid backup. Meanwhile I found an alternate solution which is also better for memory management. I think the documentation example doesn't actually work.
I will try to formulate my trick but that's not using this particular method of region that is not functioning as it should in my opinion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1077079208