issues: 1030811490
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1030811490 | I_kwDOAMm_X849cO9i | 5878 | problem appending to zarr on GCS when using json token | 48723181 | open | 0 | 16 | 2021-10-19T23:01:57Z | 2022-11-15T22:04:21Z | NONE | What happened: Appending a toy dataset to an existing zarr store in GCS along the time dimension leaves the store unchanged. What you expected to happen: The store to double in length, because I was appending a dataset with a length of 3 in the time dimension, to another dataset of the same size. Minimal Complete Verifiable Example: To reproduce fully you will need the token, but maybe people can try using their own token. ```python import fsspec import xarray as xr import json import gcsfs define a mapper to the ldeo-glaciology bucketneeds a tokenwith open('../secrets/ldeo-glaciology-bc97b12df06b.json') as token_file: token = json.load(token_file) mapper = fsspec.get_mapper('gs://ldeo-glaciology/append_test/test5', mode='w', token=token) define two simple datasetsds0 = xr.Dataset({'temperature': (['time'], [50, 51, 52])}, coords={'time': [1, 2, 3]}) ds1 = xr.Dataset({'temperature': (['time'], [53, 54, 55])}, coords={'time': [4, 5, 6]}) write the fist to bucketds0.to_zarr(mapper) append the second to the same zarr storeds1.to_zarr(mapper, mode='a', append_dim='time') load the zarr storeds_both = xr.open_zarr(mapper) this is 3, indicating that the append did not worklen(ds_both.time) ``` Anything else we need to know?: It works as expected if you instead write and append to the pangeo scratch bucket, i.e if you replace ``` with open('../secrets/ldeo-glaciology-bc97b12df06b.json') as token_file: token = json.load(token_file) mapper = fsspec.get_mapper('gs://ldeo-glaciology/append_test/test5', mode='w', token=token)
It also works as expected if I write and append to a local zarr. Thanks for your help! Environment: https://us-central1-b.gcp.pangeo.io/ Output of <tt>xr.show_versions()</tt>INSTALLED VERSIONS ------------------ commit: None python: 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.4.129+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C.UTF-8 LANG: C.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.16.2 pandas: 1.2.1 numpy: 1.20.0 scipy: 1.6.0 netCDF4: 1.5.5.1 pydap: installed h5netcdf: 0.8.1 h5py: 3.1.0 Nio: None zarr: 2.6.1 cftime: 1.4.1 nc_time_axis: 1.2.0 PseudoNetCDF: None rasterio: 1.2.0 cfgrib: 0.9.8.5 iris: None bottleneck: 1.3.2 dask: 2021.01.1 distributed: 2021.01.1 matplotlib: 3.3.4 cartopy: 0.18.0 seaborn: None numbagg: None pint: 0.16.1 setuptools: 49.6.0.post20210108 pip: 20.3.4 conda: None pytest: None IPython: 7.20.0 sphinx: 3.4.3 |
{ "url": "https://api.github.com/repos/pydata/xarray/issues/5878/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
13221727 | issue |