issue_comments

Closing since this seems to be a zarr/ zarr consolidation issue and not an xarray issue.

Upon further testing it looks like the error has to do with the consolidated zarr store. Even thou the orginal set was written with:

python ds.to_zarr(store=gcsmap, mode='w-', consolidated=True)

The new append to zarr with:

python ds.to_zarr(store=gcsmap, mode='a', consolidated=True, compute=False, append_dim='time')

Fails since it reads an empty array. This might be more of a problem for zarr itself and not xarray. Going to test rewriting a netcdf file to zarr with consolidation and if the error persists then I will move to zarr for the error.

If that is the case then would this also break for s3fs and azures bucket fs package (never used azure not sure about this statement)? Since they all should return a mutable map that gets turned into a string.

Note that get_mapper is implemented for all file systems, so there should be no need for any gcsfs-specific code.

On August 26, 2019 11:21:00 AM EDT, Justin Minsk notifications@github.com wrote:

Went back to gcsfs 0.0.4 and the current code still does not work. Going back more requires major changes to environment that conda cannot handle and would not be compatible with the current build of xarray. I think changing mutable maps to strings in this case has never worked with gcsfs and more than likely has not been seriously used until the append to zarr was added.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/pydata/xarray/issues/3251#issuecomment-524903609

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Went back to gcsfs 0.0.4 and the current code still does not work. Going back more requires major changes to environment that conda cannot handle and would not be compatible with the current build of xarray. I think changing mutable maps to strings in this case has never worked with gcsfs and more than likely has not been seriously used until the append to zarr was added.

Thanks for looking into this Justin. We really appreciate your efforts.

Sent from my iPhone

On Aug 26, 2019, at 7:52 AM, Justin Minsk notifications@github.com wrote:

Even with the changes in gcsfs syntax the same error still occurs since the old syntax just uses get_mapper() under the hood. I can continue to work on finding which version of gcsfs works with the current xarray code but the pinning on gcsfs, xarray, and dask on conda/conda-forge ends up getting out of sync very quickly and will take me a bit more time then I thought it would.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Even with the changes in gcsfs syntax the same error still occurs since the old syntax just uses get_mapper() under the hood. I can continue to work on finding which version of gcsfs works with the current xarray code but the pinning on gcsfs, xarray, and dask on conda/conda-forge ends up getting out of sync very quickly and will take me a bit more time then I thought it would.

It appears that GCSFS introduced a breaking API change with no deprecation cycle. (See https://github.com/dask/gcsfs/issues/166).

The old syntax is python gcsmap = mapping.GCSMap(gcs_bucket_name) ds = xr.open_zarr(gcsmap)

The new syntax is python gcs = gcsfs.GCSFileSystem(...) map = gcs.get_mapper() ds = xr.open_zarr(map)

I will look through previous versions of GCSFS to see where the change would have occurred (I need to anyways to get my code working). I will update this comment when I find the version that wants the mutable map to be a string and not keep it as a mutable map.

I am not sure why str should ever be called on the mapping. For sure, what it returns is not the same as before (perhaps you could go back a version and check?), but I don't know what the string would have been used for. I am on leave at the moment and unlikely to be able to get time to investigate.

Yes my bad copying scratch code from an ipython terminal I missed the assignment. It should be

python gcsmap = mapping.GCSMap(gcs_bucket_name) I reedited above to correct that issue.

The version of gcsfs is 0.3.0

conda list out:

gcsfs 0.3.0 py_0 conda-forg

Hi @jminsk-cc - thanks for this bug report.

There have been quite a few internal changes to gcsfs recently. I'm going to tag @martindurant, who might be able to help. Could you report your gcsfs version?

Also, I can't reproduce your example because it never assigns the gcsmap variable. I assume you intended to do python gcsmap = mapping.GCSMap(gcs_bucket_name) ?