home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 314326128

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
314326128 MDU6SXNzdWUzMTQzMjYxMjg= 2057 Problem reading dtype=S64 with open_zarr 1872600 closed 0     1 2018-04-14T12:42:52Z 2018-04-30T17:17:27Z 2018-04-30T17:17:27Z NONE      

@jhamman suggested I raise this SO question as an issue here.

I have a dataset that looks like: <xarray.Dataset> Dimensions: (nv: 2, reference_time: 746, time: 746, x: 4608, y: 3840) Coordinates: * reference_time (reference_time) datetime64[ns] 2018-03-07 ... * x (x) float64 -2.304e+06 -2.303e+06 -2.302e+06 ... * y (y) float64 -1.92e+06 -1.919e+06 -1.918e+06 ... * time (time) datetime64[ns] 2018-03-07T01:00:00 ... Dimensions without coordinates: nv Data variables: time_bounds (time, nv) datetime64[ns] dask.array<shape=(746, 2), chunksize=(1, 2)> ProjectionCoordinateSystem (time) |S64 b'' b'' b'' b'' b'' b'' b'' b'' ... T2D (time, y, x) float64 dask.array<shape=(746, 3840, 4608), chunksize=(1, 3840, 4608)>

When writing this dataset using ds.to_zarr containing the ProjectionCoordinateSystem variable with dtype=S64 , there can be an issue reading it using ds.open_zarr with the default auto_chunk=True.

This example illustrates the problem: ```python import xarray as xr import s3fs

f_zarr = 'rsignell/nwm/test02'
fs = s3fs.S3FileSystem(anon=False) d = s3fs.S3Map(f_zarr, s3=fs) xr.open_zarr(d) returningpython-traceback


ValueError Traceback (most recent call last) <ipython-input-3-fff1cd753e7c> in <module>() 2 fs = s3fs.S3FileSystem(anon=False) 3 d = s3fs.S3Map(f_zarr, s3=fs) ----> 4 xr.open_zarr(d)

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, auto_chunk, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables) 476 477 variables = OrderedDict([(k, maybe_chunk(k, v)) --> 478 for k, v in ds.variables.items()]) 479 return ds._replace_vars_and_dims(variables) 480 else:

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in <listcomp>(.0) 476 477 variables = OrderedDict([(k, maybe_chunk(k, v)) --> 478 for k, v in ds.variables.items()]) 479 return ds._replace_vars_and_dims(variables) 480 else:

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in maybe_chunk(name, var) 471 token2 = tokenize(name, var._data) 472 name2 = 'zarr-%s' % token2 --> 473 return var.chunk(chunks, name=name2, lock=None) 474 else: 475 return var

/opt/conda/lib/python3.6/site-packages/xarray/core/variable.py in chunk(self, chunks, name, lock) 820 data = indexing.ImplicitToExplicitIndexingAdapter( 821 data, indexing.OuterIndexer) --> 822 data = da.from_array(data, chunks, name=name, lock=lock) 823 824 return type(self)(self.dims, data, self._attrs, self._encoding,

/opt/conda/lib/python3.6/site-packages/dask/array/core.py in from_array(x, chunks, name, lock, asarray, fancy, getitem) 1988 >>> a = da.from_array(x, chunks=(1000, 1000), lock=True) # doctest: +SKIP 1989 """ -> 1990 chunks = normalize_chunks(chunks, x.shape) 1991 if name in (None, True): 1992 token = tokenize(x, chunks)

/opt/conda/lib/python3.6/site-packages/dask/array/core.py in normalize_chunks(chunks, shape) 1918 raise ValueError( 1919 "Chunks and shape must be of the same length/dimension. " -> 1920 "Got chunks=%s, shape=%s" % (chunks, shape)) 1921 1922 if shape is not None:

ValueError: Chunks and shape must be of the same length/dimension. Got chunks=(3, 64), shape=(3,) ``` The full notebook is at https://gist.github.com/rsignell-usgs/dce09aae4f7cd174a141247a56ddea2c

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2057/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 2 rows from issues_id in issues_labels
  • 1 row from issue in issue_comments
Powered by Datasette · Queries took 161.195ms · About: xarray-datasette