home / github / issues

Menu
  • GraphQL API
  • Search all tables

issues: 1685803922

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1685803922 I_kwDOAMm_X85ke1OS 7790 Fill values in time arrays (numpy.datetime64) are lost in zarr 14983768 closed 0     25 2023-04-26T22:03:06Z 2023-09-17T08:15:28Z 2023-09-17T08:15:28Z NONE      

What happened?

I have a time array of type numpy.datetime64 with fill values. When I save the dateset I created with xarray to zarr and then read that zarr store back out again with xarray, the fill values are lost.

What did you expect to happen?

I expected my fill values to still be in my time array when I read it back out of the zarr store.

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr import zarr

Create a numpy array of type np.datetime64 with one fill value and one date

time_fill_value = np.datetime64("NaT") time = np.array([time_fill_value,'2023-01-02'],dtype='M8[ns]')

Create a dataset with this one array

xr_time_array = xr.DataArray(data=time,dims=['time'],name='time') xr_ds = xr.Dataset(dict(time=xr_time_array))

print("******") print("Created with fill value (NaT)") print(xr_ds["time"])

Save the dataset to zarr

location = "from_xarray.zarr" encoding = { "time":{"_FillValue":time_fill_value} } xr_ds.to_zarr(location,encoding=encoding)

xr_read = xr.open_zarr(location) print("******") print("No fill value") print(xr_read["time"]) ```

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

When I look in the .zmetadata file generated for this zarr store, I see that the time array as been converted to float and there is a units attribute:

json { "metadata": { ".zattrs": {}, ".zgroup": { "zarr_format": 2 }, "time/.zarray": { "chunks": [ 2 ], "compressor": { "blocksize": 0, "clevel": 5, "cname": "lz4", "id": "blosc", "shuffle": 1 }, "dtype": "<f8", "fill_value": -9.223372036854776e+18, "filters": null, "order": "C", "shape": [ 2 ], "zarr_format": 2 }, "time/.zattrs": { "_ARRAY_DIMENSIONS": [ "time" ], "calendar": "proleptic_gregorian", "units": "days since 2023-01-02 00:00:00" } }, "zarr_consolidated_format": 1 }

If I open the data using zarr, the actual values in the time array look fine, given the transformation: ```python z = zarr.open(location,'r')

print(f'Fill value: {z["time"].fill_value}') print(f'Data values: {z["time"][:]}') Output: Fill value: -9.223372036854776e+18 Data values: [-9.22337204e+18 0.00000000e+00] ```

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.11.3 | packaged by conda-forge | (main, Apr 6 2023, 08:58:31) [Clang 14.0.6 ] python-bits: 64 OS: Darwin OS-release: 22.4.0 machine: arm64 processor: arm byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: None libnetcdf: None xarray: 2023.4.2 pandas: 2.0.1 numpy: 1.24.3 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.14.2 cftime: None nc_time_axis: None PseudoNetCDF: None iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 67.7.2 pip: 23.1.2 conda: None pytest: None mypy: None IPython: 8.12.0 sphinx: None
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7790/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 2 rows from issues_id in issues_labels
  • 25 rows from issue in issue_comments
Powered by Datasette · Queries took 0.781ms · About: xarray-datasette