home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 843971807

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4156#issuecomment-843971807 https://api.github.com/repos/pydata/xarray/issues/4156 843971807 MDEyOklzc3VlQ29tbWVudDg0Mzk3MTgwNw== 5637662 2021-05-19T10:33:08Z 2021-05-19T10:33:08Z CONTRIBUTOR

I have hacked something that does support the reading and writing of sparse arrays to a netcdf file, however I didn't know how and where to put this within xarray.

``` def ds_to_netcdf(ds, fn): dsorg = ds ds = dsorg.copy() for v in ds: if hasattr(ds[v].data, "nnz") and ( hasattr(ds[v].data, "to_coo") or hasattr(ds[v].data, "linear_loc") ): coord = f"{v}_xarray_index" assert coord not in ds data = ds[v].data if hasattr(data, "to_coo"): data = data.to_coo() ds[coord] = coord, data.linear_loc() dims = ds[v].dims ds[coord].attrs["compress"] = " ".join(dims) at = ds[v].attrs ds[v] = coord, data.data ds[v].attrs = at ds[v].attrs["fill_value"] = str(data.fill_value) for d in dims: if d not in ds: ds[f"_len{d}"] = len(dsorg[d])

print(ds)
ds.to_netcdf(fn)

```

``` def xr_open_dataset(fn): ds = xr.open_dataset(fn)

def fromflat(shape, i):
    index = []
    for fac in shape[::-1]:
        index.append(i % fac)
        i //= fac
    return tuple(index[::-1])

for c in ds.coords:
    if "compress" in ds[c].attrs:
        vs = c.split("_")
        if len(vs) < 5:
            continue
        if vs[-1] != "" or vs[-2] != "index" or vs[-3] != "xarray":
            continue
        v = "_".join(vs[1:-3])
        at = ds[v].attrs
        dat = ds[v].data
        fill = ds[v].attrs.pop("_fill_value", None)
        if fill:
            knownfails = {"nan": np.nan, "False": False, "True": True}
            if fill in knownfails:
                fill = knownfails[fill]
            else:
                fill = np.fromstring(fill, dtype=dat.dtype)
        dims = ds[c].attrs["compress"].split()
        shape = []
        for d in dims:
            try:
                shape.append(len(ds[d]))
            except KeyError:
                shape.append(int(ds[f"_len_{d}"].data))
                ds = ds.drop_vars(f"_len_{d}")

        locs = fromflat(shape, ds[c].data)
        data = sparse.COO(locs, ds[v].data, shape, fill_value=fill)
        ds[v] = dims, data, ds[v].attrs, ds[v].encoding
print(ds)
return ds

```

Has there been any progress since last year?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  638947370
Powered by Datasette · Queries took 0.432ms · About: xarray-datasette