issues: 253476466

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
253476466 MDU6SXNzdWUyNTM0NzY0NjY= 1536 Better compression algorithms for NetCDF 6213168 closed 0     28 2017-08-28T22:35:31Z 2018-05-08T02:25:40Z 2018-05-08T02:25:40Z MEMBER      

As of today, Dataset.to_netcdf() exclusively allows writing uncompressed or compressed with zlib. zlib was absolutely revolutionary when it was released... in 1995. Time has passed, and much better compression algorithms have appeared over time. Good news is, h5py supports LZF out of the box, and is extensible with plugins to support theoretically any other algorithm. h5netcdf exposes such interface through its new (non-legacy) API; however Dataset.to_netcdf(engine='h5netcdf') supports the legacy API exclusively.

I already tested that, once you manage to write to disk with LZF (using h5netcdf directly), open_dataset(engine='h5netcdf') transparently opens the compressed store.

Options: - write a new engine for Dataset.to_netcdf() to support the new h5netcdf API. - switch the whole engine='h5netcdf' to the new API. Drop support for the old parameters in to_netcdf(). This is less bad than it sounds, as people can switch to another engine in case of trouble. This is the cleanest solution, but also the most disruptive one. - switch the whole engine='h5netcdf' to the new API; have to_netcdf() accept both new and legacy parameters, and implement a translation layer of parameters from the legacy API to the new API. The benefit here is that, as long as the user sticks to the legacy API, he can hop between engines transparently. On the other hand I have a hard time believing anybody would care. - ?

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/1536/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables