issue_comments

The methods that need to be implemented are (in the C API) as follows:

int nc_def_var_filter(int ncid, int varid, unsigned int id, size_t nparams, const unsigned int* parms); The only tricky part is passing a vector of unsigned integers (the parms argument).

int nc_inq_var_filter(int ncid, int varid, unsigned int idp, size_t nparams, unsigned int* params); This requires passing values out via pointers. Also, this uses the standard netcdf-c trick in which the function is called twice. First with nparams defined, but params has the value NULL. This gives the caller the number of parameters. The function is called a second time after allocating the params vector and with that vector as the final argument.

The API is not yet exposed thru anything but the C api. So the python, fortran, and c++ wrappers do not yet show it. Passing it thru netcdf-python is probably pretty trivian, though.

A bit confusing, but I think the answer is yes. For example we provide a bzip2 compression plugin as an example (see examples/C/hdf5plugins in the netcdf-c distribution).

You may already know, but should note that the filter stuff in netcdf-c is now available in netcdf-c library version 4.6.0. So any filter plugin usable with hdf5 can now be used both for reading and writing thru the netcdf-c api.

The github branch filters.dmh for the netcdf-c library now exposes the HDF5 dynamic filter capability. This is documented here: https://github.com/Unidata/netcdf-c/blob/filters.dmh/docs/filters.md I welcome suggestions for improvements.

I also note that I am extending this branch to now handle szip compression. It turns out there is now a patent-free implementation called libaec (HT Rich Signell) so there is no reason not to make it available.

For what it is worth, we have a branch at netCDF-C that allows for different compression plugins using the hdf5 plugin architecture. It will not be in the 4.5.0 release, but once 4.5.0 is finished we will be looking at it for either the subsequent release. There is still testing and documentation to be completed, but @DennisHeimbigner (who implemented it) can speak more about it.

how about zstandard?

On Aug 28, 2017, at 7:22 PM, Joe Hamman notifications@github.com wrote:

This is an interesting idea. I think something similar was discussed in #66 https://github.com/pydata/xarray/issues/66.

The main problem I see is that current netCDF libraries don't support LZF so I think we're really talking about adding an HDF5 backend. In practice, this could use the new API of h5netcdf or h5py directly. Either way, my initial thought is that we want a to_hdf5() method on the dataset.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1536#issuecomment-325510075, or mute the thread https://github.com/notifications/unsubscribe-auth/Ac5fKqKijwFO916RY0ARUZq9O25QBm9Jks5sc0ungaJpZM4PFJ5x.