issue_comments

Well done, Kai!

This issue is fixed to some extent since h5netcdf 0.12.0.

h5netcdf does not reach the timings of netCDF4 engine, but the improvement is quite significant.

| Number of datasets in file | netCDF4 write (ms) | h5netcdf <= 0.11.0 write(ms) | h5netcdf >= 0.12.0 write (ms) | |-----|------|-----|-----| | 1 | 2 | 7 | 7 | | 250 | 104 | 1710 | 164 |

The issue can be closed.

Ping @aldanor.

FYI: h5netcdf has just merged a refactor of the dimension scale handling, which greatly improves the performance here. It will be released in the next version (0.13.0).

See https://github.com/h5netcdf/h5netcdf/pull/112

I'll come back if the release is out, so we can close this issue.

@aldanor Could you please have a look into https://github.com/h5netcdf/h5netcdf/pull/101 for a fix. Any comments are very much appreciated.

It appears that issues can only be moved within a GitHub organization. So I guess we'll need to start a new one.

On Thu, Mar 25, 2021 at 12:35 PM Kai Mühlbauer @.***> wrote:

@shoyer https://github.com/shoyer Could we move the entire issue? Or just open another one over at 'h5netcdf' and reference this one?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2857#issuecomment-807344131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJJFVRZOCDSW7GMLVHBYPDTFOF7DANCNFSM4HCRQYLA .

@shoyer Could we move the entire issue? Or just open another one over at 'h5netcdf' and reference this one?

I suspect this could be solved by adding an optimization into h5netcdf to only call _attach_dim_scales() (and maybe some other methods) on variables/groups that have been modified (as opposed to the entire file).

It's probably worth moving the discussion over into the h5netcdf tracker, anyways :)

OK, we might check if that depends on the data size or on the number of groups, or both.

I wonder if it would help to use the same underlying h5py.File or h5netcdf.File when appending.

This should somehow be possible. I'll try to create some proof of concept script bypassing to_netcdf, when I find the time. If there are other ideas or solutions, please comment here. Thanks @aldanor for intensive testing and minimal example.

From my understanding, part of the the problem is with the use of CachingFileManager. Every call to to_netcdf(filename....) reopens this particular file (with all the downsides) and wraps it in CachingFileManager again. I wonder if it would help to use the same underlying h5py.File or h5netcdf.File when appending.

@aldanor If I change your example to using engine=netcdf4, the times increase too, but not to the extend of the h5netcdf case.

@aldanor Thanks, that's what I expected (that the new version doesn't change the behaviour you are showing).

I think your assessment of the situation is correct. It looks like, to_netcdf is re-reading the whole file when in append-mode. Or better said, the underlying machinery re-reads the complete file. Would it be possible to use engine=netcdf4 just to see if this isn't affected?

@aldanor Which h5netcdf-version are you using? There have been changes to the _lookup_dimensions-function (which should not change behaviour). I'd try to check this out, could you help with a minimal script to reproduce?

I don't think this is expected. Can you try profiling to_netcdf() (e.g., %prun in IPython) to see what the source of slow down is?