issue_comments

This package addresses this issue: https://github.com/recipy/recipy

They are working on xarray support: https://github.com/recipy/recipy/issues/176

Thanks @shoyer! I think after this discussion that this is something we should handle within our own wrapper. There does not seem to be a clean one size fits all solution warranting inclusion in xarray, unless I'm mistaken. If so, I hope someone chimes in to correct me.

Yes, my main concern is code bloat. Storing things like the computation graph and command line flags used to invoke a script are certainly useful things to do, and I use versions of this stuff all the time. But they are orthogonal to the labeled data focus of xarray so they belong better in another library.

If you want to take this approach, you might start by using something like dask.imperative, and extracting the task dependencies from the resulting task graph. Or you could even try to work with the full dask graphs created by using dask.array with xarray, but these can get pretty big.

Getting parameters from the calling script is even easier -- just inspect sys.argv and set it as an attribute before saving files (or write your own function). The main complexity here is picking a convention, not implementing it.

Thanks @shoyer. Is the fundamental issue code boat or is the concern essentially storing a history that may not be useful? The reason I ask is because we could capture all method operation names and sore them to a log. You are right that this could get really messy and potentially meaningless as a result.

However, what about storing the text related to the calling script used to call to-netcdf? Perhaps that may work in terms of enhanced documentation. I think there could be some utility to be gained here, especially in team environments.

This might be out of scope for xarray.

It's relatively straight forward to build computation graphs, but summarizing them in a useful, succint way is hard. There are a lot of judgment calls involved.