issue_comments

Is this something that we want to mandate that backends provide?

I think it would be better to do this systematically, e.g., inside xarray.open_dataset(). We would need to verify that filename_or_obj is provided as a string, but if so we could add it into encoding on the Dataset object.

Yes, that sounds great! Potentially this would be a good opportunity for a doc update, too.

On Mon, Nov 19, 2018 at 6:36 AM Julia Signell notifications@github.com wrote:

Should I add a test that expects .encoding['source'] to ensure its continued presence?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2550#issuecomment-439913493, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1tyOrzPg3dUX2cPolk51Uoa4i7j5ks5uwsH2gaJpZM4YVZ1J .

I'm not sure .encoding['source'] should really be relied upon -- it wasn't really an intentional API decision. But I guess it's harmless enough to include it...

True, maybe we should track down why that isn't happening with your dataset

A dirty fix would be to add an attribute to each dataset.

The danger with inspecting user provided functions is that it's pretty fragile, e.g., it fails if you use provide a signature like args, *kwargs (which can happen pretty easily with decorators). Probably the best option is to come up with a new keyword argument to replace "preprocess" and to deprecate the current preprocess (if we can think of another good name). We could also do a deprecation cycle with FutureWarning, but that's pretty painful.

On Fri, Nov 9, 2018 at 12:29 PM Julia Signell notifications@github.com wrote:

Maybe we can inspect the preprocess function like this:

preprocess = lambda a, b: print(a, b)>>> preprocess .code.co_varnames ('a', 'b')

This response is ordered, so the first one can always be ds regardless of its name and then we can look for special names (like filename) in the rest.

From this answer: https://stackoverflow.com/a/4051447/4021797

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2550#issuecomment-437433736, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1kvSMx83hACWxXD3tOyIHGqyqum8ks5utbtigaJpZM4YVZ1J .

Hmm... Sorry @jsignell. I thought preprocess passed the filename too.

@jhamman The problem is that xarray needs way to figure out what arguments it can safely pass to preprocess, i.e., it needs to inspect the proprocess function and see if it can handle a filename argument. It's not obvious what the best way to do this in a backwards compatible way is...

@shoyer and @jsignell - I'd also be happy to see this added to the preprocess function. Ideally the function signature would look like:

python def preprocess(ds, filename=None): ... return ds

This would avoid a breaking change and allow us to add additional kwargs at a later date if need be.

Hmm. It really seems like the preprocess function should pass in the file-name along with the dataset.

@jsignell - perhaps not a very pretty solution but we do save the source of each variable in the encoding dictionary.

python ds['varname'].encoding['source']

Presumably, you could unpack this via a preprocess step.

There is a preprocess argument. You provide a function and it is run on every file.