issue_comments

Having started writing a test, I now think that encoding['source'] is backend specific. Here it is implemented in netcdf4: https://github.com/pydata/xarray/blob/70e9eb8fc834e4aeff42c221c04c9713eb465b8a/xarray/backends/netCDF4_.py#L386 but I don't see it for pynio for instance: https://github.com/pydata/xarray/blob/70e9eb8fc834e4aeff42c221c04c9713eb465b8a/xarray/backends/pynio_.py#L77-L81

Is this something that we want to mandate that backends provide?

Should I add a test that expects .encoding['source'] to ensure its continued presence?

Ah I don't think I understood that adding source to encoding was a new addition. In latest master ('0.11.0+3.g70e9eb8) this works fine:

```python def func(ds): var = next(var for var in ds) return ds.assign(path=ds[var].encoding['source'])

ds = xr.open_mfdataset(['./air_1.nc', './air_2.nc'], concat_dim='path', preprocess=func) ```

I do think it is misleading though that after you've concatenated the data, the encoding['source'] on a concatenated var seems to be the first path.

```python

ds['air'].encoding['source'] '~/air_1.nc' ```

I'll close this one though since there is a clear way to access the filename. Thanks for the tip @jhamman!

A dirty fix would be to add an attribute to each dataset.

I thought @jhamman was suggesting that already exists, but I couldn't find it: https://github.com/pydata/xarray/issues/2550#issuecomment-437157299

Maybe we can inspect the preprocess function like this:

```python

preprocess = lambda a, b: print(a, b) preprocess .code.co_varnames ('a', 'b') ```

This response is ordered, so the first one can always be ds regardless of its name and then we can look for special names (like filename) in the rest.

From this answer: https://stackoverflow.com/a/4051447/4021797

@jhamman that looks pretty good, but I'm not seeing the source in the encoding dict. Is this what you were expecting?

```python def func(ds): var = next(var for var in ds) return ds.assign(path=ds[var].encoding['source'])

xr.open_mfdataset(['./ST4.2018092500.01h', './ST4.2018092501.01h'], engine='pynio', concat_dim='path', preprocess=func) python-traceback

KeyError Traceback (most recent call last) <ipython-input-49-184da62ce353> in <module>() ----> 1 ds = xr.open_mfdataset(['./ST4.2018092500.01h', './ST4.2018092501.01h'], engine='pynio', concat_dim='path', preprocess=func)

/opt/conda/lib/python3.6/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, autoclose, parallel, **kwargs) 612 file_objs = [getattr_(ds, '_file_obj') for ds in datasets] 613 if preprocess is not None: --> 614 datasets = [preprocess(ds) for ds in datasets] 615 616 if parallel:

/opt/conda/lib/python3.6/site-packages/xarray/backends/api.py in <listcomp>(.0) 612 file_objs = [getattr_(ds, '_file_obj') for ds in datasets] 613 if preprocess is not None: --> 614 datasets = [preprocess(ds) for ds in datasets] 615 616 if parallel:

<ipython-input-48-fd450fa1393a> in func(ds) 1 def func(ds): 2 var = next(var for var in ds) ----> 3 return ds.assign(path=ds[var].encoding['source'])

KeyError: 'source' ```

xarray version: '0.11.0+1.g575e97ae'

There is a preprocess argument. You provide a function and it is run on every file.

Yes but the input to that function is just the ds, I couldn't figure out a way to get the filename from within a preprocess function. This is what I was doing to poke around in there:

```python def func(ds): import pdb; pdb.set_trace()

xr.open_mfdataset(['./ST4.2018092500.01h', './ST4.2018092501.01h'], engine='pynio', concat_dim='path', preprocess=func) ```