issue_comments

I'm looking at other bits of knowledge of how backends work that are still present in the generic parts of open_dataset.

We see _autodetect_engine and _normalize_path.

We aim at removing _autodetect_engine in favour of adding a new can_open(filename_or_obj, ...) backend function declared via the plugin interface.

On the other hand _normalize_path can be removed entirely once ZarrStore.open_group will accept os.PathLike objects.

@shoyer & @jhamman just to give you an idea, I aim to see open_dataset reduced to the following:

```python def open_dataset(filename_or_obj, , engine=None, chunks=None, cache=None, backend_kwargs=None, *kwargs): filename_or_obj = nomalize_filename_or_obj(filename_or_obj) if engine is None: engine = autodetect_engine(filename_or_obj) open_backend_dataset = get_opener(engine)

backend_ds = open_backend_dataset(filename_or_obj, **backend_kwargs, **kwargs)
ds = dataset_from_backend_dataset(
    backend_ds, chunks, cache, filename_or_obj=filename_or_obj, **kwargs
)
return ds

```

Where the key observation is that backend_ds variable must be either np.ndarray or subclasses of BackendArray. That is backend should not be concerned with the in-memory representation of the variables, so they know nothign about dask, cache behaviour, etc. (@shoyer this was addressed briefly today)

@shoyer and @jhamman, I'm looking into the write support and if we let a backend return a xr.Dataset, as agreed, we lose the ability to support in-place change of a file, no update of attributes or mode='a', unless the backend has a unambiguous way to identify persistent xr.Dataset.

I'm not sure what options for in-place change are supported, but I see at least mode='a' for zarr. Let's discuss this tomorrow.

We agreed with @shoyer to go with proposal 3 for reading: the backends will return a fully build xr.Dataset

Among other advantages this reduces the amount of backend-specific documentation as the documentation of xr.Dataset and xr.Variable already contains almost all that is needed by backend developers.

The one bit of documentation that needs addressing is the use of BackendArray as the data argument for xr.Variable.

Thanks @alexamici

I'm a bit behind here: what's an example of a filter? A selection of the data?

Edit: as discussed filter == encoding

Note that the above proposals only address the store / encoding part the backend API. We will address the BackendArray part later and I expect it to be trickier.