issue_comments

This is gotten a little stale at this point. Coders did make it in, though we haven't moved over everything from conventions.py yet. This interface should also probably be reconciled with the use of CachingFileManager.

@shoyer - since I've started work on a related subject in #3166, I'm wondering if you think this PR is capable of being revived? I think we'll have some time soon to pick this up in one form or another.

@shoyer regarding the xarray-grib-driver (not public yet, sorry) we have been working on the GRIB side lately and I didn't review this branch until today. Now we are coming back to the xarray side and I welcome the new "pluggability" of the encoding / decoding engine. Anyway since a lot of the coding work is already done by the ecCodes library my hope is that most of the complexity will stay outside of xarray anyway.

See https://github.com/pydata/xarray/pull/1752 for getting started on filters. I had a productive plane ride!

@alimanfoo thanks for the pointer to numcodecs. I'm sure that will come in handy eventually. Most of the "filters" xarray uses here are at a slightly higher level decoding/encoding metadata. The actual filters themselves are generally quite simple, e.g., just coercing a dtype, but there is lots of metadata to keep track of to know when they are appropriate to use.

FWIW for the filters, if it would be possible to use the numcodecs Codec API http://numcodecs.readthedocs.io/en/latest/abc.html then that could be beneficial beyond xarray, as any work you put into developing filters could then be used elsewhere (e.g., in zarr).

On Thu, Nov 30, 2017 at 12:05 PM, Stephan Hoyer notifications@github.com wrote:

OK, I'm going to try to reboot this and finish it up in the form of an API that we'll be happy with going forward. I just discovered two more xarray backends over the past two days (in Unidata's Siphon and something @alexamici https://github.com/alexamici and colleagues are writing to reading GRIB files), so clearly the demand is here.

One additional change I'd like to make is try to rewrite the encoding/decoding functions for variables into a series of invertible coding filters that can potentially be chained together in a flexible way (this is somewhat inspired by zarr). This will allow different backends to mix/match filters as necessary, depending on their particular needs. I'll start on that in another PR.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1087#issuecomment-348169779, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QmzjKBnyjuGDFN6btGfhr2eFrhoiks5s7poXgaJpZM4Kq10M .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health http://cggh.org Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a http://purl.org/net/alimanlimanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

OK, I'm going to try to reboot this and finish it up in the form of an API that we'll be happy with going forward. I just discovered two more xarray backends over the past two days (in Unidata's Siphon and something @alexamici and colleagues are writing to reading GRIB files), so clearly the demand is here.

One additional change I'd like to make is try to rewrite the encoding/decoding functions for variables into a series of invertible coding filters that can potentially be chained together in a flexible way (this is somewhat inspired by zarr). This will allow different backends to mix/match filters as necessary, depending on their particular needs. I'll start on that in another PR.

Stephan this looks awesome! Should simplify the backends a lot!

I do worry that it will be painful to refactor the existing backends. But I guess that is the cost of progress.

CC @alexamici for interest in the backends refactor