Currently the rasterio tests are running on py36 only. Should I add rasterio to the other test suites as well?

Yes, except for the one that tests bare-minimal dependencies.

Yes, please!

On Wed, May 31, 2017 at 8:48 AM, Fabien Maussion notifications@github.com wrote:

@fmaussion commented on this pull request.

In xarray/backends/rasterio_.py https://github.com/pydata/xarray/pull/1260#discussion_r119395579:

• chunks : int, tuple or dict, optional
• Chunk sizes along each dimension, e.g., 5, (5, 5) or
• {'x': 5, 'y': 5}. If chunks is provided, it used to load the new
• DataArray into a dask array. This is an experimental feature; see the
• documentation for more details.
• cache : bool, optional
• If True, cache data loaded from the underlying datastore in memory as
• NumPy arrays when accessed to avoid reading from the underlying data-
• store multiple times. Defaults to True unless you specify the chunks
• argument to use dask, in which case it defaults to False. Does not
• change the behavior of coordinates corresponding to dimensions, which
• always load their data from disk into a pandas.Index.
• lock : False, True or threading.Lock, optional
• If chunks is provided, this argument is passed on to
• :py:func:dask.array.from_array. By default, a per-variable lock is
• used when reading data from netCDF files with the netcdf4 and h5netcdf

Now I understand much more what's going on, and also why salem was having troubles when adding diagnostic variables to netcdf objects. Can I update the docs all at once here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1260#discussion_r119395579, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1p7_JyTe8jdB1omZm_4V005U_ex_ks5r_YuygaJpZM4L92Pl .

I think DataArray.chunk should probably gain those parameters. There just wasn't any need for it before. On Sat, May 27, 2017 at 9:35 AM Fabien Maussion notifications@github.com wrote:

@fmaussion commented on this pull request.

In xarray/backends/rasterio_.py https://github.com/pydata/xarray/pull/1260#discussion_r118824181:

• y = np.linspace(start=y0, num=ny, stop=(y0 + (ny - 1) * dy)) +

• Get bands

• if riods.count < 1:
• raise ValueError('Unknown dims')
• bands = np.asarray(riods.indexes) +

• Attributes

• attrs = {}
• if hasattr(riods, 'crs'):

• CRS is a dict-like object specific to rasterio

• We convert it back to a PROJ4 string using rasterio itself

• attrs['crs'] = riods.crs.to_string()

• Maybe we'd like to parse other attributes here (for later)

  +
• data = indexing.LazilyIndexedArray(RasterioArrayWrapper(riods))

Implementing cache went well (appart from this bug https://github.com/pydata/xarray/issues/1429.

For chunking I have a question: DataArray.chunk() doesn't have http://xarray.pydata.org/en/stable/generated/xarray.DataArray.chunk.html#xarray.DataArray.chunk the name_prefix, token and lock keywords. Any reason for this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1260#discussion_r118824181, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1mo_XZFgCNEq4YoJunYeeWve5XHPks5r-FDDgaJpZM4L92Pl .

side note: "rasterio CRS objects" are in fact strings corresponding to a PROJ4 string that will always be understood by pyproj, rasterio, gdal, etc. Examples: '+proj=aea +lat_1=-18 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs ' or 'EPSG:4326')

Ah, that seems already pretty clean then. I was confused by the repr for a CRS() object in your comment above: https://github.com/pydata/xarray/pull/1260#issuecomment-279064452

You should be able to simply wrap RasterioArrayWrapper in an indexing.LazilyIndexedArray and then pass it into the xarray.DataArray constructor directly.

EDIT: open_dataset also wrap data with CopyOnWriteArray and optionally MemoryCachedArray. These are also probably worth using for consistency, e.g., see these lines from xarray/backends/api.py: python data = indexing.CopyOnWriteArray(variable._data) if cache: data = indexing.MemoryCachedArray(data)

I haven't looked at the implementation yet, but I agree that a separate function open_rasterio is the way to go. Structurally, this is pretty different from opening a netCDF file, which is also why I was questioning reusing the existing xarray backends system.

my initial idea to make them lazily evaluated might work, but in an ugly way: computing both lons and lats needs to be done in one single operation, and I'm not sure how this can be done in an elegant way

This could be done sanely with dask.array (example of multiple outputs from an element-wise function).

additionally, there is no way to make them show up as coordinates (as per @shoyer 's comment: #1260 (comment))

We could make this work, it just looks pretty hacky.

Getting optional coordinates is hard is because the current "backends" system in xarray was really only designed for handling netCDF-like libraries.

For example, every backend gets passed through xarray.conventions.decode_cf to use CF conventions to decode its data. You would actually need to add a "global attribute" specifying coords='lat lon' to set coordinates variables with this system.

One alternative would be to simply write a function for reading files with rasterio straight into xarray Dataset objects.

There are probably hacks that can get this working with data stores but this really calls out for the bigger refactor like https://github.com/pydata/xarray/pull/1087.

The order of the variables and coordinates is random.

Most likely you're using a dict instead of an OrderedDict somewhere.

Is it possible to make the x and y coords lazy?

See here for an example of lazily computing arrays: https://github.com/pydata/xarray/blob/62333208fc2a80c05848a12de67a10f00a6610a1/xarray/conventions.py#L314

There are other examples in that file, for decoding CF conventions.

I'd like to have the lon and lat variables listed as optional coordinates

Can you clarify what you mean by an optional coordinate?