html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/pull/1260#issuecomment-306546073,https://api.github.com/repos/pydata/xarray/issues/1260,306546073,MDEyOklzc3VlQ29tbWVudDMwNjU0NjA3Mw==,2443309,2017-06-06T16:43:30Z,2017-06-06T16:43:30Z,MEMBER,Nice work @fmaussion. Thanks for bringing this home. ,"{""total_count"": 5, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-306445080,https://api.github.com/repos/pydata/xarray/issues/1260,306445080,MDEyOklzc3VlQ29tbWVudDMwNjQ0NTA4MA==,10050469,2017-06-06T10:25:01Z,2017-06-06T10:25:01Z,MEMBER,"OK, let's get this one in and see what people will report about it. Thanks @shoyer for your patience, @NicWayand and @jhamman for the original PR, @gidden for the testing/reviews and @sgillies for rasterio!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-305446339,https://api.github.com/repos/pydata/xarray/issues/1260,305446339,MDEyOklzc3VlQ29tbWVudDMwNTQ0NjMzOQ==,10050469,2017-06-01T09:53:24Z,2017-06-01T09:53:24Z,MEMBER,"@gidden I just updated the documentation recipe to use an accessor to compute the lons and lats, let me know what you think","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-305428608,https://api.github.com/repos/pydata/xarray/issues/1260,305428608,MDEyOklzc3VlQ29tbWVudDMwNTQyODYwOA==,10050469,2017-06-01T08:39:02Z,2017-06-01T08:39:02Z,MEMBER,"@gidden yes absolutely please give it a try, I think it's ready","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-305243114,https://api.github.com/repos/pydata/xarray/issues/1260,305243114,MDEyOklzc3VlQ29tbWVudDMwNTI0MzExNA==,1217238,2017-05-31T16:32:27Z,2017-05-31T16:32:27Z,MEMBER,"> Currently the rasterio tests are running on py36 only. Should I add rasterio to the other test suites as well? Yes, except for the one that tests bare-minimal dependencies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-305240650,https://api.github.com/repos/pydata/xarray/issues/1260,305240650,MDEyOklzc3VlQ29tbWVudDMwNTI0MDY1MA==,10050469,2017-05-31T16:23:30Z,2017-05-31T16:23:30Z,MEMBER,"OK, all green. Currently the rasterio tests are running on py36 only. Should I add rasterio to the other test suites as well?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-305231172,https://api.github.com/repos/pydata/xarray/issues/1260,305231172,MDEyOklzc3VlQ29tbWVudDMwNTIzMTE3Mg==,1217238,2017-05-31T15:51:24Z,2017-05-31T15:51:24Z,MEMBER,"Yes, please! On Wed, May 31, 2017 at 8:48 AM, Fabien Maussion wrote: > *@fmaussion* commented on this pull request. > ------------------------------ > > In xarray/backends/rasterio_.py > : > > > + chunks : int, tuple or dict, optional > + Chunk sizes along each dimension, e.g., ``5``, ``(5, 5)`` or > + ``{'x': 5, 'y': 5}``. If chunks is provided, it used to load the new > + DataArray into a dask array. This is an experimental feature; see the > + documentation for more details. > + cache : bool, optional > + If True, cache data loaded from the underlying datastore in memory as > + NumPy arrays when accessed to avoid reading from the underlying data- > + store multiple times. Defaults to True unless you specify the `chunks` > + argument to use dask, in which case it defaults to False. Does not > + change the behavior of coordinates corresponding to dimensions, which > + always load their data from disk into a ``pandas.Index``. > + lock : False, True or threading.Lock, optional > + If chunks is provided, this argument is passed on to > + :py:func:`dask.array.from_array`. By default, a per-variable lock is > + used when reading data from netCDF files with the netcdf4 and h5netcdf > > Now I understand much more what's going on, and also why salem was having > troubles when adding diagnostic variables to netcdf objects. Can I update > the docs all at once here? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-304481844,https://api.github.com/repos/pydata/xarray/issues/1260,304481844,MDEyOklzc3VlQ29tbWVudDMwNDQ4MTg0NA==,1217238,2017-05-27T23:23:02Z,2017-05-27T23:23:02Z,MEMBER,"I think DataArray.chunk should probably gain those parameters. There just wasn't any need for it before. On Sat, May 27, 2017 at 9:35 AM Fabien Maussion wrote: > *@fmaussion* commented on this pull request. > ------------------------------ > > In xarray/backends/rasterio_.py > : > > > + y = np.linspace(start=y0, num=ny, stop=(y0 + (ny - 1) * dy)) > + > + # Get bands > + if riods.count < 1: > + raise ValueError('Unknown dims') > + bands = np.asarray(riods.indexes) > + > + # Attributes > + attrs = {} > + if hasattr(riods, 'crs'): > + # CRS is a dict-like object specific to rasterio > + # We convert it back to a PROJ4 string using rasterio itself > + attrs['crs'] = riods.crs.to_string() > + # Maybe we'd like to parse other attributes here (for later) > + > + data = indexing.LazilyIndexedArray(RasterioArrayWrapper(riods)) > > Implementing cache went well (appart from this bug > . > > For chunking I have a question: DataArray.chunk() doesn't have > > the name_prefix, token and lock keywords. Any reason for this? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-304137429,https://api.github.com/repos/pydata/xarray/issues/1260,304137429,MDEyOklzc3VlQ29tbWVudDMwNDEzNzQyOQ==,10050469,2017-05-25T22:05:32Z,2017-05-25T22:05:41Z,MEMBER,"Thanks @shoyer , I have addressed all your comments but one which I didn't understand. Maybe we should wait for an answer of the rasterio devs about the dtype stuff before going on too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-303800141,https://api.github.com/repos/pydata/xarray/issues/1260,303800141,MDEyOklzc3VlQ29tbWVudDMwMzgwMDE0MQ==,10050469,2017-05-24T17:48:05Z,2017-05-24T17:48:05Z,MEMBER,"This is ready for another round of reviews! I think this has come out quite nicely. Everything is much simpler now. I have: - included all your comments - removed the GIS part - added an example on how to parse lons and lats in the new ""recipes"" section ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-303369876,https://api.github.com/repos/pydata/xarray/issues/1260,303369876,MDEyOklzc3VlQ29tbWVudDMwMzM2OTg3Ng==,10050469,2017-05-23T11:29:39Z,2017-05-23T11:29:39Z,MEMBER,"> Would it be better to use the string representation of the CRS internally after reading in? Yes this was my intention. BTW, your serialization above doesn't work because the variable ""raster"" also has a CRS attr. This problem will be solved by the next iteration of my code (when data arrays will be returned instead of datasets)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-303327051,https://api.github.com/repos/pydata/xarray/issues/1260,303327051,MDEyOklzc3VlQ29tbWVudDMwMzMyNzA1MQ==,10050469,2017-05-23T08:23:59Z,2017-05-23T08:23:59Z,MEMBER,"@shoyer @gidden tjhanks for testing, this is very useful. Indeed rasterio uses a dict-like mapping for the PROJ4 strings (source: [rasterio docs](https://mapbox.github.io/rasterio/_modules/rasterio/crs.html)). This isn't a big deal since it provides the ``to_string()`` and ``from_string()`` methods which allow the do the round-trip. I personally never noted the difference because pyproj (the python interface to the PROJ.4 library) can handle both representations: ``` In [1]: import xarray as xr In [2]: ds = xr.open_rasterio('RGB.byte.tif') In [3]: ds.crs Out[3]: CRS({'init': 'epsg:32618'}) In [4]: import pyproj In [5]: pyproj.Proj(ds.crs) Out[5]: In [6]: pyproj.Proj(ds.crs.to_string()) Out[6]: ``` My suggestion here (to avoid the serialisation problems you mention @gidden ) is to convert the CRS object to a string at read time ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-303159959,https://api.github.com/repos/pydata/xarray/issues/1260,303159959,MDEyOklzc3VlQ29tbWVudDMwMzE1OTk1OQ==,10050469,2017-05-22T17:01:58Z,2017-05-22T17:01:58Z,MEMBER,"Uh, right, this is obviously not a string!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-303153678,https://api.github.com/repos/pydata/xarray/issues/1260,303153678,MDEyOklzc3VlQ29tbWVudDMwMzE1MzY3OA==,1217238,2017-05-22T16:38:03Z,2017-05-22T16:38:03Z,MEMBER,"> side note: ""rasterio CRS objects"" are in fact strings corresponding to a PROJ4 string that will always be understood by pyproj, rasterio, gdal, etc. Examples: '+proj=aea +lat_1=-18 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs ' or 'EPSG:4326') Ah, that seems already pretty clean then. I was confused by the repr for a `CRS()` object in your comment above: https://github.com/pydata/xarray/pull/1260#issuecomment-279064452","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302951509,https://api.github.com/repos/pydata/xarray/issues/1260,302951509,MDEyOklzc3VlQ29tbWVudDMwMjk1MTUwOQ==,1217238,2017-05-21T17:43:45Z,2017-05-22T16:36:32Z,MEMBER,"You should be able to simply wrap RasterioArrayWrapper in an `indexing.LazilyIndexedArray` and then pass it into the xarray.DataArray constructor directly. EDIT: `open_dataset` also wrap data with `CopyOnWriteArray` and optionally `MemoryCachedArray`. These are also probably worth using for consistency, e.g., see these lines from `xarray/backends/api.py`: ```python data = indexing.CopyOnWriteArray(variable._data) if cache: data = indexing.MemoryCachedArray(data) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302949290,https://api.github.com/repos/pydata/xarray/issues/1260,302949290,MDEyOklzc3VlQ29tbWVudDMwMjk0OTI5MA==,10050469,2017-05-21T17:04:33Z,2017-05-21T17:04:33Z,MEMBER,"Thanks for looking into this! > I'm a slightly reluctant to add a dedicated method for convert rasterio CRS objects into lat/lon arrays. It feels a little overly specialized. OK, so this corresponds to my solution **3** above (do nothing). I will however add the relevant lines of code to the documentation so that users wanting to add lons and lats to their data can do so. This will leave room for solution **2** is someone has the time to do it later. (side note: ""rasterio CRS objects"" are in fact strings corresponding to a PROJ4 string that will always be understood by ``pyproj``, ``rasterio``, ``gdal``, etc. Examples: ``'+proj=aea +lat_1=-18 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs '`` or ``'EPSG:4326'``) > Why return a Dataset rather than a DataArray? The later feels a little more natural to me, given that a rasterio describes a single array. Agreed. Again this is built out of discussions on https://github.com/pydata/xarray/issues/790, but now obsolete. This speaks even stronger against the use of a ``DataStore``, right? Is there any class I should inherit from to create a DataArray from a rasterio file? I basically need an ``init()`` and a ``__getitem()``...","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302941653,https://api.github.com/repos/pydata/xarray/issues/1260,302941653,MDEyOklzc3VlQ29tbWVudDMwMjk0MTY1Mw==,10050469,2017-05-21T14:54:19Z,2017-05-21T16:05:14Z,MEMBER,"Thanks @gidden for the comments! Will look into it. Your questions about the ``add_latlon`` makes me think that a ``kwarg`` at read time **isn't** the right approach. Here are the scenarios where you **don't** want to have the lat/lon computed per defaut: - when your data's crs is *already* WGS84, in that case ``x`` and ``y`` are lons and lats already - when your file isn't georeferenced properly - when your use case doesn't need them ([salem](http://salem.readthedocs.io/en/latest/) for example will make better use of ``crs`` than xarray) - when your file is large: computing lons and lats on a huge 2D grid is going to be prohibitively expensive. This latter use case is important, because it might be useful for users to first *subset* their data and then compute the lat lons. For this use case we could go for two options in place of the kwarg: 1. add a top level utility function ``get_latlon_from_crs`` which would work on any dataset with a proper crs (and could be extended) 2. compute lons and lats lazily (only when asked for) 3. do nothing and let the users do their own cuisine (consistent with xarray's general purpose) I don't know how to do **2** because it implies using dask to compute two related variables at the same time. Furthermore, **2** requires dask while **1** could be extended towards other datasets which have a ``crs``. Right now I tend towards **3** (because I use salem), although I guess that many users will benefit from **1**... @gidden @shoyer @benbovy : thoughts? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302931510,https://api.github.com/repos/pydata/xarray/issues/1260,302931510,MDEyOklzc3VlQ29tbWVudDMwMjkzMTUxMA==,4160723,2017-05-21T11:42:45Z,2017-05-21T11:42:45Z,MEMBER,"> a vast majority of the datasets xarray is reading are raster datasets (although in NetCDF format), hence ""open_raster"" could be confusing. ""open_rasterio"" underlines the fact that this opens ""all datasets rasterio can open"". Yep makes sense!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302652106,https://api.github.com/repos/pydata/xarray/issues/1260,302652106,MDEyOklzc3VlQ29tbWVudDMwMjY1MjEwNg==,10050469,2017-05-19T09:14:47Z,2017-05-19T09:14:47Z,MEMBER,"> which is also why I was questioning reusing the existing xarray backends system. I am starting to understand what you mean, but in the absence of template I guess this was the easiest way to go (I overtook the design of the original PR). If you agree I'd suggest you to have a more detailed look at the current PR when you have time, and we can decide what to do from here. Since the public facing API shouldn't be affected we could also keep the current design for now and go back to it later when https://github.com/pydata/xarray/pull/1087 is ready. > Just a small suggestion: to me open_raster seems a slightly better name than open_rasterio as the dataset is a 'raster', not a 'rasterio'. Yes I thought about it too, but a vast majority of the datasets xarray is reading are raster datasets (although in NetCDF format), hence ""open_raster"" could be confusing. ""open_rasterio"" underlines the fact that this opens ""all datasets rasterio can open"". I have no strong opinion about this though ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302461027,https://api.github.com/repos/pydata/xarray/issues/1260,302461027,MDEyOklzc3VlQ29tbWVudDMwMjQ2MTAyNw==,4160723,2017-05-18T16:21:58Z,2017-05-18T16:21:58Z,MEMBER,"Excellent! I'm looking forward to see this merged! Just a small suggestion: to me `open_raster` seems a slightly better name than `open_rasterio` as the dataset is a 'raster', not a 'rasterio'.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302448132,https://api.github.com/repos/pydata/xarray/issues/1260,302448132,MDEyOklzc3VlQ29tbWVudDMwMjQ0ODEzMg==,1217238,2017-05-18T15:54:49Z,2017-05-18T15:54:49Z,MEMBER,"I haven't looked at the implementation yet, but I agree that a separate function `open_rasterio` is the way to go. Structurally, this is pretty different from opening a netCDF file, which is also why I was questioning reusing the existing xarray backends system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-302255969,https://api.github.com/repos/pydata/xarray/issues/1260,302255969,MDEyOklzc3VlQ29tbWVudDMwMjI1NTk2OQ==,10050469,2017-05-17T23:07:44Z,2017-05-17T23:10:11Z,MEMBER,"Folks, I finally managed to find a couple of hours to wrap this up: this is now ready for review. Everything seems to work the way I'd like it to work, and only one thing is missing: the lazy computation of lons and lats with dask (I don't know how to do this quickly and I have not enough time to spend on this right now, unfortunately). This has been waiting for too long now, so I suggest to merge this when ready and this feature later on. The solution retained for the API is to add a new ``open_rasterio`` top-level function: this makes sense since many keywords of ``open_dataset`` aren't relevant for rasterio datasets. It also underlines that rasterio datasets are quit different from xarray's data model. another example I could add to the soon to come xarray gallery could be: ```python import xarray as xr import matplotlib.pyplot as plt import cartopy.crs as ccrs ds = xr.open_rasterio('RGB.byte.tif', add_latlon=True) ax = plt.subplot(projection=ccrs.PlateCarree()) ds.raster.sel(band=1).plot(ax=ax, x='lon', y='lat', transform=ccrs.PlateCarree()); ax.coastlines('10m'); ```` ![rasterio_map](https://cloud.githubusercontent.com/assets/10050469/26179862/40eb7c18-3b66-11e7-99d6-3455dca4889a.png) cc @gidden @jhamman @shoyer ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-284225363,https://api.github.com/repos/pydata/xarray/issues/1260,284225363,MDEyOklzc3VlQ29tbWVudDI4NDIyNTM2Mw==,10050469,2017-03-05T12:42:45Z,2017-03-05T12:42:45Z,MEMBER,"@shoyer thanks for the tips, I think that getting the lons and lats from dask is probably the most elegant method. After trying various things I am still struggling with dask, and in particular on how to apply ``elemwise`` to functions like ``np.meshgrid`` (I get shape broadcasting errors). To get me started, I'd be grateful for an example on how to use dask to replace the code snippet below: ```python import xarray as xr import numpy as np ds = xr.DataArray(np.zeros((2, 3)), coords={'x': np.arange(3), 'y': np.arange(2)}, dims=['y', 'x']).to_dataset(name='data') # non-dask version lon, lat = np.meshgrid(ds.x, ds.y) ds['lon'] = (('y', 'x'), lon) ds['lat'] = (('y', 'x'), lat) ds.set_coords(['lon', 'lat'], inplace=True) print(ds) ``` Thanks a lot!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279897816,https://api.github.com/repos/pydata/xarray/issues/1260,279897816,MDEyOklzc3VlQ29tbWVudDI3OTg5NzgxNg==,1217238,2017-02-15T02:13:27Z,2017-02-15T02:13:27Z,MEMBER,"> my initial idea to make them lazily evaluated might work, but in an ugly way: computing both lons and lats needs to be done in one single operation, and I'm not sure how this can be done in an elegant way This could be done sanely with dask.array ([example of multiple outputs from an element-wise function](https://github.com/dask/dask/blob/1e2bf8973ff3706e68a5a55d6392a5798bd072bd/dask/array/ufunc.py#L116-L133)). > additionally, there is no way to make them show up as coordinates (as per @shoyer 's comment: #1260 (comment)) We could make this work, it just looks pretty hacky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279686053,https://api.github.com/repos/pydata/xarray/issues/1260,279686053,MDEyOklzc3VlQ29tbWVudDI3OTY4NjA1Mw==,10050469,2017-02-14T11:43:27Z,2017-02-14T11:43:27Z,MEMBER,"I made some progress with the lazy indexing, I'd be glad to have a first rough feedback on whether this is going in the right direction or not. We have a decision to make regarding the API: I think that creating the optional lon and lat coords automatically is not a good idea: - in some cases, the x and y coordinates are already lons and lats and the 2D coords are obsolete - for big data files this is going to take ages and take a lot of memory - my initial idea to make them lazily evaluated might work, but in an ugly way: computing _both_ lons and lats needs to be done in one single operation, and I'm not sure how this can be done in an elegant way - additionally, there is no way to make them show up as coordinates (as per @shoyer 's comment: https://github.com/pydata/xarray/pull/1260#issuecomment-279101252) The current implementation delegates this task to a utility function (``get_latlon_coords_from_crs``). It is currently very rasterio specific, but could be made more general -- API to be defined. Thoughts? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279101252,https://api.github.com/repos/pydata/xarray/issues/1260,279101252,MDEyOklzc3VlQ29tbWVudDI3OTEwMTI1Mg==,1217238,2017-02-11T00:14:36Z,2017-02-11T00:14:36Z,MEMBER,"Getting optional coordinates is hard is because the current ""backends"" system in xarray was really only designed for handling netCDF-like libraries. For example, every backend gets passed through `xarray.conventions.decode_cf` to use CF conventions to decode its data. You would actually need to add a ""global attribute"" specifying `coords='lat lon'` to set coordinates variables with this system. One alternative would be to simply write a function for reading files with rasterio straight into xarray Dataset objects. There are probably hacks that can get this working with data stores but this really calls out for the bigger refactor like https://github.com/pydata/xarray/pull/1087. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279076276,https://api.github.com/repos/pydata/xarray/issues/1260,279076276,MDEyOklzc3VlQ29tbWVudDI3OTA3NjI3Ng==,10050469,2017-02-10T21:52:35Z,2017-02-10T21:52:35Z,MEMBER,"thanks @shoyer , I think I can work on this a bit further now and I'll get back to you if I have more questions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279072956,https://api.github.com/repos/pydata/xarray/issues/1260,279072956,MDEyOklzc3VlQ29tbWVudDI3OTA3Mjk1Ng==,10050469,2017-02-10T21:37:45Z,2017-02-10T21:37:45Z,MEMBER,"> Can you clarify what you mean by an optional coordinate? Yes sorry, I meant the have them listed as coordinates without ``*`` instead of ``Data variables``.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279070566,https://api.github.com/repos/pydata/xarray/issues/1260,279070566,MDEyOklzc3VlQ29tbWVudDI3OTA3MDU2Ng==,1217238,2017-02-10T21:28:02Z,2017-02-10T21:28:02Z,MEMBER,"> The order of the variables and coordinates is random. Most likely you're using a dict instead of an OrderedDict somewhere. > Is it possible to make the x and y coords lazy? See here for an example of lazily computing arrays: https://github.com/pydata/xarray/blob/62333208fc2a80c05848a12de67a10f00a6610a1/xarray/conventions.py#L314 There are other examples in that file, for decoding CF conventions. > I'd like to have the lon and lat variables listed as optional coordinates Can you clarify what you mean by an optional coordinate? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158 https://github.com/pydata/xarray/pull/1260#issuecomment-279064452,https://api.github.com/repos/pydata/xarray/issues/1260,279064452,MDEyOklzc3VlQ29tbWVudDI3OTA2NDQ1Mg==,10050469,2017-02-10T21:00:16Z,2017-02-10T21:00:16Z,MEMBER,"Before I'll get more into details with what needs to be done with rasterio itself, I'd like to get some xarray internals ready first. No need to do a full review yet, but I'd appreciate help with the following points: 1. The order of the variables and coordinates is random. The current ```__repr__``` can look like this: ``` Dimensions: (band: 1, x: 4, y: 3) Coordinates: * x (x) float64 1.0 1.5 2.0 2.5 * band (band) int64 1 * y (y) float64 2.0 1.0 0.0 Data variables: lat (y, x) float64 2.0 2.0 2.0 2.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 raster (band, y, x) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ... lon (y, x) float64 1.0 1.5 2.0 2.5 1.0 1.5 2.0 2.5 1.0 1.5 2.0 2.5 Attributes: crs: CRS({'wktext': True, 'proj': 'longlat', 'no_defs': True, 'ellps': 'WGS84'}) transform: [1.0, 0.5, 0.0, 2.0, 0.0, -1.0] ``` How is this possible? 2. Is it possible to make the ``x`` and ``y`` coords lazy? Could you point me on a place in the code where this has been done already? 3. I'd like to have the ``lon`` and ``lat`` variables listed as optional coordinates, and also make them lazy. Any hint? Thanks a lot for your help, I'm afraid this is going to need a few iterations ;-) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,206905158