html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/2042#issuecomment-1093630278,https://api.github.com/repos/pydata/xarray/issues/2042,1093630278,IC_kwDOAMm_X85BL3lG,601025,2022-04-09T03:14:41Z,2022-04-09T03:14:41Z,NONE,"Thanks for closing this dcherian, I had completely forgotten about it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-1093555182,https://api.github.com/repos/pydata/xarray/issues/2042,1093555182,IC_kwDOAMm_X85BLlPu,2448579,2022-04-09T01:19:10Z,2022-04-09T01:19:10Z,MEMBER,I think rioxarray is now the recommended solution.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-565012344,https://api.github.com/repos/pydata/xarray/issues/2042,565012344,MDEyOklzc3VlQ29tbWVudDU2NTAxMjM0NA==,7105759,2019-12-12T13:44:19Z,2019-12-12T13:44:19Z,NONE,"Hi! I just wrote this tiny tifffile wrapper for my own purposes, with support for xarray: https://pypi.org/project/xtiff. Not properly tested yet, but happy to take issues/pull requests (e.g. for additional write modes). Also, feel free to integrate it into xarray. The current version is xarray-agnostic, that's why I wrote it as an independent package.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-510693104,https://api.github.com/repos/pydata/xarray/issues/2042,510693104,MDEyOklzc3VlQ29tbWVudDUxMDY5MzEwNA==,8699967,2019-07-11T23:46:38Z,2019-07-11T23:46:38Z,CONTRIBUTOR,"A new project called [rioxarray](http://github.com/corteva/rioxarray) has a `to_raster` method with the default driver of `GTiff`.
You can use it like so:
```
import rioxarray
import xarray
xds = xarray.open_rasterio(""myfile.tif"")
wgs84_xds = xds.rio.reproject(""EPSG:4326"")
wgs84_xds.rio.to_raster(""myfile_wgs84.tif"")
```
It currently only supports 2d/3d `DataArray`s. So, you would have to iterate over your variables to export each one to a raster if you have a `Dataset`.","{""total_count"": 8, ""+1"": 7, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-457584997,https://api.github.com/repos/pydata/xarray/issues/2042,457584997,MDEyOklzc3VlQ29tbWVudDQ1NzU4NDk5Nw==,601025,2019-01-25T14:13:35Z,2019-01-25T14:13:35Z,NONE,"Here is an old chunk of code I wrote awhile back to do this. Please
note three things. There is the metadata attached to the file (I think
it was through ""tags""), metadata attached to the metadata ""meta""
variable, and some metadata that is attached on a per-band basis. It
can be problematic when you assume that the info is global to the image
and is embedded somehow (it took me weeks to figure some of this out).
Also note that I do per-band and image statistics... Also, I did not
keep good enough notes and cannot remember where I got some of the hints
and are just as likely to come from published examples that have been
hacked to marginally work. Also, the .xml weirdness has to do in part
with historic artifacts of our particular dataset that is over 3.5
petabytes, and cannot easily be updated, and is easier to hack in the
code.
Hope this helps:
=====================
def to_tiff(data, fname, template=None, **kwargs):
import numpy as np
# check and promote the number of dimentio(1)ns for consistency
nbands = data.ndim
if 2 == nbands:
# expand the array so that it is least 3D (ie stacks of
surfaces)
import numpy as np
data = np.expand_dims(data,axis=0)
elif 3 != nbands: # nothing to do if it is already 3D
print(""Error: to_tiff can only currently deal with 2D and 3D
data"")
return
profile = {}
tags = {}
tmpl = None
if template:
tmpl = rasterio.open(template,'r')
profile = tmpl.profile.copy()
tags = tmpl.tags()
# the metadata should be appended. Cache here to
# simplify variable replacement below.
meta = {}
if 'meta' in profile:
meta.update(profile['meta'])
if 'meta' in kwargs:
meta.update(kwargs['meta'])
# overwrite anything inheritied from the template with
# user supplied args
profile.update(kwargs)
# overwrite bits that write the array as geotiff and
# save the cached metadata
profile['driver'] = 'GTiff'
profile['count'] = data.shape[0]
profile['width'] = data.shape[2]
profile['height'] = data.shape[1]
profile['meta'] = meta
if 'dtype' not in profile:
profile['dtype'] = type(data[0,0,0])
# if you do not remove the previously associated .xml file,
# then the tags and metadata can get corrupted.
try:
os.remove(fname)
os.remove(fname+"".xml"")
except:
pass
# now create and save the array to a file
with rasterio.open(fname,'w',**profile) as out:
for b in range(data.shape[0]):
#print(""\nprocessing band %d""%(b+1))
out.write(data[b].astype(profile['dtype']), b+1)
# caluclate the stats for each band
# not sure what the proper name for per band stats is in
QGIS
stats = {
'STATISTICS_MINIMUM': np.nanmin(data[b]),
'STATISTICS_MAXIMUM': np.nanmax(data[b]),
'STATISTICS_MEAN': np.nanmean(data[b]),
'STATISTICS_STDDEV': np.nanstd(data[b])}
out.update_tags(b+1,**stats)
#print("" stats= %s""%str(stats))
# now calculate the stats across all the bands
stats = {
'STATISTICS_MINIMUM': np.nanmin(data),
'STATISTICS_MAXIMUM': np.nanmax(data),
'STATISTICS_MEAN': np.nanmean(data),
'STATISTICS_STDDEV': np.nanstd(data)}
out.update_tags(**tags)
if 'tags' in kwargs:
out.update_tags(**kwargs['tags'])
out.update_tags(**stats)
#print(""\n overall stats= %s\n""%str(stats))
del tmpl
On Jan 23 2019 1:29 PM, Guillaume Eynard-Bontemps wrote:
> Thanks @djhoese @ebo.
>
> @ebo if you have some examples, that would be really cool!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-456960684,https://api.github.com/repos/pydata/xarray/issues/2042,456960684,MDEyOklzc3VlQ29tbWVudDQ1Njk2MDY4NA==,7217358,2019-01-23T20:49:40Z,2019-01-23T21:07:43Z,NONE,"Hi @guillaumeeb ,
I have also created geotiff files from xarray using rasterio. I was working
in with a a to_tiff method adapted to my workflow (
https://github.com/alexsalr/ciat_monitor_crops/blob/master/b_Temporal_Stack/xr_eotemp.py
), based in these methods https://github.com/robintw/XArrayAndRasterio.
This was with rasterio prior to 1.0, so I don't know if the new version
changes the behaviour.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-456954321,https://api.github.com/repos/pydata/xarray/issues/2042,456954321,MDEyOklzc3VlQ29tbWVudDQ1Njk1NDMyMQ==,17138587,2019-01-23T20:29:39Z,2019-01-23T20:29:39Z,NONE,"Thanks @djhoese @ebo.
@ebo if you have some examples, that would be really cool!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-456499728,https://api.github.com/repos/pydata/xarray/issues/2042,456499728,MDEyOklzc3VlQ29tbWVudDQ1NjQ5OTcyOA==,601025,2019-01-22T18:01:36Z,2019-01-22T18:01:36Z,NONE,"I work with geotiff all the time. A separate to_tiff is not needed.
The trick is that there are two separate sections/areas where the
metadata is stored. You will know where/how to store that information.
I do not have access to any of that code at the moment. If you cannot
find the examples I will try to hack an example or three once I get back
to work.
On Jan 22 2019 7:05 AM, David Hoese wrote:
> @guillaumeeb Not that I know of but I'm not completely in the loop
> with xarray. There is the geoxarray project that I started
> (https://github.com/geoxarray/geoxarray) but really haven't had any
> time to work on it. Otherwise you could look at the [satpy
> library](https://satpy.readthedocs.io/en/latest/) or its dependency
> library
>
> [trollimage](https://trollimage.readthedocs.io/en/latest/xrimage.html)
> which uses rasterio but it assumes some things about how data is
> structured including an `'area'` in `.attrs` from
> [pyresample](https://pyresample.readthedocs.io/en/latest/). Sorry I
> don't have a better idea.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-456410755,https://api.github.com/repos/pydata/xarray/issues/2042,456410755,MDEyOklzc3VlQ29tbWVudDQ1NjQxMDc1NQ==,1828519,2019-01-22T14:05:33Z,2019-01-22T14:05:33Z,CONTRIBUTOR,@guillaumeeb Not that I know of but I'm not completely in the loop with xarray. There is the geoxarray project that I started (https://github.com/geoxarray/geoxarray) but really haven't had any time to work on it. Otherwise you could look at the [satpy library](https://satpy.readthedocs.io/en/latest/) or its dependency library [trollimage](https://trollimage.readthedocs.io/en/latest/xrimage.html) which uses rasterio but it assumes some things about how data is structured including an `'area'` in `.attrs` from [pyresample](https://pyresample.readthedocs.io/en/latest/). Sorry I don't have a better idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-456408670,https://api.github.com/repos/pydata/xarray/issues/2042,456408670,MDEyOklzc3VlQ29tbWVudDQ1NjQwODY3MA==,17138587,2019-01-22T13:59:31Z,2019-01-22T13:59:31Z,NONE,"Did someone advanced in the direction of a `to_rasterio` or `to_tiff` implementation in Xarray? Or in a geo-xarray?
We're begining experimentations on Dask/Xarray/GeoTiff analysis at @CNES. Xarray/Dask is really useful and promising for temporal stack analysis on a given area, but still a bit out of the box with rasterio.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-406698807,https://api.github.com/repos/pydata/xarray/issues/2042,406698807,MDEyOklzc3VlQ29tbWVudDQwNjY5ODgwNw==,601025,2018-07-20T19:05:12Z,2018-07-20T19:05:12Z,NONE,"On Jul 20 2018 12:57 PM, David Hoese wrote:
> I'd like to add to this discussion the issue I brought up here #2288.
> It is something that could/should probably result in a new xarray
> add-on package for doing these type of operations. For example, I
> work
> on the pyresample and satpy projects. Pyresample uses its own
> ""AreaDefinition"" objects to define the geolocation/projection
> information. SatPy uses these AreaDefinitions by setting
> `DataArray.attrs['area']` and using then when necessary. This
> includes
> the ability to write geotiffs using rasterio and a custom array-like
> class for writing dask chunks to the geotiff between separate threads
> (does not work multiprocess, yet).
I would love to see these additions (or some recipies on how to do it
as xarray stands). As a note, I figured out a rather simple way using
with rasterio.open(...,'w',**profile) to effect the write. That might
help in the short to medium term.
I am also interested in looking at your Pyresample and well as
something similar to the morphological operators (in this context
specifically measure).
Best of success!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-406696890,https://api.github.com/repos/pydata/xarray/issues/2042,406696890,MDEyOklzc3VlQ29tbWVudDQwNjY5Njg5MA==,1828519,2018-07-20T18:57:38Z,2018-07-20T18:58:10Z,CONTRIBUTOR,"I'd like to add to this discussion the issue I brought up here #2288. It is something that could/should probably result in a new xarray add-on package for doing these type of operations. For example, I work on the pyresample and satpy projects. Pyresample uses its own ""AreaDefinition"" objects to define the geolocation/projection information. SatPy uses these AreaDefinitions by setting `DataArray.attrs['area']` and using then when necessary. This includes the ability to write geotiffs using rasterio and a custom array-like class for writing dask chunks to the geotiff between separate threads (does not work multiprocess, yet).
Edit: by ""add-on"" I mean something like ""geoxarray"" where it is an optional dependency for a user that depends completely on xarray.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385532574,https://api.github.com/repos/pydata/xarray/issues/2042,385532574,MDEyOklzc3VlQ29tbWVudDM4NTUzMjU3NA==,601025,2018-04-30T21:20:49Z,2018-04-30T21:20:49Z,NONE,"When I poked at this I could not figure out how to keep the internal cached states separate. That may have been because the processing loop was opening many different images, and not just one. I'm glad you found a way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385501221,https://api.github.com/repos/pydata/xarray/issues/2042,385501221,MDEyOklzc3VlQ29tbWVudDM4NTUwMTIyMQ==,306380,2018-04-30T19:20:04Z,2018-04-30T19:20:04Z,MEMBER,"> gdal can read/write windows:
I'm aware. See this doc listed above for rasterio: https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html#writing
Background here is that rasterio more-or-less wraps around GDAL, but with interfaces that are somewhat more idiomatic to this community.
> I wonder how you got that to work other than setting up a slave read process that handles all reads.
We've run into these issues before as well. Typically we handle them with locks of various types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385498531,https://api.github.com/repos/pydata/xarray/issues/2042,385498531,MDEyOklzc3VlQ29tbWVudDM4NTQ5ODUzMQ==,601025,2018-04-30T19:09:41Z,2018-04-30T19:09:41Z,NONE,"@mrocklin gdal can read/write windows:
```
# Read raster as arrays
banddataraster = raster.GetRasterBand(1)
dataraster = banddataraster.ReadAsArray(xoff, yoff, xcount, ycount).astype(numpy.float)
```
from: https://pcjericks.github.io/py-gdalogr-cookbook/raster_layers.html
Also see BandReadAsArray and BandWriteAsArray in http://gdal.org/python/osgeo.gdal_array-module.html (which appear to be a read/write gdal.Band.ReadAsArray method and gdal.Band.WriteArray method respectively).
But there are some got'yas there in that GDAL as far as I recall is not thread safe. I wonder how you got that to work other than setting up a slave read process that handles all reads.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385496306,https://api.github.com/repos/pydata/xarray/issues/2042,385496306,MDEyOklzc3VlQ29tbWVudDM4NTQ5NjMwNg==,601025,2018-04-30T19:01:29Z,2018-04-30T19:01:29Z,NONE,"@mrocklin it was the windowed-rw example that prompted a number of my early questions about dask.array and xarray equivalents. Maybe someting along the lines of the following would also be helpful:
https://gis.stackexchange.com/questions/158527/is-it-possible-to-read-raster-files-by-block-with-rasterio/158528#158528","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385492564,https://api.github.com/repos/pydata/xarray/issues/2042,385492564,MDEyOklzc3VlQ29tbWVudDM4NTQ5MjU2NA==,601025,2018-04-30T18:48:16Z,2018-04-30T18:48:16Z,NONE,"So far as I have run into open_rasterio takes care of most things out of the box. Besides how to deal with chunks, there is also how to deal with several types of metadata:
* the regular metadata which rasterio access by either the meta or profile variables.
* user defined metadata dictionary which rasterio use 'tags()'
* per band metadata dictionary which rasterio uses 'tags(band)'
Whether xarray/open_rasterio uses the same interface or not, there will be a need to deal with file metadata and per-band metadata.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385488636,https://api.github.com/repos/pydata/xarray/issues/2042,385488636,MDEyOklzc3VlQ29tbWVudDM4NTQ4ODYzNg==,306380,2018-04-30T18:34:21Z,2018-04-30T18:34:21Z,MEMBER,My first attempt would be to use this API: https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html#writing,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385488169,https://api.github.com/repos/pydata/xarray/issues/2042,385488169,MDEyOklzc3VlQ29tbWVudDM4NTQ4ODE2OQ==,306380,2018-04-30T18:32:44Z,2018-04-30T18:32:44Z,MEMBER,"> My impression is that there are will be some (significant) development challenges
If you're able to expand on this that would be welcome.
> that perhaps only supports a few rasterio file formats
My hope would be that rasterio/GDAL would handle the many-file-format issue for us *if* they support writing in chunks. I also lack experience here though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385475461,https://api.github.com/repos/pydata/xarray/issues/2042,385475461,MDEyOklzc3VlQ29tbWVudDM4NTQ3NTQ2MQ==,2443309,2018-04-30T17:48:13Z,2018-04-30T17:48:13Z,MEMBER,"I agree, it would be nice to have the `to_rasterio` functionality. My impression is that there are will be some (significant) development challenges, particularly related to rasterio's support for many file formats, but those can probably be sorted out by a committed developer or by partnering with a rasterio developer.
This is a bit outside my area of expertise but I imagine it will be useful to see a prototype, that perhaps only supports a few rasterio file formats, before diving into the xarray backends to implement this full bore. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-385451826,https://api.github.com/repos/pydata/xarray/issues/2042,385451826,MDEyOklzc3VlQ29tbWVudDM4NTQ1MTgyNg==,306380,2018-04-30T16:26:13Z,2018-04-30T16:26:13Z,MEMBER,"When writing https://github.com/pydata/xarray/issues/2093 I came across this issue and thought I'd weigh in.
The GIS community seems like a fairly close neighbor to XArray's current community. Some API compatibility here might be a good to expand the community. I definitely agree that GeoTiff does not implement the full XArray model, but it might be useful to support the subset of datasets that do, just so that round-trip operations can occur. For example, it might be nice if the following worked:
```python
dset = xr.open_rasterio(...)
# do modest modifications to dest
dset.to_rasterio(...)
```
My hope would be that the rasterio/GDAL data model would be consistent enough so that we could detect and err early if the dataset was not well-formed.","{""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379850842,https://api.github.com/repos/pydata/xarray/issues/2042,379850842,MDEyOklzc3VlQ29tbWVudDM3OTg1MDg0Mg==,601025,2018-04-09T18:35:19Z,2018-04-09T18:35:19Z,NONE,"On Apr 9 2018 11:43 AM, Ryan Abernathey wrote:
>> I really need to know what xarray can and is planning to do with
>> tiff's
>> so that I can not only use them but also document stuff for a dozen
>> or
>> more of my coworkers
>
> @ebo, we are very glad to hear your input about how you might use
> xarray together with geotiff data. The majority of xarray developers
> are coming from a netCDF background, so this is somewhat new
> territory
> for us. It sounds like you have a real need for the computational
> tools that xarray provides. Engaging the geotiff community could
> potentially be very advantageous for xarray, since it could bring
> lots
> of new users. On the other hand, there are already lots of powerful
> tools in the geotiff space, and we have limited resources (i.e.
> time),
> so we need to be a bit conservative.
>
> It would probably be useful to clarify how the decision making
> process works on things like this for open source projects. There is
> no xarray master plan that can provide a simple answer to your
> question of ""what xarray is planning to do with tiffs"". The main
> questions that have to be answered when deciding whether to add a big
> new feature are
> - Does this feature make sense within the ""scope"" of the project?
> (Can be difficult to answer--much discussion is usually required.)
> - Do the xarray developers have the time and expertise to implement
> and support such a feature?
>
> The first item, regarding ""scope,"" is being addressed now via this
> discussion. What are the pros and cons of attempting to add the new
> feature? Different people will have different opinions. Let's hear
> them out. A key question, as identified by @fmaussion, is whether the
> geotiff data model is compatible enough with the xarray data model
> enough to provide a full-featured writeable backend. In other words,
> can I write any arbitrary xarray dataset to geotiff and then read it
> back, with no loss of information. If the answer is ""no,"" then it
> will
> be hard to convince the xarray community that geotiff is a suitable
> candidate for a backend.
>
> If you feel strongly that we need the ability to not only directly
> read (as we can already with `open_rasterio`) but also directly
> *write* geotiff, you should lay out your arguments persuasively,
> taking into account not only the immediate impacts on your personal
> project but the impact on xarray as a whole. There may be good ways
> to
> achieve what you want without making any changes to xarray, i.e. by
> creating a small standalone package to transform geotiff to / from
> xarray (as in @Schlump's example); that option needs to be considered
> seriously.
>
> The second item (time) is a rather strong constraint: xarray is a
> volunteer effort. There are currently [369 open
> issues](https://github.com/pydata/xarray/issues) in xarray. Which
> ones
> should be the top priority? Will attempting to add a new feature lead
> to much more work down the line, in the form of unforeseen bugs?
>
> Ultimately, what happens in xarray is determined by the needs of the
> xarray developers themselves, who use xarray heavily in their daily
> science work. This may sound exclusive, but it is the opposite,
> because **anyone can become an xarray developer**. The reason we can
> read geotiffs today is because, one year ago, @fmaussion rolled up
> his
> sleeves and wrote the rasterio backend (#1260).
>
> That little number 1260 is a link to a merged [pull
> request](https://help.github.com/articles/about-pull-requests/) (aka
> ""PR""). A PR is much more powerful than a feature request; it is an
> actual implementation of the feature someone wishes to see in xarray.
> Anyone is free to make a PR to xarray, although before doing so, it
> is
> good to discuss the possible new feature via the issue tracker, as
> described in the [xarray contributing
> guide](http://xarray.pydata.org/en/stable/contributing.html). As a
> full time programmer in a lab dealing with geospatial data, **you
> yourself** are already a prime candidate to implement your desired
> feature! 😉
>
> As an example of how a new backend was incorporated into xarray, you
> can refer to #1905, in which @barronh implemented a backend for
> ""pseudo-netCDF"" a file format used by his research group. Skimming
> through that discussion will give you a good idea of some of the
> questions that arise in implementing new backend functionality.
>
> Apologies for the long digression into open-source politics. I
> thought it would be useful to clarify these things.
No need to apologize about a long digression into open-source politics,
and I fully understand and smack in the middle of that with at least 4
different projects. I also know about issue/commit numbers on
github/bitbucket/redmine/etc. NASA has formal rules about what can be
released and when. My last open-source project took 9 months to get the
software release authorized, but that was for an entire project new
code. For basic image I/O support I would not expect any problems, but
I have to get permission before releasing anything beyond snippets and
examples that do not include primary workflows. I will release as much
as I can back in the the public domain, but this starts to get
complicated as the scope grows.
I do not remember seeing anyone use the acronym PR for ""pull request""
before, so sorry for that confusion. I just could not guess it in the
context.
The argument for providing basic functionality for GeoTIFF's and
geotiffs, is that it is a common dataset used along side NetCDF and HDF.
I can, if you need me to, try to track down a stack of sites which
provide images in GeoTIFF's such as NASA's Giovanni, Digital Globe,
Planet Labs, just to name a couple off the top of my head. How many
folks here work in and around GIS folks?
I will have to post back later (probably to several separate issues) to
address several of the pointers raised above, but I have to get on to
fleshing some of this out.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379834837,https://api.github.com/repos/pydata/xarray/issues/2042,379834837,MDEyOklzc3VlQ29tbWVudDM3OTgzNDgzNw==,1197350,2018-04-09T17:43:11Z,2018-04-09T17:43:11Z,MEMBER,"> I really need to know what xarray can and is planning to do with tiff's
> so that I can not only use them but also document stuff for a dozen or
> more of my coworkers
@ebo, we are very glad to hear your input about how you might use xarray together with geotiff data. The majority of xarray developers are coming from a netCDF background, so this is somewhat new territory for us. It sounds like you have a real need for the computational tools that xarray provides. Engaging the geotiff community could potentially be very advantageous for xarray, since it could bring lots of new users. On the other hand, there are already lots of powerful tools in the geotiff space, and we have limited resources (i.e. time), so we need to be a bit conservative.
It would probably be useful to clarify how the decision making process works on things like this for open source projects. There is no xarray master plan that can provide a simple answer to your question of ""what xarray is planning to do with tiffs"". The main questions that have to be answered when deciding whether to add a big new feature are
- Does this feature make sense within the ""scope"" of the project? (Can be difficult to answer--much discussion is usually required.)
- Do the xarray developers have the time and expertise to implement and support such a feature?
The first item, regarding ""scope,"" is being addressed now via this discussion. What are the pros and cons of attempting to add the new feature? Different people will have different opinions. Let's hear them out. A key question, as identified by @fmaussion, is whether the geotiff data model is compatible enough with the xarray data model enough to provide a full-featured writeable backend. In other words, can I write any arbitrary xarray dataset to geotiff and then read it back, with no loss of information. If the answer is ""no,"" then it will be hard to convince the xarray community that geotiff is a suitable candidate for a backend.
If you feel strongly that we need the ability to not only directly read (as we can already with `open_rasterio`) but also directly *write* geotiff, you should lay out your arguments persuasively, taking into account not only the immediate impacts on your personal project but the impact on xarray as a whole. There may be good ways to achieve what you want without making any changes to xarray, i.e. by creating a small standalone package to transform geotiff to / from xarray (as in @Schlump's example); that option needs to be considered seriously.
The second item (time) is a rather strong constraint: xarray is a volunteer effort. There are currently [369 open issues](https://github.com/pydata/xarray/issues) in xarray. Which ones should be the top priority? Will attempting to add a new feature lead to much more work down the line, in the form of unforeseen bugs?
Ultimately, what happens in xarray is determined by the needs of the xarray developers themselves, who use xarray heavily in their daily science work. This may sound exclusive, but it is the opposite, because **anyone can become an xarray developer**. The reason we can read geotiffs today is because, one year ago, @fmaussion rolled up his sleeves and wrote the rasterio backend (#1260).
That little number 1260 is a link to a merged [pull request](https://help.github.com/articles/about-pull-requests/) (aka ""PR""). A PR is much more powerful than a feature request; it is an actual implementation of the feature someone wishes to see in xarray. Anyone is free to make a PR to xarray, although before doing so, it is good to discuss the possible new feature via the issue tracker, as described in the [xarray contributing guide](http://xarray.pydata.org/en/stable/contributing.html). As a full time programmer in a lab dealing with geospatial data, **you yourself** are already a prime candidate to implement your desired feature! 😉
As an example of how a new backend was incorporated into xarray, you can refer to #1905, in which @barronh implemented a backend for ""pseudo-netCDF"" a file format used by his research group. Skimming through that discussion will give you a good idea of some of the questions that arise in implementing new backend functionality.
Apologies for the long digression into open-source politics. I thought it would be useful to clarify these things.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379823407,https://api.github.com/repos/pydata/xarray/issues/2042,379823407,MDEyOklzc3VlQ29tbWVudDM3OTgyMzQwNw==,601025,2018-04-09T17:04:40Z,2018-04-09T17:04:40Z,NONE,"On Apr 9 2018 12:49 AM, Fabien Maussion wrote:
>> I do not care about the ``to_rasterio`` but I do care about a
>> ''to_tiff''
>
> Yes sorry, I meant ``to_tiff``
ah... got it.
>> If xarray has no way to output tiffs then I cannot use xarray.
>
> I'm not saying it shouldn't exist, I'm just asking whether it should
> be in the xarray codebase or elsewhere.
fair enough. I just need them to play well enough together that I can
read, process, and write a chunk/window at a time (whether that is with
a simple xr.compute() or something else).
> If you'd like to parse new attributes when opening the geotiff file
> this could be added easily. PRs are welcome!
what is a PR? Did you mean functionality request?
I'm still not clear where dask.array, xarray, rasterio, and pangeo
begin and end. I think I have posted an issue about extending the
metatdata/tags some place, but I am sure it is not as clear as it should
be, and for the life of me I am not sure where I posted that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379815753,https://api.github.com/repos/pydata/xarray/issues/2042,379815753,MDEyOklzc3VlQ29tbWVudDM3OTgxNTc1Mw==,601025,2018-04-09T16:39:28Z,2018-04-09T16:39:28Z,NONE,"On Apr 9 2018 9:22 AM, Ryan Abernathey wrote:
>> I'm already starting to think that these kind of domain specific
>> tools should exist in dedicated projects, not in the main xarray
>> codebase.
>
> 👍
I am perfectly fine with that stance, but I also think it is also
reasonable to ask/expect that if you provide a reader for some format
that you also provide writers for them -- or at least document that you
will not and why. Almost all of my current work is in geotiff format.
I have no choice, and many other people working in the geospatial domain
will be hamstrung without it. Sitting down the pipeline from me is 1/2
million archived images (there are actually closer to 2 million images,
but only 1/2 to 1 is associated with out current projects, and comprise
several petabytes of data).
I really need to know what xarray can and is planning to do with tiff's
so that I can not only use them but also document stuff for a dozen or
more of my coworkers (heck the next time we run the Python Bootcamp I
would probably offer to teach this). If you plan not to support it then
fine. I will not spend any more time with xarrays and focus on
dask.arrays or anything else that will work.
My question to you now is if supporting basic tiff I/O is in scope. If
so I can deal with all the rest of the rasterio/geospatial stuff outside
of xarray.
I will start fleshing out the stuff that Matthew Rocklin and Schlump
have provided.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379790588,https://api.github.com/repos/pydata/xarray/issues/2042,379790588,MDEyOklzc3VlQ29tbWVudDM3OTc5MDU4OA==,1197350,2018-04-09T15:22:25Z,2018-04-09T15:22:25Z,MEMBER,"> I'm already starting to think that these kind of domain specific tools should exist in dedicated projects, not in the main xarray codebase.
👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379651950,https://api.github.com/repos/pydata/xarray/issues/2042,379651950,MDEyOklzc3VlQ29tbWVudDM3OTY1MTk1MA==,10050469,2018-04-09T06:49:16Z,2018-04-09T06:49:16Z,MEMBER,"> I do not care about the ``to_rasterio`` but I do care about a ''to_tiff''
Yes sorry, I meant ``to_tiff``
> If xarray has no way to output tiffs then I cannot use xarray.
I'm not saying it shouldn't exist, I'm just asking whether it should be in the xarray codebase or elsewhere.
If you'd like to parse new attributes when opening the geotiff file this could be added easily. PRs are welcome!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379593639,https://api.github.com/repos/pydata/xarray/issues/2042,379593639,MDEyOklzc3VlQ29tbWVudDM3OTU5MzYzOQ==,601025,2018-04-09T00:03:49Z,2018-04-09T00:03:49Z,NONE,"On Apr 8 2018 11:54 AM, Schlump wrote:
>
> https://github.com/robintw/XArrayAndRasterio/blob/master/rasterio_to_xarray.py
Ahhh... Now I understand Fabien Maussion'd comment about to_rasterio.
I read the posts out of order. I will see if this does the job for me
(I will likely have to extend it a little, but I think this is a great
start).
EBo --
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379593118,https://api.github.com/repos/pydata/xarray/issues/2042,379593118,MDEyOklzc3VlQ29tbWVudDM3OTU5MzExOA==,601025,2018-04-08T23:57:20Z,2018-04-08T23:57:20Z,NONE,"On Apr 8 2018 12:45 PM, Fabien Maussion wrote:
>> if the profile and tags were propagated through open_rasterio, then
>> the second open would not be necessary and would be generally useful.
>
> We have been adding new attributes like this recently
> (https://github.com/pydata/xarray/pull/1583 and
> https://github.com/pydata/xarray/pull/1740), so I don't see much
> trouble in adding a few more. Note that the rasterio object is
> available via the (undocumented) ``_file_obj`` attribute. So a quick
> workaround for you in the mean time would be to access the info you
> need directly via this object.
>
> As for the ``to_rasterio`` method, I'm currently against it. I'm
> already starting to think that these kind of domain specific tools
> should exist in dedicated projects, not in the main xarray codebase.
> For rasterio in particular, it turns out that the geotiff/GDAL data
> model is fairly different from the xarray/NetCDF model. The rasterio
> folks have also shown only limited interest in our endeavor
> (https://github.com/mapbox/rasterio/issues/920), which is
> understandable. I don't have a strong opinion though, and I am
> curious
> if the @pydata/xarray crew sees it differently.
I do not care about the ``to_rasterio`` but I do care about a
''to_tiff'' (even if I have to do all the geospatial stuff outside of
xarray as long as I can output the image data portion of the tiff via
xarray). I also do not overly care if the xarray interface is
significantly different from the rasterio/GDAL API (however someone will
have to document the differences so that it does not continually trip
people like me up -- I should be able to help a little with this once it
gets working). Basically however it is handled I have to be able to
read a GeoTIFF, process, and write back out to a GeoTIFF. If xarray has
no way to output tiffs then I cannot use xarray.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379572622,https://api.github.com/repos/pydata/xarray/issues/2042,379572622,MDEyOklzc3VlQ29tbWVudDM3OTU3MjYyMg==,10050469,2018-04-08T18:45:40Z,2018-04-08T18:45:40Z,MEMBER,"> if the profile and tags were propagated through open_rasterio, then the second open would not be necessary and would be generally useful.
We have been adding new attributes like this recently (https://github.com/pydata/xarray/pull/1583 and https://github.com/pydata/xarray/pull/1740), so I don't see much trouble in adding a few more. Note that the rasterio object is available via the (undocumented) ``_file_obj`` attribute. So a quick workaround for you in the mean time would be to access the info you need directly via this object.
As for the ``to_rasterio`` method, I'm currently against it. I'm already starting to think that these kind of domain specific tools should exist in dedicated projects, not in the main xarray codebase. For rasterio in particular, it turns out that the geotiff/GDAL data model is fairly different from the xarray/NetCDF model. The rasterio folks have also shown only limited interest in our endeavor (https://github.com/mapbox/rasterio/issues/920), which is understandable. I don't have a strong opinion though, and I am curious if the @pydata/xarray crew sees it differently.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596
https://github.com/pydata/xarray/issues/2042#issuecomment-379568862,https://api.github.com/repos/pydata/xarray/issues/2042,379568862,MDEyOklzc3VlQ29tbWVudDM3OTU2ODg2Mg==,8473452,2018-04-08T17:54:37Z,2018-04-08T17:54:37Z,NONE,"https://github.com/robintw/XArrayAndRasterio/blob/master/rasterio_to_xarray.py
","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 2, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312203596