I think rioxarray is now the recommended solution.

gdal can read/write windows:

I'm aware. See this doc listed above for rasterio: https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html#writing

Background here is that rasterio more-or-less wraps around GDAL, but with interfaces that are somewhat more idiomatic to this community.

I wonder how you got that to work other than setting up a slave read process that handles all reads.

We've run into these issues before as well. Typically we handle them with locks of various types.

My first attempt would be to use this API: https://rasterio.readthedocs.io/en/latest/topics/windowed-rw.html#writing

My impression is that there are will be some (significant) development challenges

If you're able to expand on this that would be welcome.

that perhaps only supports a few rasterio file formats

My hope would be that rasterio/GDAL would handle the many-file-format issue for us if they support writing in chunks. I also lack experience here though.

I agree, it would be nice to have the to_rasterio functionality. My impression is that there are will be some (significant) development challenges, particularly related to rasterio's support for many file formats, but those can probably be sorted out by a committed developer or by partnering with a rasterio developer.

This is a bit outside my area of expertise but I imagine it will be useful to see a prototype, that perhaps only supports a few rasterio file formats, before diving into the xarray backends to implement this full bore.

When writing https://github.com/pydata/xarray/issues/2093 I came across this issue and thought I'd weigh in.

The GIS community seems like a fairly close neighbor to XArray's current community. Some API compatibility here might be a good to expand the community. I definitely agree that GeoTiff does not implement the full XArray model, but it might be useful to support the subset of datasets that do, just so that round-trip operations can occur. For example, it might be nice if the following worked:

```python dset = xr.open_rasterio(...)

do modest modifications to dest

dset.to_rasterio(...) ```

My hope would be that the rasterio/GDAL data model would be consistent enough so that we could detect and err early if the dataset was not well-formed.

I really need to know what xarray can and is planning to do with tiff's so that I can not only use them but also document stuff for a dozen or more of my coworkers

@ebo, we are very glad to hear your input about how you might use xarray together with geotiff data. The majority of xarray developers are coming from a netCDF background, so this is somewhat new territory for us. It sounds like you have a real need for the computational tools that xarray provides. Engaging the geotiff community could potentially be very advantageous for xarray, since it could bring lots of new users. On the other hand, there are already lots of powerful tools in the geotiff space, and we have limited resources (i.e. time), so we need to be a bit conservative.

It would probably be useful to clarify how the decision making process works on things like this for open source projects. There is no xarray master plan that can provide a simple answer to your question of "what xarray is planning to do with tiffs". The main questions that have to be answered when deciding whether to add a big new feature are - Does this feature make sense within the "scope" of the project? (Can be difficult to answer--much discussion is usually required.) - Do the xarray developers have the time and expertise to implement and support such a feature?

The first item, regarding "scope," is being addressed now via this discussion. What are the pros and cons of attempting to add the new feature? Different people will have different opinions. Let's hear them out. A key question, as identified by @fmaussion, is whether the geotiff data model is compatible enough with the xarray data model enough to provide a full-featured writeable backend. In other words, can I write any arbitrary xarray dataset to geotiff and then read it back, with no loss of information. If the answer is "no," then it will be hard to convince the xarray community that geotiff is a suitable candidate for a backend.

If you feel strongly that we need the ability to not only directly read (as we can already with open_rasterio) but also directly write geotiff, you should lay out your arguments persuasively, taking into account not only the immediate impacts on your personal project but the impact on xarray as a whole. There may be good ways to achieve what you want without making any changes to xarray, i.e. by creating a small standalone package to transform geotiff to / from xarray (as in @Schlump's example); that option needs to be considered seriously.

The second item (time) is a rather strong constraint: xarray is a volunteer effort. There are currently 369 open issues in xarray. Which ones should be the top priority? Will attempting to add a new feature lead to much more work down the line, in the form of unforeseen bugs?

Ultimately, what happens in xarray is determined by the needs of the xarray developers themselves, who use xarray heavily in their daily science work. This may sound exclusive, but it is the opposite, because anyone can become an xarray developer. The reason we can read geotiffs today is because, one year ago, @fmaussion rolled up his sleeves and wrote the rasterio backend (#1260).

That little number 1260 is a link to a merged pull request (aka "PR"). A PR is much more powerful than a feature request; it is an actual implementation of the feature someone wishes to see in xarray. Anyone is free to make a PR to xarray, although before doing so, it is good to discuss the possible new feature via the issue tracker, as described in the xarray contributing guide. As a full time programmer in a lab dealing with geospatial data, you yourself are already a prime candidate to implement your desired feature! 😉

As an example of how a new backend was incorporated into xarray, you can refer to #1905, in which @barronh implemented a backend for "pseudo-netCDF" a file format used by his research group. Skimming through that discussion will give you a good idea of some of the questions that arise in implementing new backend functionality.

Apologies for the long digression into open-source politics. I thought it would be useful to clarify these things.

I'm already starting to think that these kind of domain specific tools should exist in dedicated projects, not in the main xarray codebase.

👍

I do not care about the to_rasterio but I do care about a ''to_tiff''

Yes sorry, I meant to_tiff

If xarray has no way to output tiffs then I cannot use xarray.

I'm not saying it shouldn't exist, I'm just asking whether it should be in the xarray codebase or elsewhere.

If you'd like to parse new attributes when opening the geotiff file this could be added easily. PRs are welcome!

if the profile and tags were propagated through open_rasterio, then the second open would not be necessary and would be generally useful.

We have been adding new attributes like this recently (https://github.com/pydata/xarray/pull/1583 and https://github.com/pydata/xarray/pull/1740), so I don't see much trouble in adding a few more. Note that the rasterio object is available via the (undocumented) _file_obj attribute. So a quick workaround for you in the mean time would be to access the info you need directly via this object.

As for the to_rasterio method, I'm currently against it. I'm already starting to think that these kind of domain specific tools should exist in dedicated projects, not in the main xarray codebase. For rasterio in particular, it turns out that the geotiff/GDAL data model is fairly different from the xarray/NetCDF model. The rasterio folks have also shown only limited interest in our endeavor (https://github.com/mapbox/rasterio/issues/920), which is understandable. I don't have a strong opinion though, and I am curious if the @pydata/xarray crew sees it differently.