In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

On Jun 18 2018 4:03 PM, Fabien Maussion wrote:

Has a default GeoTIFF chunk been implemented?

No, unfortunately.

ok. Maybe the overall chunking issue has been sorted. I will try to look into this and see what is working now related to this issue.

Has a default GeoTIFF chunk been implemented?

No, unfortunately.

one of the issues related to this has been closed. Has a default GeoTIFF chunk been implemented?

that would definitely work for me.

On May 7 2018 6:43 AM, Zac Hatfield-Dodds wrote:

With the benefit of almost a year's worth of procrastination, I think the best approach is to take the heuristics from #1440, but only support chunks=True - if a decent default heuristic isn't good enough, the user can specify exact chunks.

The underlying logic for this issue would be identical to that of

1440, so supporting both is "just" a matter of plumbing it in

correctly.

With the benefit of almost a year's worth of procrastination, I think the best approach is to take the heuristics from #1440, but only support chunks=True - if a decent default heuristic isn't good enough, the user can specify exact chunks.

The underlying logic for this issue would be identical to that of #1440, so supporting both is "just" a matter of plumbing it in correctly.

Most of the standard internal chunked (or what I believe to be called 'tiled' by the GIS community) is 256x256 (see: http://www.gdal.org/frmt_gtiff.html TILED=YES BLOCKXSIZE=n and BLOCKYSIZE=n). This is used when viewing images within a given region of interest or window. You can really tell the difference in speed between the tiled and stripped images (which has a blocksize 1x<width>).

@mrocklin, I agree that we might want to aggregate some number of them, but we would need to get some automation up front and sort out how we want to determine the expansion. Adding to the #1440 discussion mentioned, there will likely be advantage in increasing the block sizes in given directions.

1440 is related but more focused on netcdf.

My guess is that geotiff chunks will be much smaller than is ideal for dask.array. We might want to expand those chunk sizes by some multiple.

There is precedent for auto-aligning dask chunks with the underlying dataset chunks. This is what we do with the auto_chunk argument in open_zarr: http://xarray.pydata.org/en/latest/generated/xarray.open_zarr.html#xarray.open_zarr

On Mon, Apr 30, 2018 at 12:21 PM, Matthew Rocklin notifications@github.com wrote:

Given a tiled GeoTIFF image I'm looking for the best practice in reading it as a chunked dataset. I did this in this notebook https://gist.github.com/mrocklin/3df315e93d4bdeccf76db93caca2a9bd by first opening the file with rasterio, looking at the block sizes, and then using those to inform the argument to chunks= in xarray.open_rasterio. This works, but is somewhat cumbersome because I also had to dive into the rasterio API. Do we want to provide defaults here?

In dask.array every time this has come up we've always shot it down, automatic chunking is error prone and hard to do well. However in these cases the object we're being given usually also conveys its chunking in a way that matches how dask.array thinks about it, so the extra cognitive load on the user has been somewhat low. Rasterio's model and API feel much more foreign to me though than a project like NetCDF or H5Py. I find myself wanting a chunks=True or chunks='100MB' option.

Thoughts on this? Is this in-scope? If so then what is the right API and what is the right policy for how to make xarray/dask.array chunks larger than GeoTIFF chunks?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2093, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJFJoFj8_VGvsJ5oZQD4WTgW8Xx3lSyks5ttzoKgaJpZM4Ts2Hm .