issue_comments

Okay thanks, closing now. We can always reopen this if necessary.

Yes, I think it should be closed. There are better ways to accomplish the desired goals.

Specifically, allowing the user to pass kwargs to concat via open_mfdataset would be useful.

@rabernat - I'm just catching up on this issue. Is you last comment indicating that we should close this PR?

Sounds good to me!

Since the expensive part (for me) is actually reading all the coordinates, I'm not sure that this PR makes sense any more.

The same thing I am going for here could probably be accomplished by allowing the user to pass join='exact' via open_mfdataset. A related optimization would be to allow the user to pass coords='minimal' (or other concat coords options) via open_mfdataset.

For really big datasets, I think we will want to go the NCML approach, generating the xarray metadata as a pre-processing step. Then we could add a function like open_ncml_dataset to xarray which would parse this metadata and construct the dataset in a more efficient way (i.e. not reading redundant coordinates).

Well, we could potentially write a fast path constructor for loading multiple netcdf files that avoids open_dataset. We just need another way to specify the schema, e.g., using NCML. On Fri, May 19, 2017 at 10:53 AM Ryan Abernathey notifications@github.com wrote:

As I think about this further, I realize it might be futile to avoid reading the dimensions from all the files. This is a basic part of how open_dataset works.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1413#issuecomment-302724756, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1pDsz3dD_xmfKFgg-WYk3LBCP1raks5r7az9gaJpZM4NeYj- .

As I think about this further, I realize it might be futile to avoid reading the dimensions from all the files. This is a basic part of how open_dataset works.

What is xr.align(..., join='exact') supposed to do?

It verifies that all dimensions have the same length, and coordinates along all dimensions (used for indexing) also match. Unlike the normal version of align, it doesn't do any indexing -- the outputs are always the same as the inputs.

It does not check that the necessary dimensions and variables exist in all datasets. But we should do that as part of the logic in concat anyways, since the xarray data model always requires knowing variables and their dimensions.

Given a collection of datasets, how do I know if setting prealigned=True will work?

I guess we would want to check that (a) the necessary variables and dimensions exist in all datasets and (b) the dimensions have the same length. We would want to bypass the actual reading of the indices. I agree it would be nicer to subsume this logic into align.

What is xr.align(..., join='exact') supposed to do?

What happens if things go wrong?

I can add more careful checks once we sort out the align question.

This enhancement makes a lot of sense to me.

Two things worth considering:

1. Given a collection of datasets, how do I know if setting prealigned=True will work? This is where my PR adding xr.align(..., join='exact') could help (I can finish that up). Maybe it's worth adding xr.is_aligned or something similar.
2. What happens if things go wrong? It's okay if the behavior is undefined (or could give wrong results) but we should document that. Ideally we should raise sensible errors at some later time, e.g., when the dask arrays are computed. This might or might not be possible to do efficiently with dask, if the result of all the equality checks are consolidated and added into the dask graphs of the results.

Let me expand on what this does.

Many netCDF datasets consist of multiple files with identical coordinates, except for one (e.g. time). With xarray we can open these datasets with open_mfdataset, which calls concat on the list of individual dataset objects. concat calls align, which loads all of the dimension indices (and, optionally, non-dimension coordinates) from each file and checks them for consistency / alignment.

This align step is potentially quite expensive for big collections of files with large indices. For example, an unstructured grid or particle-based dataset would just have a single dimension coordinate, with the same length as the data variables. If the user knows that the datasets are already aligned, this PR enables the alignment step to be skipped by passing the argument prealigned=True to concat. My goal is to avoid touching the disk as much as possible.

This PR is a draft in progress. I still need to propagate the prealigned argument up to auto_combine and open_mfdataset.

An alternative API would be to add another option to the coords keywork, i.e. coords='prealigned'.

Feedback welcome.