home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 523960862

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2064#issuecomment-523960862 https://api.github.com/repos/pydata/xarray/issues/2064 523960862 MDEyOklzc3VlQ29tbWVudDUyMzk2MDg2Mg== 2448579 2019-08-22T15:42:10Z 2019-08-22T15:42:10Z MEMBER

I have a draft solution in #3239. It adds a new mode called "sensible" that acts like "all" when the concat dimension doesn't exist in the dataset and acts like "minimal" when the dimension is present. We can decide whether this is the right way i.e. add a new mode but the more fundamental problem is below.

The issue is dealing with variables that should not be concatentated in "minimal" mode (e.g. time-invariant non dim coords when concatenating in time). In this case, we want to skip the equality checks in _calc_concat_over. This is a common reason for poor open_mfdataset performance.

I thought the clean way to do this would be to add the compat kwarg to concat and then add compat='override' since the current behaviour is effectively compat='equals'.

However, merge takes compat too and concat and merge support different compat arguments at present. This makes it complicated to easily thread compat down from combine or open_mfdataset without adding concat_compat and merge_compat which is silly.

So do we want to support all the other compat modes in concat? Things like broadcast_equals or no_conflicts are funny because they're basically merge operations and it means concat acts like both stack, concat and merge. OTOH if you have a set of variables with the same name from different datasets and you want to pick one of those (i.e. no concatenation), then you're basically doing merge anyway. This would require some refactoring since concat assumes the first dataset is a template for the rest.

@shoyer What do you think?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  314764258
Powered by Datasette · Queries took 1.938ms · About: xarray-datasette