issues

Table actions

2 rows where state = "closed" and user = 761677 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 1

state 1

  • closed · 2 ✖

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1412846601 I_kwDOAMm_X85UNlQJ 7180 open_mfdataset for merging of netCDF files with different time values jorgsk 761677 closed 0     2 2022-10-18T09:06:14Z 2022-10-19T08:47:25Z 2022-10-19T08:47:25Z NONE      

What is your issue?

In xarray version 0.16 I could combinine multiple netCDF files with different values on the time coordinate along a new dimension in a straight forward way:

``` def rename_time(ds: xr.Dataset) -> xr.Dataset: return ds.rename_dims({'time': 'sim_time'})

ds = xr.open_mfdataset(ncfiles, concat_dim='simulation', preprocess=rename_time,
                                       combine='nested', coords=['time'])

``` The merged dataset got a new time dimension called "sim_time" which had dimensions of simulation and time, so a 2D array.:

When upgrading xarray to 2022.9.0, this code no longer works:

ds_allsims = xr.open_mfdataset(coarser_files, concat_dim='simulation', preprocess=rename_time, File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/backends/api.py", line 1003, in open_mfdataset combined = _nested_combine( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/combine.py", line 365, in _nested_combine combined = _combine_nd( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/combine.py", line 239, in _combine_nd combined_ids = _combine_all_along_first_dim( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/combine.py", line 275, in _combine_all_along_first_dim new_combined_ids[new_id] = _combine_1d( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/combine.py", line 298, in _combine_1d combined = concat( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/concat.py", line 243, in concat return _dataset_concat( File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/concat.py", line 569, in _dataset_concat combined_idx = indexes[0].concat(indexes, dim, positions) File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/indexes.py", line 327, in concat new_pd_index = cls._concat_indexes(indexes, dim, positions) File "/home/jorgens/mambaforge/envs/era_acute_post_processor/lib/python3.10/site-packages/xarray/core/indexes.py", line 307, in _concat_indexes raise ValueError( ValueError: Cannot concatenate along dimension 'simulation' indexes with dimensions: 'sim_time'

Looking around, I found this answer on stack overflow for a similar question: https://stackoverflow.com/a/55500434 I had to modify the preprocessor in the answer to make it work: def rename_time(ds: xr.Dataset) -> xr.Dataset: ds['sim_time'] = ds.time ds = ds.drop('time').rename({'time': 'ntime'}) return ds However, this creates a dataset structure which is less useful in my case: Here, time is lost as a coordinate, and I have to explicitly work the new "sim_time" variable to make sense of my data variables.

This is not impossible to work with, but I preferred the previous output.

My questions are: 1) is the change in behavior in xarray as intended and 2) is there a way for me to get the old dataset structure out of a new version of xarray?

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7180/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue
255897797 MDU6SXNzdWUyNTU4OTc3OTc= 1559 Subsetting not reducing memory consumption with mfdataset on Windows jorgsk 761677 closed 0     5 2017-09-07T10:58:30Z 2017-09-14T09:51:51Z 2017-09-14T09:51:51Z NONE      

On Windows, with the following code I run out of memory: d = xr.open_mfdataset(filepath) sed = d.sediment_mass_per_unit_area.isel(time=-1) sed.load() d.close() but when using "open_dataset" I load the data instantly: d = xr.open_dataset(filepath) sed = d.sediment_mass_per_unit_area.isel(time=-1) sed.load() d.close() On linux both methods use minimal memory. The target file is 1.5 GB, and the target variable is 1460x862x900 floats. Filepath is a single file, not multiple files. Xarray 0.9.6 used on both systems, installed with conda.

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/1559/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);