issue_comments

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

@darothen That sounds great!

I think we should be clearer. The issue that @NicWayand and I are highlighting is the coercing observational data, which often comes with some fairly heinous formatting issues, into an xarray format. The stacking of these data along a new dimension is usually the last step in this process, and one that can be frustrating. An example of this in practice can be found in this notebook (please be forgiving, it is one of the first things I ever wrote in python).

https://github.com/klapo/CalRad/blob/master/CR.SurfObs.DataIngest.xray.ipynb

The data flow looks like this: - read the csv summarizing each station - read data from one set of stations using pandas - clean the data - assign the data in a pandas DataFrame to a dictionary of DataFrames - rinse and repeat for the other set of data - concat the dictionary of DataFrames into a single DataFrame - convert to an xarray DataSet

This example is a little ludicrous because I didn't know what I was doing, but I think that's the point. There is a lot of ambiguity on which tools to use at what point. Concatenating a dictionary of DataFrames into a single DataFrame and then converting to a DataSet was the only solution I could get to work, after a lot of trial and error, for putting these data in an xarray DataSet.

@klapo! Great to see you here!

Happy to iterate with you on documenting this functionality. For reference, I wrote a package for my dissertation work to help automate the task of constructing multi-dimensional Datasets which include dimensions corresponding to experimental/ensemble factors. One of my on-going projects is to actually fully abstract this (I have a not-uploaded branch of the project which tries to build the notion of an "EnsembleDataset", which has the same relationship to a Dataset that an pandas Panel used to have to a DataFrame).

Also, just a small thing in the docs for concat

The example includes this snippet xr.concat([arr[0], arr[1]], pd.Index([-90, -100], name='new_dim')) but as far as I can tell, name is not an argument accepted by concat

I have an example that I just struggled through that might be relevant to this idea. I'm running a point model using some arbitrary number of experiments (for the below example there are 28 experiments). Each experiment is opened and then stored in a dictionary resultsDict. The below excerpt extracts all of my scalar variables, concatenates them along an experiment dimension, and finally combines all scalar variables into a DataSet. I often find myself struggling to combine data (for instance meteorological stations) into a DataSet and I can never remember how to use merge and/or concat.

``` resultsDataSet = xr.Dataset() for k in scalar_data_vars: if not 'scalar' in k: continue

# Assign scalar value to a dataArray
darray = xr.concat([resultsDict[scen][scalar_data_vars[0]] for scen in resultsDict], dim='expID')
# Remove hru dimension, as it is unused
darray = darray.squeeze('hru')

resultsDataSet[k] = darray

print(resultsDataSet) which yields <xarray.Dataset> Dimensions: (expID: 28, time: 8041) Coordinates: * time (time) datetime64[ns] 2008-10-01 ... hru int32 1 Dimensions without coordinates: expID Data variables: scalarRainPlusMelt (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSWE (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSnowSublimation (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarInfiltration (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSurfaceRunoff (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSurfaceTemp (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSenHeatTotal (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarLatHeatTotal (expID, time) float64 -9.999e+03 -9.999e+03 ... scalarSnowDepth (expID, time) float64 -9.999e+03 -9.999e+03 ... ```

And here is a helper function that can do this more generally, which I wrote a while back.

``` def combinevars(ds_in, dat_vars, new_dim_name='new_dim', combinevarname='new_var'): ds_out = xr.Dataset() ds_out = xr.concat([ds_in[dv] for dv in dat_vars], dim='new_dim') ds_out = ds_out.rename({'new_dim': new_dim_name}) ds_out.coords[new_dim_name] = dat_vars ds_out.name = combinevarname

return ds_out

```