Add summary methods to Dataset object. For example, it would be great if you could summarize a entire dataset in a single line.

(1) Mean of all variables in dataset.

python mean_ds = ds.mean()

(2) Mean of all variables in dataset along a dimension:

python time_mean_ds = ds.mean(dim='time')

In the case where a dimension is specified and there are variables that don't use that dimension, I'd imagine you would just pass that variable through unchanged.

Related to #122.

Requirements: - [x] Better documentation: - [x] Tutorial introduces DataArray before and independently of Dataset - [x] Revise README to emphasize that xray generalizes pandas to N-dimensions - [x] New FAQ section to clarify relationship of xray to pandas, Iris and CDAT (#112) - [x] Update What's New - [x] More consistent names: - [x] dimensions -> dims and coordinates -> coords (#190) - [x] noncoordinates -> noncoords - [x] Lay groundwork for non-index coordinates (#197): - [x] Require specifying attrs with a keyword argument in Dataset.__init__ (to make room for coords) - [x] Don't allow indexing array.coords[0] - [x] Remove the to argument from Dataset.apply (it will be much less clearly useful when we have non-index coords) - [x] Add warning in the docs (clarify that "linked dataset variables" are going away)

Nice to have: - [x] Support modifying DataArray dimensions/coordinates in place (#180) - [ ] Automatic alignment in mathematical operations (#184) - [ ] Revised interface for CF encoding/decoding (#155, #175)

Should resolve all remaining items in #183.

This makes the xray API a little more similar to pandas, which makes heavy use of name = None for objects that can but don't always have names like Series and Index.

It will be a particular useful option to have around when we add a direct constructor for DataArray objects (#115). For now, arrays will probably only end up being named None if they are the result of some mathematical operation where the name could be ambiguous.

Right now, we use ds.attrs but the keyword argument is still attributes. We should resolve this inconsistency.

We also use dimensions and coordinates instead of the natural abbreviations dims and coords (although dims is used in the Variable constructor). Should we switch to the abbreviated versions for consistency with attrs?

Note that I switched to attrs in part because of its use in other packages (h5py, pytables and blz). There is not as clear precedent for what to call dimensions and coordinates.

Cleaned up a few cases where attributes was used instead of attrs in function signatures.

Fixes: #190 - [x] Switch names in xray itself - [x] Switch names in tests - [x] Switch names in documentation

If 'x' was a scalar variable in a dataset and you set a new variable with 'x' as a dimension, you could end up with a broken Dataset object.

This is why mutating state is a bad idea.

Now Dataset.getitem raises a KeyError when it can't find a variable.

This PR changes the DataArray representation so that it displays more of the information associated with a data array: - "Coordinates" are indicated by their name and the repr of the corresponding pandas.Index object (to indicate how they are used as indices). - "Linked" dataset variables are also listed. - These are other variables in the dataset associated with a DataArray which are also indexed along with the DataArray. - They accessible from the dataset attribute or by indexing the data array with a string. - Perhaps their most convenient aspect is that they enable groupby operations by name for DataArray objets. - This is an admitedly somewhat confusing (though convenient) notion that I am considering removing, but we if we don't remove them we should certainly expose their existence more clearly, given the potential benefits in expressiveness and costs in performance.

Questions to resolve: - Is "Linked dataset variables" the best name for these? - Perhaps it would be useful to show more information about these linked variables, such as their dimensions and/or shape?

Examples of the new repr are on nbviewer: http://nbviewer.ipython.org/gist/shoyer/94936e5b71613683d95a

Related: #164

Fixes #182

Allowing boolean arrays for .loc would make xray more consistent with pandas: http://pandas.pydata.org/pandas-docs/stable/indexing.html#different-choices-for-indexing-loc-iloc-and-ix

There is basically no ambiguity since True and False are more or less useless coordinate labels

With this patch, it is possible to perform the following operations: - data_array.name = 'foo' - data_array.coordinates = ... - data_array.coordinates[0] = ... - data_array.coordinates['x'] = ... - dataset.coordinates['x'] = ... - dataset.rename(..., inplace=True)

It is no longer possible to set data_array.variable = ...., which was technically part of the public API but I would guess unused.

The key thing is to (shallow) copy the underlying dataset before making any changes. The shallow copy is probably OK here -- modifying dimensions and coordinates is unlikely to be done in any inner loops.

Dataset reduce methods (#131) suggested to me that it would be nice to support applying functions which map over all data arrays in a dataset. The signature of Dataset.apply could be modeled after GroupBy.apply and the implementation would be similar to #137 (but simpler).

For example, I should be able to write ds.apply(np.mean).

Note: It's still worth having #137 as a separate implementation because it can do some additional validation for dimensions and skip variables where the aggregation doesn't make sense.

Fixes #140

With this PR, it is possible to close the data store from which a dataset was loaded via ds.close() or automatically when a dataset is used with a context manager:

python with xray.open_dataset('data.nc') as ds: ...

The ability to cleanly close files opened from disk is pretty essential -- we probably should have had this a while ago. It should not be necessary to use the low-level/unstable datastore API to get this functionality.

Implementation question: With this current implementation, calling ds.close() on (and using a context manager with) a dataset not linked to any file objects is a no-op. Should we raise an exception instead? Something like IOError('no file object to close')?

CC @ToddSmall

A first attempt at implementing Dataset reduction methods.

131

In #161, I renamed xray.Coordinate to xray.Index.

To better distinguish xray's Index from pandas, let's call it XIndex instead.

For users of pandas, the xray interface would be more obvious if we referred to what we currently call "coordinates" as "indices."

This would entail renaming the coordinates property to indices, xray.Coordinate to xray.Index and the xray.Coordinate.as_index property to as_pandas_index (all with deprecation warnings).

Possible downsides: 1. The xray data model would be less obvious to people familiar with the NetCDF. 2. There is some potential for confusion between xray.Index and pandas.Index: - The only real difference is that xray's Index is a xray.Variable object, and thus is dimension aware and has attributes. - In principle, xray.Index should have all the necessary properties to act like an index (or rather, it already has most of these properties and should get the rest). - Unfortunately, pandas doesn't accept non-pandas.Index objects as indices, nor will it properly convert an xray.Index into a pandas.Index.

At PyData SV, @mrocklin suggested that by default, array broadcasting should fall back on numpy's shape based broadcasting. This would also simplify directly constructing DataArray objects (#115).

The trick will be to make this work with xray's internals, which currently assume that dimensions are always named by strings.

This change mainly involves an alternative import of OrderedDict, modified dict and set comprehensions, and using unittest2 for testing.

This should fix a bug @mgarvert encountered with concatenating variables with different array attributes.

In the process of fixing this issue, I encountered and fixed another bug with utils.remove_incompatible_items.

According to the toy benchmark below, this shaves off between 20% (diff-indexes) to 40% (same-indexes) of xray's overhead for array math:

import numpy as np import xray x = np.random.randn(1000, 1000) y = np.random.randn(1000, 1000) dx = xray.DataArray(x) dy = xray.DataArray(y) %timeit x + x # raw-numpy %timeit dx + dx # same-indexes %timeit dx + dy # diff-indexes

You can now do ds.groupby('time.month').mean() to apply the mean over all groups and variables in a dataset.

It is not optimized like the DataArray.groupby summary methods but it should work.

Thanks @jhamman for laying the groundwork for this!

``` import cPickle as pickle import xray import numpy as np import pandas as pd

foo_values = np.random.RandomState(0).rand(3,4) times = pd.date_range('2001-02-03', periods=3) ds = xray.Dataset({'time': ('time', times), 'foo': (['time', 'space'], foo_values)})

with open('mypickle.pkl', 'w') as f: pickle.dump(ds, f)

with open('mypickle.pkl') as f: myds = pickle.load(f)

myds ```

This code results in: <repr(<xray.dataset.Dataset at 0x7f95a3290d90>) failed: AttributeError: mapping>

Fixes #142 Fixes #148

All existing code should still work but issue a FutureWarning if any of the old names are used.

Full list of updates:

| Old | New | | --- | --- | | Coordinate | Index | | coordinates | indexes | | noncoordinates | nonindexes | | indexed | isel | | labeled | sel | | select | select_vars | | unselect | drop_vars |

Most of these are both Dataset and DataArray methods/properties.

I'd like to rename the Dataset/DataArray methods labeled and indexed, so that they are more obviously variants on a theme, similarly to how pandas distinguishes between methods .loc and .iloc (and .at/.iat, etc.). Some options include: 1. Rename indexed to ilabeled. 2. Rename indexed/labeled to isel/sel.

I like option 2 (particularly because it's shorter), but to avoid confusion with the select method, we would need to also rename select/unselect to something else. I would suggest select_vars and drop_vars.

Almost ready, except for failing tests on Python 3.

This recently added some complexity for a very nominal speed benefit. And it appears that it breaks joblib serialization, somehow (even though pickle works). So for now, revert it -- and consider filing a joblib bug if we can narrow it down.

Fixes #156, #157

To elaborate on the changes: 1. When an array is indexed, its encoding will be reset. This takes care of the invalid chunksize issue. More generally, this seems like the right choice because it's not clear that the right encoding will be the same after slicing an array, anyways. 2. If an array has encoding['dtype'] = np.dtype('S1') (e.g., it was originally encoded in characters), it will be stacked up to be saved as a character array, even if it's being saved to a NetCDF4 file. Previously, the array would be cast to 'S1' without stacking, which would result in silent loss of data.

Fixes #115.

Related: #116, #117.

Note: a remaining major task will be to rewrite/reorganize the docs to introduce DataArray first, entirely independently of Dataset. This will make it easier for new users to figure out how to get started with xray, since DataArray is much simpler.

It shouldn't be necessary to put arrays in a Dataset to make a DataArray.

See also: https://github.com/xray/xray/issues/85#issuecomment-38875079

This was causing an error in the following situation:

ds = xray.Dataset() ds['time'] = ('time', [np.datetime64('2001-05-01') for i in range(5)]) ds['variable'] = ('time', np.arange(5.)) ds.to_netcdf('test.nc') ds = xray.open_dataset('./test.nc') ss = ds.indexed(time=slice(0, 2)) ss.dumps()

Thanks @shoyer for the fix.