This would be the complement of the DataArray constructor, converting an xray.DataArray into a 1D series, 2D DataFrame or 3D panel, whichever is appropriate.

to_pandas would also makes sense for Dataset, if it could convert 0d datasets to series, e.g., pd.Series({k: v.item() for k, v in ds.items()}) (there is currently no direct way to do this), and revert to to_dataframe for higher dimensional input. - [x] DataArray method - [ ] Dataset method

I'd like to create an empty DataArray, i.e., one with only NA values. The docstring of DataArray says that data=None is allowed, if a dataset argument is provided. However, the docstring doesn't say anything about a dataset argument. 1. I think there's a bug in the docstring 2. I'd like to pass data=None and get a DataArray with the coords/dims set up properly (as defined by the coords and dims kwargs), but with a values array of NA.

For dask: - [x] default threadpool for dask.array - [x] fix indexing bugs for dask.array - [x] make a decision on (and if necessary implement) renaming "block" to "chunk" - [x] fix repeated use of da.insert

For xray: - [x] update xray for the updated dask (https://github.com/xray/xray/pull/395) - [x] figure out how to handle caching with the .load() method on dask arrays - [x] cleanup the xray documentation on dask array. - [x] write an introductory blog post

Things we can add in an incremental release: - make non-aggregating grouped operations more useable - automatic lazy apply for grouped operations on xray objects

CC @mrocklin

sel() now supports the method parameter, which works like the paramter of the same name on reindex(). It provides a simple interface for doing nearest- neighbor interpolation:

``` In [12]: ds.sel(x=1.1, method='nearest') Out[12]: <xray.Dataset> Dimensions: () Coordinates: x int64 1 Data variables: y int64 2

In [13]: ds.sel(x=[1.1, 2.1], method='pad') Out[13]: <xray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 1 2 Data variables: y (x) int64 2 3 ```

Example usage:

``` In [12]: ds = xray.Dataset({'x': np.arange(1000)})

In [13]: with xray.set_options(display_width=40): ....: print(ds) ....: <xray.Dataset> Dimensions: (x: 1000) Coordinates: * x (x) int64 0 1 2 3 4 5 6 ... Data variables: empty

In [14]: with xray.set_options(display_width=60): ....: print(ds) ....: <xray.Dataset> Dimensions: (x: 1000) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ... Data variables: empty ```

Fixes #406.

Accessing data from remote datasets now has retrying logic (with exponential backoff) that should make it robust to occasional bad responses from DAP servers.

netcdf4python supports milliseconds and microseconds:

https://github.com/Unidata/netcdf4-python/commit/22d439d6d3602171dc2c23bca0ade31d3c49ad20

would it be possible to support in X-ray?

Fixes #369

These methods make it easy to switch back and forth between data arrays and datasets:

``` In [4]: ds = xray.Dataset({'a': 1, 'b': ('x', [1, 2, 3])}, ...: coords={'c': 42}, attrs={'Conventions': 'None'}) ...:

In [5]: ds.to_array() Out[5]: <xray.DataArray (variables: 2, x: 3)> array([[1, 1, 1], [1, 2, 3]]) Coordinates: c int64 42 * x (x) int64 0 1 2 * variables (variables) |S1 'a' 'b' Attributes: Conventions: None

In [6]: ds.to_array().to_dataset(dim='variables') Out[6]: <xray.Dataset> Dimensions: (x: 3) Coordinates: c int64 42 * x (x) int64 0 1 2 Data variables: a (x) int64 1 1 1 b (x) int64 1 2 3 Attributes: Conventions: None ```

Fixes #132

CC @IamJeffG

Also rename .chunk_data() -> .chunk() (but nobody is using that, yet).

This involves a big internal rename: block -> chunk

Dask is a library for out of core computation somewhat similar to biggus in conception, but with slightly grander aspirations. For examples of how Dask could be applied to weather data, see this blog post by @mrocklin: http://matthewrocklin.com/blog/work/2015/02/13/Towards-OOC-Slicing-and-Stacking/

It would be interesting to explore using dask internally in xray, so that we can implement lazy/out-of-core aggregations, concat and groupby to complement the existing lazy indexing. This functionality would be quite useful for xray, and even more so than merely supporting datasets-on-disk (#199).

A related issue is #79: we can easily imagine using Dask with groupby/apply to power out-of-core and multi-threaded computation.

Todos for xray: - [x] refactor Variable.concat to make use of functions like concatenate and stack instead of in-place array modification (Dask arrays do not support mutation, for good reasons) - [x] refactor reindex_variables to not make direct use of mutation (e.g., by using da.insert below) - [x] add some sort of internal abstraction to represent "computable" arrays that are not necessarily numpy.ndarray objects (done: this is the data attribute) - [x] expose reblock in the public API - [x] load datasets into dask arrays from disk - [x] load dataset from multiple files into dask - [x] ~~some sort of API for user controlled lazy apply on dask arrays (using groupby, mostly likely)~~ (not necessary for initial release) - [x] save from dask arrays - [x] an API for lazy ufuncs like sin and sqrt - [x] robustly handle indexing along orthogonal dimensions if dask can't handle it directly.

Todos for dask (to be clear, none of these are blockers for a proof of concept): - [x] support for NaN skipping aggregations - [x] ~~support for interleaved concatenation (necessary for transformations by group, which are quite common)~~ (turns out to be a one-liner with concatenate and take, see below) - [x] ~~support for something like take_nd from pandas: like np.take, but with -1 as a sentinel value for "missing" (necessary for many alignment operations)~~ da.insert, modeled after np.insert would solve this problem. - [x] ~~support "orthogonal" MATLAB-like array-based indexing along multiple dimensions~~ (taking along one axis at a time is close enough) - [x] broadcast_to: see https://github.com/numpy/numpy/pull/5371

The previous tests were actually not being run because I named the test method incorrectly :(

xref #328

Need to apply a transformation function across one dimension of a DataArray, where that non-xray function speaks in ndarrays. Currently the only ways to do this involve wrapping the function. An example:

``` import numpy as np import xray from scipy.ndimage.morphology import binary_opening

da = xray.DataArray(np.random.random_integers(0, 1, (10, 10, 3)), dims=['row', 'col', 'time'])

I want to apply an operation the 2D image at each point in time

da.groupby('time').apply(binary_opening)

AttributeError: 'numpy.ndarray' object has no attribute 'dims'

def wrap_binary_opening(da, kwargs): return xray.DataArray(binary_opening(da.values, kwargs), da.coords)

da.groupby('time').apply(wrap_binary_opening) da.groupby('time').apply(wrap_binary_opening, iterations=2) # func may take custom args ```

My proposed solution is that apply would automatically coerce func's return value to a DataArray.