issue_comments: 308935684

Keep the comments coming! I think we can distinguish between benchmarking for regressions and benchmarking for development and introspection.

The former will require some thought as to what machines we want to rely on and how to achieve consistency throughout the development track. It sounds like there are a number of options that we could pursue toward those ends.

The latter use of benchmarking is useful on a single machine with only a few commits of history. For the four benchmarks in my sample dataset_io.py, we get the following interesting results (for one environment): --[ 0.00%] Benchmarking conda-py2.7-bottleneck-dask-netcdf4-numpy-pandas-scipy ---[ 3.12%] Running dataset_io.IOSingleNetCDF.time_load_dataset_netcdf4 134.34ms ---[ 6.25%] Running dataset_io.IOSingleNetCDF.time_load_dataset_scipy 82.60ms ---[ 9.38%] Running dataset_io.IOSingleNetCDF.time_write_dataset_netcdf4 57.71ms ---[ 12.50%] Running dataset_io.IOSingleNetCDF.time_write_dataset_scipy 267.29ms

So the relative performance is useful information in deciding how to use and/or develop xarray. (Granted the exact factors will change depending on machine/architecture/dataset).