issue_comments

Yes! Thanks @wesm and @TomAugspurger.

Yep, thanks again for setting that up.

On Thu, Jul 27, 2017 at 11:39 AM, Wes McKinney notifications@github.com wrote:

cool, are these numbers coming off the pandabox?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1457#issuecomment-318417790, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHIk24wmNhChH3nCVT3AGqR_Q6EHa9ks5sSL1IgaJpZM4N74gy .

cool, are these numbers coming off the pandabox?

Awesome, thanks @TomAugspurger !

These are now being run and published to https://tomaugspurger.github.io/asv-collection/xarray/

I'm plan to find a more permanent home to publish the results rather than my personal github pages site, but that may take a while before I can get to it.

I will merge by the end of the day if no one has any more comments.

Thanks @TomAugspurger - see https://github.com/TomAugspurger/asv-runner/issues/1.

All, I added a series of multi-file benchmarks. I think for a first PR, this is ready to fly and we can add more benchmarks as needed.

About hardware, we should be able to run these on the machine running the pandas benchmarks. Once it's merged I should be able to add it easily to https://github.com/TomAugspurger/asv-runner/blob/master/tests/full.yml and the benchmarks will be run and published (to https://tomaugspurger.github.io/asv-collection/ right now; not the permanent home)

I think this a great start!

I would really like to see a performance test for open_mfdataset, since this is my own personal bottleneck.

Regarding the dependence on hardware, I/O speeds, etc, we should be able to resolve this by running on specific instance types on a cloud platform. We could configure environments with local SSD storage, network storage, etc, in order to cover different scenarios.

We should do this to the extent that it is helpful in driving development. Even just a few realistic use cases can be helpful, especially for guarding against performance regressions. On Thu, Jul 13, 2017 at 3:37 PM Joe Hamman notifications@github.com wrote:

@rabernat https://github.com/rabernat - do you have any thoughts on this?

@pydata/xarray https://github.com/orgs/pydata/teams/xarray - I'm trying to decide if this is worth spending any more time on. What sort of coverage would we want before we merge this first PR?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1457#issuecomment-315220704, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1rVPL6fCRqwi1Szmtq09chkah9odks5sNpwQgaJpZM4N74gy .

@rabernat - do you have any thoughts on this?

@pydata/xarray - I'm trying to decide if this is worth spending any more time on. What sort of coverage would we want before we merge this first PR?

Keep the comments coming! I think we can distinguish between benchmarking for regressions and benchmarking for development and introspection.

The former will require some thought as to what machines we want to rely on and how to achieve consistency throughout the development track. It sounds like there are a number of options that we could pursue toward those ends.

The latter use of benchmarking is useful on a single machine with only a few commits of history. For the four benchmarks in my sample dataset_io.py, we get the following interesting results (for one environment): --[ 0.00%] Benchmarking conda-py2.7-bottleneck-dask-netcdf4-numpy-pandas-scipy ---[ 3.12%] Running dataset_io.IOSingleNetCDF.time_load_dataset_netcdf4 134.34ms ---[ 6.25%] Running dataset_io.IOSingleNetCDF.time_load_dataset_scipy 82.60ms ---[ 9.38%] Running dataset_io.IOSingleNetCDF.time_write_dataset_netcdf4 57.71ms ---[ 12.50%] Running dataset_io.IOSingleNetCDF.time_write_dataset_scipy 267.29ms

So the relative performance is useful information in deciding how to use and/or develop xarray. (Granted the exact factors will change depending on machine/architecture/dataset).

This is a great start! Thanks @jhamman !

Our most common performance problems are handling pandas 'oddities', like non-standard indexes. Generally when an operation that is generally vectorized becomes un-vectorized, and starts looping in python. But that's probably not a big use case for most.

What are the instances others have seen performance issues? Are there ever issues with the standard transform operations, such as merge?

(addendum, I just saw the comments above): I think there's some real benefit in benchmarks to ensure we don't add code that slow down operations by an order of magnitude slower - i.e. outside the bounds of reasonable error. That's broader than optimizing around them, particularly since xarray is all python, and shouldn't be doing performance intensive work internally.

The tests for Hypothesis take almost twice as long to run on Travis at certain times of day, so I certainly wouldn't use it for benchmarking anything!

Also concerned that a dedicated benchmarking machine may lead to software (accidentally!) optimized for a particular architecture or balance of machine resources without due consideration. Maybe @wesm could investigate fault injection to (eg) slow down disk access or add latency for some sets of benchmarks?

@wesm just setup a machine for dedicated benchmarking of pandas and possibly other pydata/scipy project (if there's extra capacity as expected). @TomAugspurger has been working on getting it setup. So that's potentially an option, at least for single machine benchmarks.

The lore I've heard is that benchmarking on shared cloud resources (e.g., Travis-CI) can have reproducibility issues due to resource contention and/or jobs getting scheduled on slightly different machine types. I don't know how true this still is, or if there are good work arounds for particular cloud platforms. I suspect this should be solvable, though. I can certainly make an internal inquiry about benchmarking on GCP if we can't find answers on our own.

I like the idea of benchmarks, but have some serious concerns. For Dask and IO-bound work in general, benchmark results will vary widely depending on the hardware and (if relevant) network properties. Results will be noncomparable between SSD and HDD, local and remote network access, and in general depend heavily on the specific IO patterns and storage/compute relationship of the computer.

This isn't a reason not to benchmark though, just a call for very cautious interpretation - it's clearly useful to catch some of the subtle-but-pathological performance problems that have cropped up. In short, I think benchmarks should have a very clear warnings section in the documentation, and no decision should be taken to change code without benchmarking on a variety of computers (SSD/HDD, PC/cluster, local/remote data...).

Also JSON cannot include comments, and there are a number of entries that you need to update, but that's a passing concern.