issue_comments

Thanks @Zac-HD !

I would say we should merge it now and iterate in future PRs.

Merge away then!

blockers to merging this as-is?

This looks pretty good to me in its current state. I would say we should merge it now and iterate in future PRs.

other APIs you think it would be reasonably easy to write property tests for? Here's a nice list of properties to test 😄

Almost anywhere where we currently make heavy use of parametrize would be a good candidate. Some other possibilities:

• Consistency with pandas for groupby/rolling aggregations.
• Roundtrip writing/reading data to netCDF. There are a couple of known exceptions (e.g., dtypes not supported by netCDF and MultiIndex) but otherwise every xarray object should be serializable to netCDF and back without data loss.
• Roundtrip to/from pandas Series/DataFrame with to_series()/to_dataframe()/to_xarray().
• Indexing consistency tests for backends: all indexing operations should be supported consistently on data accessed from any backend.
• NumPy vs Dask: any operation on dask arrays should be consistent with the operation on numpy arrays (e.g., f(xarray_obj.chunk()).compute() == f(xarray_obj)).
• Indexing followed by xarray.concat: should get back the same result.
• Binary arithmetic on xarray objects with Python operators (+, -, etc) and NumPy ufuncs (np.add, np.subtract, etc).

@shoyer & @fmaussion - I've just given up on the plotting tests as being more effort than they're worth. Are there any:

• blockers to merging this as-is?
• other APIs you think it would be reasonably easy to write property tests for? Here's a nice list of properties to test :smile:

Set the backend to Agg on travis as you don't have a xserever running. You probably want to manually force a draw as well.

@jklymak - I'm getting RuntimeError: Invalid DISPLAY variable in the Qt backend now that I've added plt.clf(). It works on my machine now (:tada:, thanks!) - any suggestions for Travis?

(I'm also getting Zarr errors, but I assume those will go away soon as I didn't cause them)

...that also explains why I was having trouble reproducing the error, whoops. I'll see how it goes with those problems excluded later tonight!

As pointed out on the matplotlib gitter:

If you run

```python import numpy as np import xarray as xr import matplotlib.pyplot as plt

for i in range(200): xr.DataArray(np.array([[0, 0], [0, 0]], dtype=np.uint8)).plot.pcolormesh() at step 165 you will get: File "/Users/jklymak/matplotlib/lib/matplotlib/figure.py", line 236, in update raise ValueError('left cannot be >= right') ValueError: left cannot be >= right ``` Why? Because you have made a plot that if it displays looks like:

Are you sure your test isn't doing something similar? At some point there just isn't room for more colorbars! Adding a plt.clf() can cure the problem.

Its also is possible you are hitting floating point overflows with your test. At some point Matplotlib needs to be able to manipulate the data that comes in, and if you operate near the maximum number your data type can handle, you'll have problems. Just like you would if you just did

python a = 2*xr.DataArray(np.array([[0, 0], [0, 1e308]])) you will get: /Users/jklymak/anaconda3/envs/matplotlibdev/lib/python3.6/site-packages/xarray/core/variable.py:1165: RuntimeWarning: overflow encountered in multiply So maybe your hypothesis tester could be constrained to stay away from floating point overflows?

Matplotlib indeed has flaws and quirks, but if you are finding bugs it would be good to isolate them.

If we don't want to trigger by default, we can do something like this and require passing this to run them:

pytest --property-tests

@shoyer - that depends mostly on whether you want to run these tests as part of a standard run. A test-time dependency on Hypothesis is very cheap compared to the other dev dependencies, so I'd think more about the impact on eg CI resources than contributors.

Upside is more power and coverage of edge-cases with odd data; downside is that they take a lot longer by virtue of trying hundreds of examples, and in this case also having to generate arrays takes a while (~log(elements) average via sparse filling).

@tacaswell - I would be delighted to write a test suite like this for matplotlib! The only reason I haven't is because I thought it would be rude to report so many bugs that I don't have time to help fix. If we can get a group project going though I'd be very enthusiastic 😄

• For this suite I was passing in 2D arrays of unsigned ints, signed ints, or floats (in all sizes), with edge sizes between 2 and 10.
• The failing outputs were reported as 2x2 arrays (~30 times), or 3x2 (once - there's a cap on number of shrinks that I think I hit there).
• Values tended to be either uniform small integers - 0 or 1, dtype int8 or uint8; or for some an array of floats with one corner zero and the others very large.

I didn't keep the exact tracebacks, but I remember seeing many come from overflow in tick spacing calculations. Again, happy to write a test suite and make more detailed reports upstream if people want to fix this - in which case let's open an issue there!

What do the failing data sets look like? Does it get easier or harder to find failures if you go up to 10x10? What sort of exceptions are you getting?

You can shove a fair amount of configuration in to pytest.ini to make these opt-in.

One thing that comes to mind is organization... would it make sense to put this alongside the current xarray tests, e.g., have xarray/tests/unit and xarray/tests/property?

I guess one downside of this would be that it could change how we need to invoke py.test by default, if we don't want to trigger all the property based tests.

This looks like a great start to me -- thank you!

You're welcome! Same questions as above though, plus "is there anything else you need in this initial PR?". If not, can we merge it

It's impressive that it's possible to break every plotting type with matplotlib :).

As much as I love matplotlib, it's a steaming pile of hacks and I want to avoid it more than I want it cleaned up 😥 (entirely because the process is dysfunctional, not the code)

This looks like a great start to me -- thank you!

It's impressive that it's possible to break every plotting type with matplotlib :).

Ping @fmaussion / @shoyer - would love your opinions on this, including high-value targets to test.

(btw Appveyor had an internal error; build is otherwise green)