issue_comments

@dcherian Just to be complete, I thought the following one-liner would work as well:

sums, x = np.histogram(latitude, bins, weights=array)

but apparently it produces slightly different results for reasons I don't understand

the IDL histogram function but in numpy.

Apparently not as awesome!

Yeah, the present solution is less general, but most of the time I'm just counting stuff, and this is much faster than what I was doing, so I'm happy ;-)

@dcherian I just tested your numpy suggestions, and I'm getting 100x speedups compared to my naive numpy approach (~200µs vs ~20ms). Thankyouthankyouthankyou! I've been doing this for years, I can't believe I've never run into that particular solution. It's like the IDL histogram function but in numpy. I'm going to use this like crazy Thanks again

@dcherian this means that xarray's groupby_bins will always be slow unless flox is installed, correct? I have unfortunately little or no say on what packages are installed on the system that runs my code.

Apparently not. https://stackoverflow.com/questions/48227792/how-to-change-dpi-of-my-pandas-dataframe-plot

Has this behavior been fixed by the change to NetCDF4 ? https://github.com/Unidata/netcdf4-python/issues/743

with that keyword the open_mfdataset goes through ! my code crashes later because the data is too big to fit in memory, but that's another problem ;-)

Hey @pwolfram, I installed your branch and tried to open_mfdataset on 1474 hdf5 files.

I got the following: File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 524, in open_mfdataset File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 524, in <listcomp> File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 299, in open_dataset File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 203, in __init__ File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 178, in _open_netcdf4_group File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:13992) OSError: Too many open files

I hope everything's ok now. Thanks for the hand-holding

Can anyone knowledgeable tell me how to fix the appveyor build? It's complaining of a failing test_dask_distributed_integration_test...

thanks ! Do we need to worry about the "appveyor" build fails? The failure seems to happen somewhere unrelated

I have added a couple of exceptions to the _infer_xy_labels function and updated the tests as best I could

ok, I see. In case there's a vote, I vote for raising an error in 1D too :-)

I don't understand the "raise an error" option. Would it mean that creating a Dataset with 1D arrays with no named dimension would raise an error? That would be very impractical.

you mean, within numpy?

I'm just chiming in to signify my interest in seeing this issue solved. I have just hit "OSError: Too many open files". The data itself is not even huge, but it's scattered across many files and it's a PITA to revert to manual concatenation -- I've grown used to dask doing the work for me ;-)

As it is, the xarray code becomes tied up with cartopy. If cartopy somehow changes its syntax (e.g. by renaming the transform kw) this code won't work.

Maybe there's another, more abstract way to do this?

One concern I have is that it is conceivable that one might pass a transform keyword to plot() intended to interact with another library that hijacks matplotlib (i.e. not cartopy). In that case, matplotlib will get an unintended projection argument. That is a corner case though.

I don't understand what you mean by 'adding you own to the signature'. Adding my own what to what signature?

I am confused.

I planned to identify the plotting function that accepted the transform keyword (used in the code sample above), and add its value in subplot_kws if it wasn't already.

However it looks like no xarray code contains the transform keyword. However I definitely can use it in calls to plot(). I don't understand what is going on. Can someone enlighten me?

I have found in issue #604 a example of how to do this. I have submitted in issue #1203 a change to the docs.

I was briefly confused by the examples as at first they seemed to me to do nothing -- read data and forget it. This change prevents any similar confusion :-)