issue_comments

Another approach here (rather than writing something new with Numba) would be to write a pure NumPy engine for groupby that relies on reordering data and np.add.accumulate. This could yield performance within a factor of 2-3x slower than pandas. See this comment for an example: https://github.com/numpy/numpy/issues/7265#issuecomment-198796408

Agreed! If you'd like to make a pull request that would be greatly appreciated

On Sun, Nov 15, 2015 at 10:10 PM, Antony Lee notifications@github.com wrote:

Perhaps worth mentioning in the docs? The difference turned out to be a major bottleneck in my code.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/issues/659#issuecomment-156925589.

Yes, switching to pandas for these operations is certainly a recommended approach :).

Yes, I'm afraid this is a known issue. Grouped aggregations are currently implemented with a loop in pure Python, which, of course, is pretty slow.

I've done some exploratory work to rewrite them in Numba, which shows some encouraging preliminary results:

``` from numba import guvectorize, jit import pandas as pd import numpy as np

@guvectorize(['(float64[:], int64[:], float64[:])'], '(x),(x),(y)', nopython=True) def _grouped_mean(values, int_labels, target): count = np.zeros(len(target), np.int64) for i in range(len(values)): val = values[i] if not np.isnan(val): lab = int_labels[i] target[lab] += val count[lab] += 1 target /= count

def move_axis_to_end(array, axis): array = np.asarray(array) return np.rollaxis(array, axis, start=array.ndim)

def grouped_mean(values, by, axis=-1): int_labels, uniques = pd.factorize(by, sort=True) values = move_axis_to_end(values, axis) target = np.zeros(values.shape[:-1] + uniques.shape) _grouped_mean(values, int_labels, target) return target, uniques

values = np.random.RandomState(0).rand(int(1e6)) values[::50] = np.nan by = np.random.randint(50, size=int(1e6)) df = pd.DataFrame({'x': values, 'y': by})

np.testing.assert_allclose(grouped_mean(values, by)[0], df.groupby('y')['x'].mean())

%timeit grouped_mean(values, by) # 100 loops, best of 3: 15.3 ms per loop %timeit df.groupby('y').mean() # 10 loops, best of 3: 21.4 ms per loop ```

Unfortunately, I'm unlikely to have time to work on this in the near future. If you or anyone else is interested in taking the lead on this, it would be greatly appreciated!

Note that we can't reuse the routines from pandas because they are only designed for 1D or at most 2D data.