Improve groupby's agg performance on large data; from 16s to 20ms for 20k rows

Improve groupby's agg performance on large data; from 16s to 20ms for 20k rows #631