[ENH] Simple benchmarking suite.

[sstanovnik]

To ease my benchmarking of pandas code, I created a simple benchmark suite. It's based on unittest and is timeit-like, but does not use timeit.

I've included some basic tests. See if this is something we want included with Orange. I'm only not sure about the location, for now, I've put this in /benchmark/

Example output on current master:

Running benchmark with CPython v3.5.2 on Darwin-15.6.0-x86_64-i386-64bit (OSX 10.11.6 x86_64)
[adult_create_X] with 50 loops, best of 3:
    min 0.518 usec per loop
    avg 0.528 usec per loop

[adult_filter] skipped: Not a pandas environment.
[adult_read] with 5 loops, best of 3:
    min 748 msec per loop
    avg 786 msec per loop
[iris_basic_stats] with 50 loops, best of 3:
    min 206 usec per loop
    avg 247 usec per loop
[iris_contingency] with 10 loops, best of 3:
    min 200 usec per loop
    avg 211 usec per loop
[iris_create_X] with 100 loops, best of 3:
    min 0.491 usec per loop
    avg 0.525 usec per loop
[iris_discretize] with 10 loops, best of 3:
    min 662 usec per loop
    avg 991 usec per loop
[iris_distributions] with 20 loops, best of 3:
    min 110 usec per loop
    avg 115 usec per loop

[iris_iteration_pandas] skipped: Not a pandas environment.
[iris_iteration_pre_pandas] with 10 loops, best of 3:
    min 681 usec per loop
    avg 782 usec per loop
[iris_read] with 100 loops, best of 3:
    min 2.49 msec per loop
    avg 2.87 msec per loop

[codecov-io]

Current coverage is 88.26% (diff: 100%)

Merging #1510 into master will not change coverage

@@             master      #1510   diff @@
==========================================
  Files            77         77          
  Lines          7624       7624          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits           6729       6729          
  Misses          895        895          
  Partials          0          0          

Powered by Codecov. Last update 1ee248a...341dbcc

[kernc]

Could this be shaped into a set of performance guarantee tests which always run and fail if some change results in poorer performance over some threshold?

[kernc]

but does not use timeit.

You say this as if it were a feature? 😃

[sstanovnik]

I don't think it can be done, atleast without significant effort. There is no travis-like tool that integrates with github, the only similar one I found was asv, which seems local-only. Running on travis is impossible because it does not provide a consistent performance environment as far as I'm aware. The threshold is the other problem: you have to allow for some noise and, even then, you'd get a bazillion PRs that worsen the performance slightly and then another one that, maybe even because of noise, finally fails, without it even being its fault.

timeit-likeness is a feature only in that the output is similar and it runs tests numberxrepeat times :)

[astaric]

What is this?