criterion performance measurements

overview

want to understand this report?

fft/cpp 0 1048576 1

lower bound estimate upper bound
Mean execution time 0.18612994539663197 0.1866152678720653 0.18712305891439318
Standard deviation 2.221712412708202e-3 2.537568632049103e-3 3.1527663592137998e-3

Outlying measurements have slight (6.585928960356663e-2%) effect on estimated standard deviation.

fft/cpp 0 1048576 2

lower bound estimate upper bound
Mean execution time 0.30006938564702884 0.3005229197733103 0.3011122928850353
Standard deviation 2.0071155664530627e-3 2.634254622792714e-3 4.074459246401739e-3

Outlying measurements have no (9.899999999999833e-3%) effect on estimated standard deviation.

fft/cpp 0 1048576 15

lower bound estimate upper bound
Mean execution time 1.7468081818811596 1.751003001808375 1.7554461537592112
Standard deviation 1.9986325076851635e-2 2.2154665697566397e-2 2.8005892141848277e-2

Outlying measurements have slight (3.339247213808505e-2%) effect on estimated standard deviation.

fft/cpp 0 1048576 52

lower bound estimate upper bound
Mean execution time 5.825725344299523 5.836965740799155 5.853630427002162
Standard deviation 5.023501701449372e-2 6.939794009372788e-2 9.53106322710898e-2

Outlying measurements have no (9.899999999999807e-3%) effect on estimated standard deviation.

fft/cpp 0 1048576 102

lower bound estimate upper bound
Mean execution time 11.664622770427917 11.674999095081539 11.69011600124761
Standard deviation 4.6691289747188926e-2 6.354404722792452e-2 0.10020939438339699

Outlying measurements have no (9.899999999999961e-3%) effect on estimated standard deviation.

fft/cpp 1 1048576 1

lower bound estimate upper bound
Mean execution time 0.10986997711583973 0.11005817758962515 0.1102610884897411
Standard deviation 7.232471295892089e-4 1.0045100962609238e-3 1.457388376802546e-3

Outlying measurements have no (9.900000000000004e-3%) effect on estimated standard deviation.

fft/cpp 1 1048576 2

lower bound estimate upper bound
Mean execution time 0.16671858417913327 0.16714817869588738 0.1675586162798107
Standard deviation 1.9471114738052592e-3 2.1491242528565335e-3 2.3893886786595106e-3

Outlying measurements have slight (5.6597463647776236e-2%) effect on estimated standard deviation.

fft/cpp 1 1048576 15

lower bound estimate upper bound
Mean execution time 0.8890782533876594 0.8895452867738901 0.890005653976649
Standard deviation 2.104664817971811e-3 2.3753162416684103e-3 2.727150195808458e-3

Outlying measurements have no (9.899999999999999e-3%) effect on estimated standard deviation.

fft/cpp 1 1048576 52

lower bound estimate upper bound
Mean execution time 2.9457283769838516 2.9473769174806774 2.94952778923437
Standard deviation 7.3492416407999524e-3 9.614689675239148e-3 1.3093067867444384e-2

Outlying measurements have no (9.899999999999923e-3%) effect on estimated standard deviation.

fft/cpp 1 1048576 102

lower bound estimate upper bound
Mean execution time 5.721246892093867 5.724914542316646 5.729733749508111
Standard deviation 1.6730455669730165e-2 2.1327736248227126e-2 2.8076867470015282e-2

Outlying measurements have no (9.89999999999981e-3%) effect on estimated standard deviation.

fft/fft 0 1048576 1

lower bound estimate upper bound
Mean execution time 0.9609226905100045 0.9618111692659558 0.9627566443674269
Standard deviation 4.15636484758741e-3 4.715764969449037e-3 5.421131448546586e-3

Outlying measurements have no (9.899999999999999e-3%) effect on estimated standard deviation.

fft/fft 0 1048576 2

lower bound estimate upper bound
Mean execution time 1.2774529467813664 1.280989700435847 1.2858826814882456
Standard deviation 1.5948748983194223e-2 2.1201588481617385e-2 2.6448025148638775e-2

Outlying measurements have slight (9.429606588504784e-2%) effect on estimated standard deviation.

fft/fft 0 1048576 15

lower bound estimate upper bound
Mean execution time 5.151861415981503 5.154335230468959 5.158089054702967
Standard deviation 1.1004299158051551e-2 1.5509229612324532e-2 2.3238128168085053e-2

Outlying measurements have no (9.9e-3%) effect on estimated standard deviation.

fft/fft 0 1048576 52

lower bound estimate upper bound
Mean execution time 16.577773078560078 16.602952629207817 16.631316815494753
Standard deviation 0.1192508794621941 0.13715610795560934 0.16999046456295736

Outlying measurements have no (9.899999999999905e-3%) effect on estimated standard deviation.

fft/fft 0 1048576 102

lower bound estimate upper bound
Mean execution time 32.03925145971703 32.075127545475226 32.117118848442274
Standard deviation 0.16856264683344982 0.19787550744948054 0.23827979283248185

Outlying measurements have no (9.899999999999912e-3%) effect on estimated standard deviation.

fft/fft 1 1048576 1

lower bound estimate upper bound
Mean execution time 0.8494043718568977 0.8495764838449655 0.8499454461328685
Standard deviation 7.153296548485741e-4 1.2318724320136995e-3 2.4570227062198947e-3

Outlying measurements have no (9.9e-3%) effect on estimated standard deviation.

fft/fft 1 1048576 2

lower bound estimate upper bound
Mean execution time 1.0947435818903148 1.096440709709376 1.0988651119463146
Standard deviation 7.8098769594812534e-3 1.027017706198584e-2 1.3424753346812597e-2

Outlying measurements have no (9.899999999999888e-3%) effect on estimated standard deviation.

fft/fft 1 1048576 15

lower bound estimate upper bound
Mean execution time 4.294673284172267 4.295565436958519 4.296714626430718
Standard deviation 4.214397156119645e-3 5.199570802378709e-3 6.366333521183222e-3

Outlying measurements have no (9.899999999999727e-3%) effect on estimated standard deviation.

fft/fft 1 1048576 52

lower bound estimate upper bound
Mean execution time 13.63081655371115 13.63513171303198 13.64235839712545
Standard deviation 1.857087792060353e-2 2.7896121303418495e-2 4.1112732323592416e-2

Outlying measurements have no (9.89999999999986e-3%) effect on estimated standard deviation.

fft/fft 1 1048576 102

lower bound estimate upper bound
Mean execution time 26.269781883835048 26.273336030124867 26.276330085872857
Standard deviation 1.3354587771940304e-2 1.673988556186407e-2 2.125828842109896e-2

Outlying measurements have no (9.899999999999999e-3%) effect on estimated standard deviation.

fft/vector-fftw 0 1048576 1

lower bound estimate upper bound
Mean execution time 0.43520556557103984 0.43560343611165864 0.43648464310094715
Standard deviation 1.591329179165768e-3 2.887026809079535e-3 5.57358290002061e-3

Outlying measurements have no (9.899999999999961e-3%) effect on estimated standard deviation.

fft/vector-fftw 0 1048576 2

lower bound estimate upper bound
Mean execution time 0.5229938637010755 0.5257210718385876 0.5298130761377514
Standard deviation 1.256205256596619e-2 1.6841914375146243e-2 2.4559921518271455e-2

Outlying measurements have moderate (0.2771960526092918%) effect on estimated standard deviation.

fft/vector-fftw 0 1048576 15

lower bound estimate upper bound
Mean execution time 1.5845820485346012 1.5885962783090775 1.5939280830614266
Standard deviation 1.886053542979143e-2 2.3557269934731225e-2 3.443726224664691e-2

Outlying measurements have slight (7.545245333044358e-2%) effect on estimated standard deviation.

fft/vector-fftw 0 1048576 52

lower bound estimate upper bound
Mean execution time 4.669444783806058 4.681644967674464 4.693495298504084
Standard deviation 5.499592395267495e-2 6.15737089771931e-2 6.939925729657412e-2

Outlying measurements have slight (6.55738877027396e-2%) effect on estimated standard deviation.

fft/vector-fftw 0 1048576 102

lower bound estimate upper bound
Mean execution time 9.47732762443945 9.534357153057313 9.575632241844394
Standard deviation 0.18704145328526348 0.2456129139818392 0.3171265896035054

Outlying measurements have moderate (0.19961661271325593%) effect on estimated standard deviation.

fft/vector-fftw 1 1048576 1

lower bound estimate upper bound
Mean execution time 0.4431934415094555 0.4453832756273448 0.44805799353048187
Standard deviation 1.0909482187118265e-2 1.240428452768549e-2 1.4600398369243102e-2

Outlying measurements have moderate (0.2193438562812842%) effect on estimated standard deviation.

fft/vector-fftw 1 1048576 2

lower bound estimate upper bound
Mean execution time 0.5291992388956248 0.5327537928812207 0.5369731961481271
Standard deviation 1.743625381451908e-2 1.9677867553484184e-2 2.375334790967199e-2

Outlying measurements have moderate (0.33597014608176407%) effect on estimated standard deviation.

fft/vector-fftw 1 1048576 15

lower bound estimate upper bound
Mean execution time 1.6141594229929148 1.6204619013063608 1.6275767384760076
Standard deviation 2.9821999462811492e-2 3.432590848746605e-2 3.9692775289501464e-2

Outlying measurements have moderate (0.14216505209710983%) effect on estimated standard deviation.

fft/vector-fftw 1 1048576 52

lower bound estimate upper bound
Mean execution time 4.708895794033259 4.716575387596339 4.724301196216793
Standard deviation 3.65535507111469e-2 3.938009860125324e-2 4.255226025065104e-2

Outlying measurements have no (9.900000000000178e-3%) effect on estimated standard deviation.

fft/vector-fftw 1 1048576 102

lower bound estimate upper bound
Mean execution time 8.987678891300408 9.029337042450155 9.076703275322172
Standard deviation 0.20033161227238386 0.22833692961079885 0.25458039845546804

Outlying measurements have moderate (0.19022674045295346%) effect on estimated standard deviation.

fft/repa 0 4096 1

lower bound estimate upper bound
Mean execution time 7.215504515096545e-2 7.218703854009507e-2 7.222766029760241e-2
Standard deviation 1.4606532239287012e-4 1.8336156171157184e-4 2.8102303292816864e-4

Outlying measurements have no (9.899999999999876e-3%) effect on estimated standard deviation.

fft/repa 0 4096 2

lower bound estimate upper bound
Mean execution time 0.11999816048070791 0.12004701960012319 0.12009939539358017
Standard deviation 2.2819718292447702e-4 2.592249028276347e-4 3.0951131605447876e-4

Outlying measurements have no (9.9e-3%) effect on estimated standard deviation.

fft/repa 0 4096 15

lower bound estimate upper bound
Mean execution time 0.74537278759405 0.7454958831064402 0.7456110440485176
Standard deviation 5.406025208297956e-4 6.106743126058248e-4 6.978268229442036e-4

Outlying measurements have no (9.899999999999715e-3%) effect on estimated standard deviation.

fft/repa 0 4096 52

lower bound estimate upper bound
Mean execution time 2.5922464023821052 2.5924857889406385 2.59273145782873
Standard deviation 1.0661156018198458e-3 1.2463222656005463e-3 1.5368561620726872e-3

Outlying measurements have no (9.89999999999998e-3%) effect on estimated standard deviation.

fft/repa 0 4096 102

lower bound estimate upper bound
Mean execution time 5.2414451872102905 5.241937953113764 5.2423958979837595
Standard deviation 2.0911317727955064e-3 2.4393200943840336e-3 3.1249370237842002e-3

Outlying measurements have no (9.9e-3%) effect on estimated standard deviation.

fft/pure-fft 0 4096 1

lower bound estimate upper bound
Mean execution time 0.12323273050710559 0.12340774881765247 0.12361601459905498
Standard deviation 7.936676718000133e-4 9.704788428611137e-4 1.1866473213146447e-3

Outlying measurements have no (9.89999999999989e-3%) effect on estimated standard deviation.

fft/pure-fft 0 4096 2

lower bound estimate upper bound
Mean execution time 0.21512646067067978 0.21528300392553207 0.2154508529894054
Standard deviation 7.398329305629838e-4 8.327342627439728e-4 9.640459293035182e-4

Outlying measurements have no (9.899999999999944e-3%) effect on estimated standard deviation.

fft/pure-fft 0 4096 15

lower bound estimate upper bound
Mean execution time 1.4103369437448678 1.4108058177225287 1.4113172255747024
Standard deviation 2.153844752526997e-3 2.5083683281131594e-3 3.157450358265639e-3

Outlying measurements have no (9.899999999999985e-3%) effect on estimated standard deviation.

fft/pure-fft 0 4096 52

lower bound estimate upper bound
Mean execution time 4.809959086059775 4.8119317017786205 4.814275447010248
Standard deviation 8.882307550501609e-3 1.103176227998068e-2 1.412360595899558e-2

Outlying measurements have no (9.9e-3%) effect on estimated standard deviation.

fft/pure-fft 0 4096 102

lower bound estimate upper bound
Mean execution time 9.389050718425958 9.394914628147331 9.401325243591518
Standard deviation 2.706683015800584e-2 3.1295079471281725e-2 3.608563935385618e-2

Outlying measurements have no (9.899999999999999e-3%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.