criterion performance measurements

overview

want to understand this report?

simple/nub [1]

lower bound estimate upper bound
Mean execution time 3.757625193349935e-8 3.781085510074313e-8 3.8252179579346605e-8
Standard deviation 9.85249956184582e-10 1.6010088690593945e-9 2.5378612435806385e-9

Outlying measurements have moderate (0.3953501471591051%) effect on estimated standard deviation.

simple/nub [1..10]

lower bound estimate upper bound
Mean execution time 7.480366116845878e-7 7.503388930791329e-7 7.541333701968792e-7
Standard deviation 1.0213003162528511e-8 1.4827501677952259e-8 2.2100629281644402e-8

Outlying measurements have moderate (0.12323783624720008%) effect on estimated standard deviation.

simple/nub [1..100]

lower bound estimate upper bound
Mean execution time 5.691851969311633e-5 5.7032865243653464e-5 5.725316401074331e-5
Standard deviation 4.4205068072009574e-7 7.866471429862453e-7 1.2946311934415994e-6

Outlying measurements have slight (6.597554989398262e-2%) effect on estimated standard deviation.

simple/nub [1..1000]

lower bound estimate upper bound
Mean execution time 5.540790407794217e-3 5.549774019854764e-3 5.564467756884794e-3
Standard deviation 3.960845337260408e-5 5.7610584021495294e-5 8.149383062426018e-5

Outlying measurements have no (9.899999999999956e-3%) effect on estimated standard deviation.

simple/nub (replicate 1000 1)

lower bound estimate upper bound
Mean execution time 1.5848331236445798e-5 1.589948285887123e-5 1.5971194213312692e-5
Standard deviation 2.3508994276362648e-7 3.0830437851443363e-7 4.071045376257791e-7

Outlying measurements have moderate (0.12297715079808672%) effect on estimated standard deviation.

simple/ordNub [1]

lower bound estimate upper bound
Mean execution time 3.6288780400376484e-8 3.644275932196124e-8 3.6690771641609886e-8
Standard deviation 6.727105378716715e-10 9.835364729889442e-10 1.415855108982019e-9

Outlying measurements have moderate (0.20961738012838177%) effect on estimated standard deviation.

simple/ordNub [1..10]

lower bound estimate upper bound
Mean execution time 7.182659840927772e-7 7.214246351305428e-7 7.274805399877833e-7
Standard deviation 1.3621532054494507e-8 2.1648928050637593e-8 3.911072750629427e-8

Outlying measurements have moderate (0.24827059816988464%) effect on estimated standard deviation.

simple/ordNub [1..100]

lower bound estimate upper bound
Mean execution time 1.5264215293267733e-5 1.5348067614811315e-5 1.5458932254093546e-5
Standard deviation 3.9681073943661674e-7 4.921714830174253e-7 6.102519856152593e-7

Outlying measurements have moderate (0.27722118637209103%) effect on estimated standard deviation.

simple/ordNub [1..1000]

lower bound estimate upper bound
Mean execution time 2.6686387739868615e-4 2.684441952138312e-4 2.7087924363505513e-4
Standard deviation 7.431464659516629e-6 9.917048464022355e-6 1.518868452332666e-5

Outlying measurements have moderate (0.33597511599326035%) effect on estimated standard deviation.

simple/ordNub (replicate 1000 1)

lower bound estimate upper bound
Mean execution time 1.0444461048787671e-5 1.047098511571272e-5 1.052141064519269e-5
Standard deviation 1.0092492975359744e-7 1.8071012641327875e-7 2.941454051429137e-7

Outlying measurements have moderate (0.10365231303554566%) effect on estimated standard deviation.

benchmarks:

lower bound estimate upper bound
Mean execution time 1.500860938729657e-8 1.5057863966836603e-8 1.515777152091937e-8
Standard deviation 1.7861074630884827e-10 3.443653954534959e-10 5.908468439214954e-10

Outlying measurements have moderate (0.1613661281101069%) effect on estimated standard deviation.

100 nub

lower bound estimate upper bound
Mean execution time 5.738100855487089e-3 5.75603946937869e-3 5.7859705377991e-3
Standard deviation 8.006019087342247e-5 1.1637446272655366e-4 1.7649661764426245e-4

Outlying measurements have moderate (0.13250781925514807%) effect on estimated standard deviation.

50 nub

lower bound estimate upper bound
Mean execution time 2.999088614123562e-3 3.0160616327698014e-3 3.041879980700711e-3
Standard deviation 7.786585250891225e-5 1.0630795035512745e-4 1.4615793757879908e-4

Outlying measurements have moderate (0.3163039024688868%) effect on estimated standard deviation.

10 nub

lower bound estimate upper bound
Mean execution time 7.391630852905413e-4 7.417964184967178e-4 7.459937775818011e-4
Standard deviation 1.1864660079701539e-5 1.6745047394271792e-5 2.3940789026026208e-5

Outlying measurements have moderate (0.16113835045292196%) effect on estimated standard deviation.

5 nub

lower bound estimate upper bound
Mean execution time 4.3811377111511914e-4 4.419445616348335e-4 4.467439276321481e-4
Standard deviation 1.718139578252032e-5 2.1878354147760183e-5 3.0930905168794936e-5

Outlying measurements have moderate (0.4746919991352466%) effect on estimated standard deviation.

1 nub

lower bound estimate upper bound
Mean execution time 1.6130154319797397e-4 1.6198527470093088e-4 1.6323617751261708e-4
Standard deviation 3.002781751749168e-6 4.6072937520403214e-6 8.063405462664703e-6

Outlying measurements have moderate (0.22887495498119245%) effect on estimated standard deviation.

100 localNub

lower bound estimate upper bound
Mean execution time 5.772554724353057e-3 5.786661951678495e-3 5.8093689371521255e-3
Standard deviation 6.450410937927353e-5 8.971144822646501e-5 1.397679413989703e-4

Outlying measurements have slight (8.473418576531792e-2%) effect on estimated standard deviation.

50 localNub

lower bound estimate upper bound
Mean execution time 2.9874132563049594e-3 2.9981349398071565e-3 3.0141590525085728e-3
Standard deviation 4.882686881548226e-5 6.671719673496743e-5 9.401161583001845e-5

Outlying measurements have moderate (0.15192231780996568%) effect on estimated standard deviation.

10 localNub

lower bound estimate upper bound
Mean execution time 7.415746892181537e-4 7.449447358337542e-4 7.513438904968403e-4
Standard deviation 1.359937652252373e-5 2.302782601622472e-5 3.687822073083338e-5

Outlying measurements have moderate (0.25807962861271205%) effect on estimated standard deviation.

5 localNub

lower bound estimate upper bound
Mean execution time 4.2775150362091757e-4 4.3087180677490924e-4 4.3424781385498744e-4
Standard deviation 1.4128447145009337e-5 1.6584199124428354e-5 2.0709779385456256e-5

Outlying measurements have moderate (0.3555612158994931%) effect on estimated standard deviation.

1 localNub

lower bound estimate upper bound
Mean execution time 1.6026813111233491e-4 1.608800721320288e-4 1.6209892089030252e-4
Standard deviation 2.524391063355487e-6 4.246582598750814e-6 7.849721033493185e-6

Outlying measurements have moderate (0.2001102806249412%) effect on estimated standard deviation.

100 ordNub

lower bound estimate upper bound
Mean execution time 3.1422278263916584e-4 3.15453499344488e-4 3.173756299292048e-4
Standard deviation 5.589229992185309e-6 7.786024120209112e-6 1.1176618261426014e-5

Outlying measurements have moderate (0.1807409498323374%) effect on estimated standard deviation.

50 ordNub

lower bound estimate upper bound
Mean execution time 2.627523490020799e-4 2.640803404876756e-4 2.66221736692306e-4
Standard deviation 5.8546635201334744e-6 8.455452589317331e-6 1.2285411065213186e-5

Outlying measurements have moderate (0.27718217178455157%) effect on estimated standard deviation.

10 ordNub

lower bound estimate upper bound
Mean execution time 1.7958788188019142e-4 1.8019406111755723e-4 1.8131254227676734e-4
Standard deviation 2.606323060676608e-6 4.102434135786844e-6 7.22352144272395e-6

Outlying measurements have moderate (0.16128804585412476%) effect on estimated standard deviation.

5 ordNub

lower bound estimate upper bound
Mean execution time 1.517319529193143e-4 1.5217016626770298e-4 1.5291665483886996e-4
Standard deviation 1.920158605251481e-6 2.8693607783864116e-6 4.3515951047746455e-6

Outlying measurements have moderate (0.11367193060301757%) effect on estimated standard deviation.

1 ordNub

lower bound estimate upper bound
Mean execution time 1.0844672357308723e-4 1.0938098380519524e-4 1.108118358777747e-4
Standard deviation 4.372713998264023e-6 5.867577480956698e-6 9.371904132361243e-6

Outlying measurements have severe (0.5147476153433842%) effect on estimated standard deviation.

100 ordNubState

lower bound estimate upper bound
Mean execution time 2.910477965014676e-3 2.934813349383573e-3 2.965774386065702e-3
Standard deviation 1.10036919913866e-4 1.3967710675703423e-4 2.079387159421413e-4

Outlying measurements have moderate (0.4547620602361708%) effect on estimated standard deviation.

50 ordNubState

lower bound estimate upper bound
Mean execution time 2.67104372276614e-3 2.686855642932156e-3 2.7073214937622346e-3
Standard deviation 7.566419289598708e-5 9.209925716021782e-5 1.1409306206765078e-4

Outlying measurements have moderate (0.30638253099992985%) effect on estimated standard deviation.

10 ordNubState

lower bound estimate upper bound
Mean execution time 2.4405239511902125e-3 2.4569962908203413e-3 2.4833844591553014e-3
Standard deviation 7.529469826043195e-5 1.0474415149836955e-4 1.6852242538112472e-4

Outlying measurements have moderate (0.4048341050829361%) effect on estimated standard deviation.

5 ordNubState

lower bound estimate upper bound
Mean execution time 2.415556757586698e-3 2.432343809741238e-3 2.4518106867248814e-3
Standard deviation 8.138618057297475e-5 9.23968421960196e-5 1.0752203793188458e-4

Outlying measurements have moderate (0.34596667141524823%) effect on estimated standard deviation.

1 ordNubState

lower bound estimate upper bound
Mean execution time 2.352418749469022e-3 2.367036192553738e-3 2.384726851123074e-3
Standard deviation 7.047731702794914e-5 8.2149950816592e-5 9.611514463776044e-5

Outlying measurements have moderate (0.30671641913114456%) effect on estimated standard deviation.

100 ordNubStateLazy

lower bound estimate upper bound
Mean execution time 2.05763801826785e-3 2.0713518549377715e-3 2.088229506152372e-3
Standard deviation 6.631828454493632e-5 7.748738649317136e-5 9.854941815447424e-5

Outlying measurements have moderate (0.3363226536856988%) effect on estimated standard deviation.

50 ordNubStateLazy

lower bound estimate upper bound
Mean execution time 1.9839118410522735e-3 1.9934295107300083e-3 2.0083902765686314e-3
Standard deviation 4.150828433086119e-5 6.027109191160883e-5 8.262017102002458e-5

Outlying measurements have moderate (0.24844552095007752%) effect on estimated standard deviation.

10 ordNubStateLazy

lower bound estimate upper bound
Mean execution time 1.8480680872375768e-3 1.8608902384216583e-3 1.8984459330017372e-3
Standard deviation 4.295196713867742e-5 1.0202637303171985e-4 2.194670468309026e-4

Outlying measurements have severe (0.5248426190791382%) effect on estimated standard deviation.

5 ordNubStateLazy

lower bound estimate upper bound
Mean execution time 1.8216560770447055e-3 1.829004137652616e-3 1.8402741838867466e-3
Standard deviation 3.273415745552623e-5 4.61654349603979e-5 6.881447487213734e-5

Outlying measurements have moderate (0.19018985266866223%) effect on estimated standard deviation.

1 ordNubStateLazy

lower bound estimate upper bound
Mean execution time 1.6614840914184839e-3 1.6679714609558383e-3 1.6781590868408483e-3
Standard deviation 2.930101092484572e-5 4.104310566105166e-5 5.8262001262434546e-5

Outlying measurements have moderate (0.1806841169520227%) effect on estimated standard deviation.

100 ordNubStateDlist

lower bound estimate upper bound
Mean execution time 5.089965955871676e-4 5.115985370137626e-4 5.169303711710705e-4
Standard deviation 1.0981994621602804e-5 1.825888176378724e-5 3.303856853592452e-5

Outlying measurements have moderate (0.31664148627924693%) effect on estimated standard deviation.

50 ordNubStateDlist

lower bound estimate upper bound
Mean execution time 4.1142877164917704e-4 4.138654095275949e-4 4.18452582989509e-4
Standard deviation 9.761611917691235e-6 1.6615327946162687e-5 2.6962813542417068e-5

Outlying measurements have moderate (0.3752548340450915%) effect on estimated standard deviation.

10 ordNubStateDlist

lower bound estimate upper bound
Mean execution time 2.9235797606201635e-4 2.9467023891500304e-4 3.000175702799528e-4
Standard deviation 8.937127975927884e-6 1.7149888924019318e-5 3.4185671617671015e-5

Outlying measurements have severe (0.5550128289258484%) effect on estimated standard deviation.

5 ordNubStateDlist

lower bound estimate upper bound
Mean execution time 2.586543309916224e-4 2.594236282735237e-4 2.604559807210334e-4
Standard deviation 3.6073717170807936e-6 4.5530897677941615e-6 6.934479089326931e-6

Outlying measurements have moderate (0.10386220995164383%) effect on estimated standard deviation.

1 ordNubStateDlist

lower bound estimate upper bound
Mean execution time 2.1045035534493983e-4 2.1131717717895901e-4 2.1312711479231006e-4
Standard deviation 3.5482492972356083e-6 6.127676613679356e-6 1.1439491791216442e-5

Outlying measurements have moderate (0.23838610300962604%) effect on estimated standard deviation.

100 ordNubStateLazyDlist

lower bound estimate upper bound
Mean execution time 5.128208295959564e-4 5.150349434672133e-4 5.180700119791757e-4
Standard deviation 1.004698437503386e-5 1.322921706220646e-5 1.876817326201683e-5

Outlying measurements have moderate (0.1995560755415029%) effect on estimated standard deviation.

50 ordNubStateLazyDlist

lower bound estimate upper bound
Mean execution time 4.0802415433960663e-4 4.099332911117624e-4 4.1285749498444295e-4
Standard deviation 8.484195172918509e-6 1.1960589461223399e-5 1.7196875585770635e-5

Outlying measurements have moderate (0.23852847240126115%) effect on estimated standard deviation.

10 ordNubStateLazyDlist

lower bound estimate upper bound
Mean execution time 2.89872859739181e-4 2.9130456330668606e-4 2.94045582237757e-4
Standard deviation 6.086866824262018e-6 9.76755096070446e-6 1.757593781896333e-5

Outlying measurements have moderate (0.2966443011443589%) effect on estimated standard deviation.

5 ordNubStateLazyDlist

lower bound estimate upper bound
Mean execution time 2.600367613861131e-4 2.612610407897996e-4 2.633591242858932e-4
Standard deviation 5.324786002142375e-6 8.04895494856847e-6 1.3508120510408214e-5

Outlying measurements have moderate (0.25799898809880273%) effect on estimated standard deviation.

1 ordNubStateLazyDlist

lower bound estimate upper bound
Mean execution time 2.0947663100281112e-4 2.10358779745487e-4 2.1159706624069563e-4
Standard deviation 4.072446392910867e-6 5.332417675948829e-6 7.542178270410888e-6

Outlying measurements have moderate (0.19027358077638765%) effect on estimated standard deviation.

100 nubBy

lower bound estimate upper bound
Mean execution time 7.061669676440454e-3 7.083868830340605e-3 7.119347899096708e-3
Standard deviation 1.0070724724377886e-4 1.4132774623729254e-4 2.0101611887256965e-4

Outlying measurements have moderate (0.1323058215857479%) effect on estimated standard deviation.

50 nubBy

lower bound estimate upper bound
Mean execution time 3.750278799670438e-3 3.758661596911648e-3 3.7736748148376733e-3
Standard deviation 3.447133709213671e-5 5.603562916078572e-5 8.502486114515167e-5

Outlying measurements have slight (7.550077255279915e-2%) effect on estimated standard deviation.

10 nubBy

lower bound estimate upper bound
Mean execution time 1.0000764573303354e-3 1.003062650033583e-3 1.008109971353163e-3
Standard deviation 1.3055170835307036e-5 1.9434754884315085e-5 2.874053705698317e-5

Outlying measurements have moderate (0.12296596193509188%) effect on estimated standard deviation.

5 nubBy

lower bound estimate upper bound
Mean execution time 5.179881549336842e-4 5.206107593038013e-4 5.247123535929454e-4
Standard deviation 1.1650311961638458e-5 1.654245107265438e-5 2.3642292091998627e-5

Outlying measurements have moderate (0.2678671549928868%) effect on estimated standard deviation.

1 nubBy

lower bound estimate upper bound
Mean execution time 2.9886147358765204e-4 3.001527486120661e-4 3.0331322529663637e-4
Standard deviation 4.926483138689462e-6 9.711617025268069e-6 1.964195785821559e-5

Outlying measurements have moderate (0.27744786574746233%) effect on estimated standard deviation.

100 ordNubBy

lower bound estimate upper bound
Mean execution time 5.789104597229098e-4 5.837646619934172e-4 5.933371679443455e-4
Standard deviation 2.022496437967801e-5 3.359235744342464e-5 5.701757021531044e-5

Outlying measurements have severe (0.5546692156438254%) effect on estimated standard deviation.

50 ordNubBy

lower bound estimate upper bound
Mean execution time 4.998945689656665e-4 5.031322932698664e-4 5.091754095214937e-4
Standard deviation 1.33609602085767e-5 2.193866512457328e-5 3.316153577327675e-5

Outlying measurements have moderate (0.4148049922676044%) effect on estimated standard deviation.

10 ordNubBy

lower bound estimate upper bound
Mean execution time 4.024606567962717e-4 4.0557201925354715e-4 4.095840078933785e-4
Standard deviation 1.4754863000906668e-5 1.809819100267715e-5 2.152503511812766e-5

Outlying measurements have moderate (0.4248088373248735%) effect on estimated standard deviation.

5 ordNubBy

lower bound estimate upper bound
Mean execution time 3.749435764892649e-4 3.775798899276803e-4 3.814130646331858e-4
Standard deviation 1.2318042480362338e-5 1.6105631609113144e-5 2.161656314609775e-5

Outlying measurements have moderate (0.4048511105704381%) effect on estimated standard deviation.

1 ordNubBy

lower bound estimate upper bound
Mean execution time 3.1464955189575746e-4 3.160233197485407e-4 3.187088666235408e-4
Standard deviation 5.281495780581269e-6 9.521072980597007e-6 1.5696533198717187e-5

Outlying measurements have moderate (0.24836321003836315%) effect on estimated standard deviation.

other/nub yitz 1

lower bound estimate upper bound
Mean execution time 1.6405437876159947e-3 1.6489337374145788e-3 1.6618297983581824e-3
Standard deviation 3.7520290934458463e-5 5.250790851010502e-5 7.444265270738427e-5

Outlying measurements have moderate (0.2679192733790396%) effect on estimated standard deviation.

other/ordNub yitz 1

lower bound estimate upper bound
Mean execution time 1.55090555443118e-3 1.5563176561767856e-3 1.5600536753113071e-3
Standard deviation 1.6471154068593e-5 2.275497442988379e-5 3.094951709247399e-5

Outlying measurements have slight (7.532217699541738e-2%) effect on estimated standard deviation.

other/nub yitz 2

lower bound estimate upper bound
Mean execution time 2.809221594470243e-3 2.8273175646240513e-3 2.8464005876953397e-3
Standard deviation 8.821591567482597e-5 9.551410029231446e-5 1.1296732734157991e-4

Outlying measurements have moderate (0.2968429774855128%) effect on estimated standard deviation.

other/ordNub yitz 2

lower bound estimate upper bound
Mean execution time 1.3526128221924112e-3 1.3569663454468049e-3 1.3621209551269808e-3
Standard deviation 2.0649320133914378e-5 2.4416149890353535e-5 2.944375693692283e-5

Outlying measurements have moderate (0.10416031817873998%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.