Week 05 · Phase 2 · Measuring for real

Functions and the Stopwatch

Wrap work in a function, then time it honestly with perf_counter and repeats.

Big question: How long does my code actually take?functionstiming≈2.5 hours
By the end of this week you can
  • write a function with def, give it inputs and get a value back;
  • time a piece of code with time.perf_counter();
  • explain why one measurement is never enough;
  • run the same function at several input sizes and put the results in a table;
  • spot the three things that make a benchmark lie;
  • know why the first timed run is usually thrown away, and how fine your stopwatch really is;
  • choose between perf_counter and the timeit module for a given job.

5.1A function is a named piece of work

Up to now, changing n meant editing the code and running it again. A function lets you say "do that job, with this input" as many times as you like.

def, parameter, return
def sum_to(n):
    total = 0
    for number in range(1, n + 1):
        total = total + number
    return total

print(sum_to(10))
print(sum_to(1000))
55 500500

Three parts, and each has a job:

A useful picture: a function is a recipe card. Writing the card teaches the kitchen a dish; it does not cook anything. You only get food when someone reads the card and follows it — that is a call. The parameter n is the blank on the card that says "however many portions you asked for", filled in fresh each time the card is used.

Defining is not running

Running the def block produces no output — it only teaches Python the word. Nothing happens until you call it: sum_to(10).

5.2The stopwatch

Python's time module has a high-resolution clock. Read it before, read it after, subtract.

timing pattern — memorise this
import time

start = time.perf_counter()
result = sum_to(1000000)
elapsed = time.perf_counter() - start

print(f"result {result}, took {elapsed:.4f} seconds")
result 500000500000, took 0.0521 seconds

Your number will differ from mine, and from your own number thirty seconds from now. That is not a flaw in the method — it is the first thing you have to learn to handle.

Use perf_counter, not the clock on the wall

time.perf_counter() is designed for measuring short intervals. Do not use time.time() for benchmarks — it can jump when the system clock adjusts. The number it returns is meaningless on its own; only the difference matters.

5.3One measurement is not a measurement

Your computer is doing dozens of other things while it runs your loop. Time the same call five times and you will get five different answers:

repeat and take the best
import time

def time_it(func, n, repeats=5):
    """Run func(n) several times, return the fastest time in seconds."""
    best = None
    for _ in range(repeats):
        start = time.perf_counter()
        func(n)
        elapsed = time.perf_counter() - start
        if best is None or elapsed < best:
            best = elapsed
    return best

print(f"{time_it(sum_to, 1000000):.4f} s")

This little time_it is the harness the whole course leans on. Every week from here on hands it a function and a size and reads back a trustworthy number, so it is worth understanding line by line now.

Why the fastest run and not the average? Because interference only ever makes code slower. The quickest run is the one that got the least interrupted — the closest thing to the true cost. (Averages are fine too, as long as you say which you used.)

The three ways a benchmark lies
  1. Too small. Anything under a millisecond is mostly noise. Increase n until the run takes at least 0.05 s.
  2. Something else was running. Close the video tab. On Colab, other people's load can bleed in — repeat and take the best.
  3. You measured the wrong thing. Building a million-item test list inside the timed block measures list-building, not your algorithm. Build the data before start.

5.4Your first benchmark table

Now the pattern that every remaining week of this course uses. Same function, several sizes, one table:

benchmark.py
sizes = [100000, 200000, 400000, 800000]

print(f"{'n':>10} {'seconds':>12} {'ratio':>8}")
previous = None
for n in sizes:
    t = time_it(sum_to, n)
    ratio = t / previous if previous else 1.0
    print(f"{n:>10} {t:>12.5f} {ratio:>8.2f}")
    previous = t
n seconds ratio 100000 0.00512 1.00 200000 0.01031 2.01 400000 0.02064 2.00 800000 0.04119 2.00

Read the ratio column, not the seconds column. The seconds depend on your laptop; the ratio is a property of the algorithm. Double the input, double the time: this function's cost grows in a straight line. Next week you will use that column to identify every growth pattern in the course.

5.5Two functions, one job

There is a closed-form formula for the same sum: n(n+1)/2. Same answer, different method:

loop vs formula
def sum_formula(n):
    return n * (n + 1) // 2

for n in [1000, 100000, 10000000]:
    a = time_it(sum_to, n)
    b = time_it(sum_formula, n)
    print(f"n={n:>9}  loop {a:.6f}s   formula {b:.9f}s   x{a/b:,.0f} faster")
n= 1000 loop 0.000041s formula 0.000000160s x256 faster n= 100000 loop 0.004900s formula 0.000000170s x28824 faster n= 10000000 loop 0.512000s formula 0.000000180s x2844444 faster

Look at the formula column: it barely moves. The loop's time grows with n; the formula's does not grow at all. And notice the last column growing too — the gap between the two methods widens with the data. That is the thing this course teaches you to see coming, before you write the slow one.

5.6Going deeper: warm-up, resolution and timeit

You have a working stopwatch. Three refinements separate a rough timing from one you would stake a grade — or a production decision — on.

The first run is a little lie

Time the same call five times in a row and watch the very first number:

watch the first run
import time

data = list(range(1000000))          # built once, before timing

for run in range(5):
    start = time.perf_counter()
    total = sum(data)
    print(f"run {run}: {time.perf_counter() - start:.5f} s")
run 0: 0.01204 s run 1: 0.00781 s run 2: 0.00773 s run 3: 0.00772 s run 4: 0.00774 s

The first run is reliably slower. It pays one-off costs the others do not: the CPU's cache is cold, memory is allocated fresh, and Python may still be settling. That first run is called a warm-up, and serious benchmarks throw it away.

Why our harness already survives this

Look again at time_it: it takes the fastest of the repeats. The cold first run is the slowest, so it is never the minimum — taking the best quietly discards the warm-up for you. If you switch to averaging instead, add an explicit warm-up call before the loop (func(n) on its own, result ignored), otherwise the cold run drags the average up.

How fine is the ruler?

A stopwatch that ticks in whole seconds cannot measure a 3-millisecond event. Every clock has a smallest step it can see, its resolution. Ask Python what yours is:

clock info
import time

info = time.get_clock_info("perf_counter")
print("resolution:", info.resolution, "seconds")
print("monotonic:", info.monotonic)
resolution: 1e-09 seconds monotonic: True

A nanosecond sounds plenty fine, and the raw clock is. But the practical floor is much higher: the overhead of calling a Python function and reading the clock twice is itself hundreds of nanoseconds, and that is your real smallest trustworthy interval. This is the arithmetic behind the rule from §5.3 — if a run lasts only a few times the smallest step you can measure, the trailing digits are decoration. Grow n until the time is a comfortable multiple of the resolution. (monotonic: True confirms the clock only ever moves forwards, which is why a difference can never come out negative.)

Let the standard library do it: timeit

For a very fast snippet — a single arithmetic expression, a one-line call — perf_counter around one call measures mostly its own overhead. The timeit module exists for exactly this: it runs the snippet many times, reports the total, and turns off the garbage collector during timing to remove one source of noise.

timeit for tiny snippets
import timeit

# run sum_to(1000) ten thousand times, get the total, divide back
total = timeit.timeit("sum_to(1000)", globals=globals(), number=10000)
per_call = total / 10000
print(f"{per_call * 1e6:.2f} microseconds per call")
41.30 microseconds per call

Use the right tool for the scale. For the tiny, use timeit; for the doubling experiments in the rest of the course — where each input size takes tens of milliseconds and you want its own number and a ratio — the time_it harness is the better fit. Both read the same underlying perf_counter; they just package it differently.

Why noise is a percentage, not an amount

When the operating system steals the CPU to service the network card, it takes a roughly fixed slice of time — say a quarter of a millisecond. On a 0.5 ms run that is a 50% error; on a 2-second run it is 0.0000125%. The interruption is the same size; what changes is its share of your measurement. That is the deep reason we grow n: not to make the algorithm slower for its own sake, but to make every fixed disturbance shrink to a rounding error relative to the signal.

Report noise the honest way

"It varied by 0.3 ms" tells nobody anything. "It varied by 40% at n = 1 000 but under 1% at n = 1 000 000" tells the whole story — and shows you understand where your numbers are trustworthy and where they are not.

5.7Try it yourself

Task 1 — functions from last week

Turn the linear search from week 4 into def contains(data, target) that returns True or False. Test it on a small list.

Solution
contains.py
def contains(data, target):
    for item in data:
        if item == target:
            return True      # return leaves the function immediately
    return False

print(contains([3, 9, 4], 9), contains([3, 9, 4], 5))
Task 2 — time the worst case

Time contains searching for a missing item in lists of 100 000, 200 000, 400 000 and 800 000 items. Build each list before starting the clock. Report the ratio column.

What you should see

Ratios close to 2.0. A missing item forces a full pass, so the time follows the length of the list exactly. If your ratios are wild (0.4, 6.1), your n is too small or you timed the list construction.

Task 3 — a deliberately dishonest benchmark

Move the line that builds the test list inside the timed region and re-run Task 2. What happens to the numbers, and why is the result meaningless?

Answer

Times jump several-fold and the ratios still look roughly linear — which is exactly what makes it dangerous. You are now measuring "build a list of n items" plus your search, and you can no longer tell which part dominates. Always ask: what exactly is inside my stopwatch?

Task 4 — see the warm-up for yourself

Time sum_to(2000000) eight times in a row with a plain perf_counter loop, printing each run. Then print the fastest and the mean of all eight, and the fastest and the mean of runs 2–8 only (dropping the first). Which statistic changed the most when you dropped run 1, and why?

What to look for

The mean moves noticeably when you drop the cold first run; the fastest barely moves, because the first run was never the minimum in the first place. This is the point of §5.6: taking the best is a free warm-up, but an average needs the first run removed by hand.

Task 5 — perf_counter vs timeit

Measure the cost of one call to sum_to(1000) two ways: once with a single perf_counter pair around one call, and once with timeit.timeit("sum_to(1000)", globals=globals(), number=10000) divided by 10 000. Which gives the steadier answer across three repeats, and why?

Answer

The single perf_counter reading jumps around — a 41-microsecond call is only a few hundred times the measurement overhead, so a single interruption swamps it. The timeit figure is steady, because averaging over 10 000 runs turns each fixed disturbance into a tiny percentage (§5.6, "noise is a percentage"). For calls this fast, timeit is the correct instrument.

Task 6 — relative noise shrinks with n

For n = 1 000 and again for n = 2 000 000, time sum_to(n) ten times and report the spread as a percentage: (slowest − fastest) / fastest × 100. Explain the two numbers using the idea from §5.6.

Expected

The small n shows a large spread (tens of percent or more); the large n shows a small one (a few percent or less). The interruptions are the same size in milliseconds either way — but as a share of a longer run they shrink. This is why we grow n until the run is well clear of the noise before trusting a ratio.

5.8Self-check

A function without a return statement gives back:

It runs happily and hands back None. Printing inside a function is not the same as returning.

Why take the fastest of several runs rather than the slowest?

Other processes, disk activity and garbage collection add time; nothing subtracts it.

Your benchmark reports 0.00002 s for n = 10 and 0.00003 s for n = 20. What should you do?

At this scale you are measuring the clock and the loop overhead, not the algorithm. Grow n until the signal is well above the noise.

The very first of five identical timed runs is noticeably slower than the other four. The usual reason is:

The first call pays start-up costs the rest do not. Taking the minimum already ignores it; an average would need it removed by hand.

You want to measure one call to a snippet that takes about 40 microseconds. The best tool is:

At tens of microseconds a single reading is mostly overhead. timeit repeats the snippet many times so each fixed disturbance becomes a negligible percentage.

5.9Homework

Due before week 6 — about 3 hours
  1. Create AA_Week05.ipynb, define time_it, and solve Tasks 1–6.
  2. Benchmark three functions at four sizes each: sum_to, sum_formula, and contains (worst case). One table each, with a ratio column.
  3. Add a step counter to contains and report both steps and seconds for each size.
  4. Run the warm-up demonstration from §5.6 and Task 4 on your own machine or Colab. Paste your eight run times and state, in one sentence, whether taking the fastest saved you from the cold first run.
  5. Measure the relative noise (Task 6) at your smallest and largest size, and quote both percentages.
  6. Write five sentences: where do the step counts and the seconds agree, and where does the timing wobble? Which of the three ways to lie (§5.3) did you have to guard against, and did growing n shrink your relative noise as §5.6 predicts?

5.10Words from this week

TermMeaning in plain words
functionA named block of work you can call with different inputs.
parameter / argumentThe name in the definition / the value you pass in.
returnHand a value back to whoever called the function, and stop.
benchmarkA measurement of how long code takes, done carefully enough to trust.
perf_counterPython's high-resolution stopwatch; only differences are meaningful.
warm-up runA first, discarded run that pays one-off costs so the timed runs are fair.
resolutionThe smallest interval a clock can distinguish; times near it are untrustworthy.
timeitA standard-library tool that repeats a tiny snippet many times for a steady figure.
noiseVariation in timings caused by everything else the machine is doing.
Where this leads

You can time one input size. Week 6 times several sizes at once and reads the pattern in the numbers — the doubling experiment.