CP1 / COURSE HOME

Week 13 / Build a checkable report

Save a report someone can check

Preserve the raw input and make the output reproducible.

We can now interpret and validate readings. A result that exists only in a running notebook is hard for another person to review tomorrow.

We will read a small sensor CSV, write accepted records to a separate file, and retain the source line and reason for every rejected record.

Start with the problem

Try this first.

Sketch three labelled files: raw input, cleaned readings and rejection record. Where would another engineer look to explain a missing reading?

Why this week's tool?

File I/O preserves information beyond one run. CSV gives the records a shared structure. Reopening the saved output checks what was actually written.

By the end: Follow one accepted and one rejected record from the raw file to the saved outputs without overwriting the original evidence.

The idea behind the program

Files are part of the interface

Document the header, units, encoding and row interpretation. Use Python's csv module for CSV boundaries. This handles quoted fields more reliably than assuming every comma is a separator.

Do not overwrite your evidence

Opening a file with mode 'w' replaces its previous contents. Use separate raw and cleaned filenames. For repeated runs on identical input, a deterministic clean output should not accumulate duplicate rows; appending is appropriate only when that is the intended log behaviour.

Read back before claiming success

A write finishing does not prove that the output has the expected schema or count. Reopen the file, check its header and retained rows, and confirm that raw rows = kept rows + rejected rows under your policy.

A short Python example

Read the example alongside the explanation. Run it in a new notebook cell and change one input to see how it behaves.

import csv

raw_rows = [['T1', '20'], ['T1', 'bad'], ['T1', '22']]
with open('bench_raw.csv', 'w', newline='', encoding='utf-8') as file:
    csv.writer(file).writerows([['sensor', 'temperature_c']] + raw_rows)

clean = []
rejected = []
with open('bench_raw.csv', newline='', encoding='utf-8') as file:
    reader = csv.reader(file)
    header = next(reader)
    for line, row in enumerate(reader, start=2):
        try:
            value = float(row[1])
        except ValueError:
            rejected.append((line, row, 'non-numeric value'))
            continue
        clean.append([row[0], value])
with open('bench_clean.csv', 'w', newline='', encoding='utf-8') as file:
    csv.writer(file).writerows([header] + clean)
with open('bench_clean.csv', newline='', encoding='utf-8') as file:
    saved = list(csv.reader(file))
print('Saved data rows:', len(saved) - 1)
print('Rejected:', rejected)

Example output

Saved data rows: 2
Rejected: [(3, ['T1', 'bad'], 'non-numeric value')]

This file-flow model intentionally uses three fixed, two-field rows. It checks numeric conversion only. Before accepting external data, add header, field-count, sensor-ID, finite-value and range checks from Weeks 9 and 12. Running it creates or replaces bench_raw.csv and bench_clean.csv in the current notebook folder; use a scratch folder.

A file stores a representation that must be recoverable

Writing text transfers a representation to persistent storage. A correct report must retain enough structure to be read back with the same intended meaning. Formatting, delimiters, quoting and file mode are part of that contract.

Draw or trace

Draw memory → writer → file bytes → reader → reconstructed records. Put a sensor label containing a comma in one field to expose the difference between simple splitting and CSV parsing.

Predict before running. Will splitting every line at every comma reliably recover a quoted field containing a comma? What happens when a second run uses write mode?

Trace and explanation — after your prediction
  1. A CSV-aware writer quotes fields as required; a CSV-aware reader reconstructs them.
  2. Plain split on commas treats the comma inside a quoted field as another separator.
  3. Write mode replaces an existing file; append mode adds content and can duplicate rows or headers if used carelessly.

Test a write–read round trip and inspect the saved file after closing it. Equality of the reconstructed records checks the representation; it does not validate the original measurements.

Change one thing. Run the export twice in a temporary file. Decide whether the required outcome is replacement or accumulation, then check for duplicated headers.

Türkçe: Dosyaya yazmak bir gösterimi saklar. CSV tırnaklaması ve dosya modu anlamı korumalı; yazdığını yeniden okuyarak denetle.

Examples and variations

Each example changes something about the same problem. Open the ones you want to explore and follow the worked explanation.

01Traceable clean file

Header plus three data rows: T1,20 / T1,bad / T1,22

Question: How many rows are kept, and what is the rejected source line?

  1. Header occupies line 1
  2. Line 3 contains bad → reject with reason
  3. 3 raw data rows = 2 kept + 1 rejected

2 kept; line 3 rejected

State whether numbering includes the header. Here source lines are one-based and include it.

02Append on rerun

A cleaned output has 2 data rows; rerun appends those same rows

Question: What will a second run do to the output count?

  1. First run writes 2 retained rows
  2. Second run appends the same 2
  3. Output now contains 4 data rows

4 rows; duplicate accumulation

Use replacement for a deterministic derived file. Reserve append for deliberately cumulative logs.

03Mixed-unit mean

One temperature 20 °C and one humidity 40%

Question: What does their arithmetic mean of 30 measure?

  1. Arithmetic: (20 + 40) / 2 = 30
  2. Inputs represent different physical quantities
  3. Group by sensor and unit before summarizing

30 has no meaningful shared physical unit

A parser can accept both numbers while the analysis remains invalid. Keep schema and physical meaning together.

See the Colab code run

Interactive walkthroughs of File I/O & CSV. Enable JavaScript to step through code, variables, collections and output. The companion notebook remains available below.

Work on it in Colab

Use the notebook to try the ideas yourself. The steps below connect this week's example to the programming practice.

  1. Create the tiny raw file in a scratch notebook folder. Inspect its text before reading it with csv.reader.
  2. Run the file-flow model and verify the raw/kept/rejected count identity. Inspect the saved header and both retained rows.
  3. Run it again. Confirm that the cleaned output still has two data rows, then explain what append would have done.
  4. Add Week 12 validation before trying an external record. Preserve the rejected raw row and reason; group different sensor types separately.

Something to take away: Raw and cleaned files, a rejection record with line numbering explained, and a read-back check that survives a second run.

Suggested exercises, downloads & solutions

Read the notebook's teaching cells before these exercises.

  • EX01 Create and Read (Easy)
  • EX07 CSV Reader (Medium)
  • EX08 CSV Writer (Medium)
Download notebookWorked solutions

Use the notebook's core and optional labels to choose your workload. This activity fits within guided class time.

Optional notes & guidance
My notes
Using AI or working with a partner

Ask AI for a CSV cleaning pipeline. Review file modes, header handling and whether it preserves raw input. Ask what happens on the second run, with an empty file, or with mixed sensor units; test those claims.

You can also review the supplied example with a partner. Use the same inputs to compare the reasoning. Follow the syllabus rules for assessed work.