Independent project · Case study · Automation + Data

Quality rules before business metrics.

Sales Data Pipeline

A tested Python pipeline that turns imperfect retail CSV exports into validated reports without silently losing bad records or publishing incomplete outputs.

  • Python 3.10+
  • pandas
  • pytest
  • Coverage.py
  • Ruff
  • GitHub Actions
Source code Documentation

At a glance

Problem
Imperfect retail exports
Capability
Validated reports
Run
Reproducible sample

A decision that could have gone the other way

The deliberately messy sample rejects more rows than it accepts, eight against seven. A sample tuned to look clean would have demonstrated nothing. This one exercises the pipeline's failure path, and every rejected row leaves with its reason attached.

Business problem

Retail CSV exports may contain duplicate order IDs, malformed dates, missing customer or product fields, invalid quantities, negative prices, inconsistent statuses, and whitespace errors. Reporting directly from those rows can overstate revenue and corrupt customer, product, category, and monthly summaries.

Key engineering decisions

Structural failures such as missing columns stop the run, while row-level failures are accumulated in ValidationResult and written to rejected_orders.csv with one or more explicit reasons. Accepted records are normalized before transformation. Monetary values use integer cents at the calculation boundary rather than binary floating point.

Reliability

Counts, units, gross revenue, and recognized revenue are reconciled across outputs. A full report set is written to an isolated staging directory and replaces the previous output only after every artifact succeeds. Strict mode turns unacceptable data quality into exit code 1, while input/configuration and export failures use distinct codes.

Results

The committed sample run processes 15 records: 7 accepted and 8 rejected. It writes seven artifacts, including cleaned and rejected datasets, customer, product, category, and monthly summaries, plus pipeline_summary.json with run metadata, quality rates, issue counts, and business metrics.

Technical scope

Data-quality engineering, pandas transformations, auditable rejection handling, exact monetary logic, cross-table reconciliation, atomic file publishing, command-line contracts, logging, pytest coverage, Ruff checks, and a Python 3.10 / 3.12 CI matrix.

Engineering takeaway

Design the failure path first

Bad rows and failed exports must be observable.

Multiple rejection reasons, integer-cent calculations, reconciliation and atomic output replacement keep every sample result explainable.

  • Data quality
  • Reconciliation
  • Atomic publish

Design rationale

Rejected rows remain visible and a staging directory prevents partial exports from replacing the last complete report set.

Implementation status

Implemented

  • Validation and rejection reports
  • Revenue summaries and reconciliation
  • Atomic output replacement

All selected work