Sales Data Pipeline

A tested Python pipeline that turns imperfect retail CSV exports into validated reports without silently losing bad records or publishing incomplete outputs.

Business problem

Retail CSV exports may contain duplicate order IDs, malformed dates, missing customer or product fields, invalid quantities, negative prices, inconsistent statuses, and whitespace errors. Reporting directly from those rows can overstate revenue and corrupt customer, product, category, and monthly summaries.

Key engineering decisions

Structural failures such as missing columns stop the run, while row-level failures are accumulated in ValidationResult and written to rejected_orders.csv with one or more explicit reasons. Accepted records are normalized before transformation. Monetary values use integer cents at the calculation boundary rather than binary floating point.

Reliability

Counts, units, gross revenue, and recognized revenue are reconciled across outputs. A full report set is written to an isolated staging directory and replaces the previous output only after every artifact succeeds. Strict mode turns unacceptable data quality into exit code 1, while input/configuration and export failures use distinct codes.

Results

The committed sample run processes 15 records: 7 accepted and 8 rejected. It writes seven artifacts, including cleaned and rejected datasets, customer, product, category, and monthly summaries, plus pipeline_summary.json with run metadata, quality rates, issue counts, and business metrics.

Technical scope

Data-quality engineering, pandas transformations, auditable rejection handling, exact monetary logic, cross-table reconciliation, atomic file publishing, command-line contracts, logging, pytest coverage, Ruff checks, and a Python 3.10 / 3.12 CI matrix.

Raw CSV
  ↓
Validation ── Accepted / Rejected + reasons
  ↓
Transform → Reconciliation → Atomic Reports