Clinical Data Platform

Traceable clinical research ingestion across studies and sources.

StackPython 3.12 · FastAPI · FHIR · SQLAlchemy 2 · PostgreSQL · Celery · Redis

A FastAPI backend that normalizes CSV and FHIR research data while preserving validation results, provenance, study boundaries and asynchronous import diagnostics.

Business problem

Research teams receive inconsistent source exports. One invalid row must not block valid records, FHIR references must resolve correctly, two sites using the same local patient ID must not collide, repeated uploads must be idempotent for the same target, and researchers must see only patients bound to studies they are authorized to access.

Key engineering decisions

CSV and FHIR parsers only map source formats. A shared pipeline owns normalization, validation, domain services, persistence, study binding, reporting, and provenance. External identity is the pair (source_namespace, external_id). Import idempotency hashes the file checksum together with the target study and namespace.

Reliability

Celery workers and Redis provide queueing, retries, timeouts, and failure diagnostics. Each attempt records completed, partial, or failed status with versioned row- or entry-level errors. SourceRecord history links normalized resources to every import that created or re-observed them. JSON logs, request IDs, liveness, and dependency readiness expose operational state.

Results

The Docker Compose demo starts migrations, FastAPI, PostgreSQL, Redis, and a Celery worker, then imports synthetic CSV and FHIR data and queries patients through a study-restricted researcher key. The release includes 87 unit tests and 7 real-service integration tests.

Technical scope

FHIR-aware data modelling, resilient import design, study-scoped authorization, namespaced identity, idempotency, provenance, PostgreSQL persistence, Celery task operations, auditability, Docker orchestration, migration testing, coverage enforcement, and explicit clinical-system limitations.

CSV ─┐
     ├→ Shared Import Pipeline → Domain Services → PostgreSQL
FHIR ┘                           ↓
                              Provenance / Audit

Redis → Celery Worker → Pipeline