Time-Aware Synthetic Data Platform

Schema-aware synthetic data for databases, APIs, and pipelines. Realistic, relational, privacy-safe.

Quick Start • Demo • Scenarios • Workflows • Docs • Walkthrough • Contributing

Data Forge generates business-valid, cross-table, time-consistent, privacy-safe synthetic data—not hand-written fixtures, but test-ready data that respects schemas, foreign keys, business rules, and optional anomaly injection. Built for demos, UAT, integration testing, and pipeline development.

Open-source • Python backend • Next.js frontend • Local-first • Made for data engineers

cd data-forge uv sync

If uv is unavailable, use a standard virtual environment:

python -m venv .venv .venv\Scripts\python -m pip install --upgrade pip .venv\Scripts\python -m pip install -e ".[dev]"

Optional extras:

• uv sync --extra warehouse or .venv\Scripts\python -m pip install -e ".[warehouse]"
• uv sync --extra streamlit or .venv\Scripts\python -m pip install -e ".[streamlit]"
• uv sync --extra all or .venv\Scripts\python -m pip install -e ".[all]"

uv run data-forge generate --pack saas_billing --scale 1000 -o output -f parquet uv run data-forge generate --pack ecommerce --scale 2000 -o output -f sql

# Terminal 1: API uv run uvicorn data_forge.api.main:app --reload --port 8000 # Terminal 2: Frontend cd frontend && npm run dev

Open http://localhost:3000. See docs/demo-walkthrough.md for a step-by-step walkthrough.

docker compose up --build

This starts:

• API on http://localhost:8000
• Frontend on http://localhost:3001

Optional local Postgres stack for adapter workflows:

docker compose --profile db up --build

One-command demo (no cloud credentials):

make demo-data # or: ./scripts/run_demo.ps1 (Windows) / ./scripts/run_demo.sh (Linux/macOS)

Outputs go to demo_output/. Start the API and frontend to inspect runs and artifacts in the UI.

• Scenarios — Save configs from Advanced or the Create Wizard; load in wizard or Advanced; update or save-as-new; edit metadata on the scenario detail page. Import from examples/scenarios/.
• Runs — Every generation or benchmark creates a run (history, timeline, logs, artifacts). Compare two runs side-by-side (config, summary, benchmark, raw diff).
• Artifacts — Datasets, event streams, dbt seeds, GE suites, DAGs; filter by run or type.

1. Generate a dataset — Wizard or Advanced → pick pack or custom schema (Schema Studio) → set scale/options → run. Output in Runs and Artifacts.
2. Run simulation — Advanced → Pipeline Simulation → enable event streams, set density/pattern/replay → run.
3. Benchmark warehouse load — Advanced → Benchmark → profile and scale preset → run. View throughput and duration on run detail.
4. Compare runs — Runs → open a run → “Compare with another run”, or /runs/compare. Use raw JSON diff for debugging.
5. Save scenario — After configuring, “Save as scenario” (or “Update scenario” if loaded). Reuse from Scenarios or wizard.

data-forge/ ├── src/data_forge/ # Backend │ ├── api/ # FastAPI app, routers, stores, middleware │ ├── models/ # Schema, config, generation, manifest │ ├── engine.py # Core run_generation, export_result │ ├── schema_ingest/ # SQL, JSON Schema, OpenAPI parsers │ ├── rule_engine/ # YAML business rules │ ├── generators/ # Primitives, distributions, FK resolution │ ├── adapters/ # SQLite, DuckDB, Postgres, Snowflake, BigQuery │ ├── exporters/ # CSV, JSON, Parquet, SQL │ ├── domain_packs/ # Pre-built schemas and rules │ ├── simulation/ # Event streams, time patterns │ └── services/ # Run, scenario, lineage, retention, metrics ├── frontend/ # Next.js product UI (wizard, runs, Schema Studio) ├── tests/ # Pytest backend tests ├── frontend/e2e/ # Playwright E2E tests ├── examples/scenarios/ # Example scenario JSONs ├── scripts/ # validate_all.*, run_demo.* ├── docs/ # Architecture, API reference, testing, CI, security └── .github/workflows/ # CI (backend, frontend, E2E) 

• CLI — data-forge generate, benchmark, validate, reconcile, packs; full control from the shell.
• API — Start runs, list runs/artifacts/scenarios, compare runs, preflight, benchmark. Local JSON persistence.
• UI — Wizard and Advanced config, run history and detail, scenario library, artifact browser, Schema Studio (custom schemas, validation, version diff), validation center, run comparison.
• CI — GitHub Actions: backend ruff, mypy, pytest; frontend type-check, tests, build; Playwright E2E. Local: make validate-all. See docs/ci-cd.md.
• API — POST /api/runs/generate, GET /api/runs, GET /api/runs/compare, POST /api/benchmark, GET /api/scenarios, GET /api/artifacts, POST /api/preflight.

• Simulation: Event streams (e.g. order lifecycle, payments), time patterns, replay modes. Packs: ecommerce, fintech, logistics, IoT, social, saas_billing.
• Benchmark: Scale presets (small ~10k → xlarge ~10M), workload profiles, throughput (rows/s), duration, memory estimates.

uv run data-forge benchmark --pack saas_billing --scale 5000 --iterations 3 --output-json bench.json

make validate-all # or: scripts/validate_all.ps1 | scripts/validate_all.sh

Steps: backend ruff, mypy, pytest; frontend typecheck, unit tests, build. E2E: make e2e or cd frontend && npm run e2e. See CONTRIBUTING.md and docs/testing.md.

For the full docs index, versioning, and release checklist: docs/INDEX.md, docs/versioning.md, docs/release-checklist.md. See CHANGELOG.md for version history.

UI screenshots and demo assets: docs/screenshots/. Target filenames and checklist: docs/screenshots/SCREENSHOT-CHECKLIST.md.

We welcome contributions. CONTRIBUTING.md covers setup, full validation (make validate-all, E2E), and adding scenarios/packs/tests. CODE_OF_CONDUCT.md and SECURITY.md for community and security.

• Local-first: No cloud deployment; run API and frontend locally or in CI. Storage is file-based (or optional SQLite) for runs and scenarios.
• E2E: Local Playwright runs auto-start API + frontend through frontend/playwright.config.ts; CI starts both explicitly. See docs/testing.md.
• Build: On some environments (e.g. OneDrive-synced folders), npm run build may fail with EPERM; run from a non-synced path or close other tools. Type-check and unit tests still validate the codebase.
• Maturity: Data Forge is open-source and actively developed; some integrations and adapters are evolving. Check docs/architecture-current-state.md and docs/INDEX.md for current scope.

MIT.