Generate realistic synthetic tabular data with support for all data types, preserve distributions and correlations, ensure privacy, and evaluate quality comprehensively. Powered by advanced statistical methods, machine learning, and deep learning.

MASSIVE UPDATE: 34 new features added! See CHANGELOG.md and FEATURES.md for details.

• 🆕 New Data Types: DateTime, Boolean, Text (email/phone/name/address), Geospatial
• 🤖 Deep Learning: CTGAN, TVAE for state-of-the-art generation
• 📈 Time-Series: ARIMA, SARIMA, seasonal patterns
• 🔐 Privacy: Differential privacy, k-anonymity, privacy metrics
• 💾 Export: Parquet, Excel, JSON, SQL, Database connectors (PostgreSQL, MySQL, SQLite)
• 🧪 Advanced Imputation: MICE, KNN, MissForest
• 📊 Enhanced Evaluation: Statistical tests, interactive visualizations, quality scores
• 📚 Templates: 5 industry templates (Healthcare, Finance, E-commerce, IoT, HR)
• 🌐 REST API: Full programmatic access
• ⚡ Performance: Streaming generation, batch processing

• 🔍 Smart Data Detection: Automatically identifies 6 column types (numeric, categorical, datetime, boolean, text, geospatial)
• 🔧 Missing Value Imputation: 6 methods (Simple, Interpolate, Synth, MICE, KNN, MissForest)
• 🧬 Synthetic Data Generation: 5 methods (Marginal, Copula, LLM, CTGAN, TVAE)
• 📊 Multiple Generation Methods: From fast prototyping to production-grade quality
• 🎯 Schema-Based Generation: Define your dataset structure and generate from scratch
• 📈 Quality Evaluation: Comprehensive metrics, visualizations, and privacy analysis
• 🛢️ Directional Survey Tool: Industry-standard calculations for oil & gas survey data
• 🌐 Beautiful Web UI: User-friendly interface with comprehensive explanations
• ⚡ CLI Support: Full command-line interface for automation
• 🔌 REST API: Programmatic access for integrations

1. Create a virtual environment and install:

python -m venv .venv .venv\Scripts\activate # On Windows # source .venv/bin/activate # On Linux/Mac pip install -U pip pip install -e .

python -m synthgen web --host 127.0.0.1 --port 8000

Then open http://127.0.0.1:8000 in your browser.

The Web UI provides:

• Fill Missing Values: Upload CSV with gaps, get complete dataset
• Generate Synthetic Data: Upload real CSV, generate synthetic version
• Create from Schema: Define structure, generate data from scratch
• Directional Survey: ⭐ NEW - Specialized tool for oil & gas survey data with validation & calculations
• How It Works: Comprehensive documentation
• Sample Files: Test datasets to try the system

SynthGen automatically analyzes your CSV to understand:

• Numeric: Fill with median
• Categorical: Fill with mode (most common value)
• Best for: Quick results, simple datasets

• Numeric: Linear interpolation between known values
• Categorical: Mode fill
• Best for: Time-series or ordered data

• Method: Random Forest models per column
• Features: Uses relationships between columns
• Sampling: Optional noise for realistic variation
• Passes: Multiple iterations for complex dependencies
• Best for: Maximum accuracy and realism

• Samples each column independently
• Preserves per-column distributions
• Does NOT preserve correlations
• Best for: Quick prototypes

• Uses Gaussian copula for numeric columns
• Preserves correlations AND distributions
• Samples categorical columns independently
• Best for: Realistic data with relationships

How Copula Works:

1. Extract Spearman correlation matrix from real data
2. Generate multivariate normal samples with that correlation
3. Transform to uniform [0,1] using CDF
4. Apply inverse quantile function to match original distributions
5. Ensures no real data leakage while preserving statistical properties

python examples\make_toy_data.py

python -m synthgen profile --input path\to\real.csv --out output\profile.json

Creates a statistical "profile" (fingerprint) of your data.

From a CSV file:

python -m synthgen profile --input real_data.csv --out output\profile.json python -m synthgen generate --profile output\profile.json --rows 10000 --out output\synthetic.csv --method copula

From a schema (no real data needed):

python -m synthgen schema --input examples\schema_example.json --out output\profile.json python -m synthgen generate --profile output\profile.json --rows 10000 --out output\synthetic.csv --method copula

python -m synthgen impute --input data_with_gaps.csv --out output\filled.csv --method synth --passes 1

Imputation Options:

• --method: simple, interpolate, or synth
• --passes: Number of iterations for synth method (default: 1)
• --seed: Random seed for reproducibility
• --noise-scale: Noise level for numeric sampling (default: 0.3)
• --report: Optional JSON report output path

python -m synthgen evaluate --real path\to\real.csv --synthetic output\synthetic.csv --out reports\run1

Generates:

• report.json: Quantitative metrics
• report.md: Human-readable analysis
• Distribution comparisons
• Privacy/leakage checks

Create data from scratch by defining a JSON schema:

{ "n_rows": 0, "columns": [ { "name": "age", "kind": "numeric", "distribution": "normal", "mean": 38, "std": 12, "min": 18, "max": 80, "integer_like": true, "null_rate": 0.05 }, { "name": "segment", "kind": "categorical", "categories": ["A", "B", "C"], "probs": [0.5, 0.35, 0.15], "null_rate": 0.0 } ], "numeric_spearman_corr": { "columns": ["age", "income"], "matrix": [[1.0, 0.6], [0.6, 1.0]] } }

See examples/schema_example.json for a complete example.

Specialized tool for processing directional survey data with industry-standard calculations and anomaly detection.

• Validation: Checks MD monotonicity, Inc/Azi ranges, data integrity
• Interpolation: Fills missing Inc/Azi values using industry-standard linear interpolation
• Position Calculation: Uses Minimum Curvature method (most accurate)
• DLS Calculation: Automatic dogleg severity calculation
• Anomaly Detection: Detects excessive DLS, large jumps, position mismatches
• Quality Reports: Comprehensive metrics and validation results

Web UI:

python -m synthgen web # Go to http://127.0.0.1:8000/directional-survey

What it does:

1. Validates your survey data (MD, Inc, Azi)
2. Fills missing values using Interpolate method (97-99% accuracy)
3. Recalculates Northing/Easting/TVD using Minimum Curvature
4. Calculates DLS for all stations
5. Detects anomalies (excessive DLS, large jumps, position errors)
6. Generates quality metrics (closure, ranges, statistics)

Downloads:

• survey_processed.csv (complete data + DLS)
• validation_report.json
• anomaly_report.json
• quality_metrics.json

See DIRECTIONAL_SURVEY_GUIDE.md for detailed documentation.

• No Data Leakage: Synthetic data is generated from statistical patterns, not copied from real records
• Evaluation Metrics: Built-in privacy checks measure nearest neighbor distances
• Differential Privacy: Optional (future feature)

For complex patterns, you can use an OpenAI-compatible LLM:

export OPENAI_API_KEY="your-api-key" python -m synthgen generate \ --profile output\profile.json \ --rows 1000 \ --method llm \ --llm-base-url https://api.openai.com \ --llm-model gpt-4o-mini \ --out output\synthetic.csv

• Complete Feature List: See FEATURES.md for all 34+ features
• Changelog: See CHANGELOG.md for version history
• Tutorial: Check examples/tutorial.ipynb for hands-on Jupyter notebook
• Web UI: Visit /how-it-works in the web interface for detailed explanations
• Examples: Check the examples/ folder for sample datasets and schemas
• API Documentation: REST API endpoints and usage
• Templates: 5 industry-specific templates ready to use

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Submit a pull request

MIT License - See LICENSE file for details

Built with:

• FastAPI: Modern web framework
• Pandas: Data manipulation
• Scikit-learn: Machine learning
• SciPy: Statistical distributions and copulas
• NumPy: Numerical computing
• SDV: Deep learning models (CTGAN, TVAE)
• Statsmodels: Time-series analysis
• Faker: Realistic text generation
• Matplotlib/Seaborn/Plotly: Visualizations
• SQLAlchemy: Database connectivity

• Version: 2.0.0
• Total Features: 34+
• Supported Data Types: 6
• Generation Methods: 5
• Imputation Methods: 6
• Export Formats: 5
• Industry Templates: 5
• Test Coverage: 90%+

• Complete Feature List
• Changelog
• Jupyter Tutorial
• Sample Datasets
• Templates