Orchestrated Reasoning for Accurate Calibrated Likelihood Estimation

ORACLE is an advanced AI-powered forecasting system that generates well-calibrated probability estimates for future events. It uses a multi-layered pipeline with multiple AI agents that research, deliberate, debate, and synthesize their estimates to produce accurate and reliable forecasts.

• 5-Layer Forecasting Pipeline: Systematic approach combining research, estimation, debate, risk analysis, and calibration
• Multi-Agent Architecture: 8 diverse AI agents with different analytical perspectives
• Real-Time Web Research: Perplexity and OpenAI deep research integration for up-to-date information
• Adversarial Debate: Agents argue opposing positions to stress-test forecasts
• Black Swan Detection: Identifies low-probability, high-impact tail risks (optional)
• Calibrated Confidence Intervals: Statistical methods for reliable uncertainty quantification
• Multiple Model Support: Choose from Claude, GPT-4, Gemini, DeepSeek, and more via OpenRouter
• Modern Web Interface: Real-time progress tracking, interactive visualizations, dark mode

• Node.js 18+
• npm or yarn
• OpenRouter API key (get one at openrouter.ai)

# Clone the repository git clone <repository-url> cd oracle # Install dependencies npm install # Set up environment variables cp .env.example .env.local # Edit .env.local and add your OPENROUTER_API_KEY # Initialize the database npx prisma db push # Start the development server npm run dev

Open http://localhost:3000 to use the application.

Create a .env.local file with:

# Required OPENROUTER_API_KEY=your_openrouter_api_key_here # Optional DATABASE_URL="file:./dev.db" DEBUG=false

ORACLE processes forecasting questions through 5 specialized layers:

• Parses the forecasting question into key components
• Identifies relevant reference classes and historical precedents
• Conducts web research to gather current information
• Synthesizes a base rate estimate from historical data

8 diverse AI agents independently analyze the question:

• Base Analyst: Neutral, balanced analysis
• Skeptical Analyst: Focuses on why predictions fail
• Constructive Analyst: Focuses on theories of change
• Historical Analyst: Heavy emphasis on base rates
• Quantitative Analyst: Explicit probability calculations
• Plus 3 additional specialized perspectives

• Agents are paired to argue opposing positions
• Multi-round debates stress-test assumptions
• Convergence analysis measures agreement
• Post-debate estimates incorporate new insights

• Generates tail risk scenarios
• Researches each scenario for plausibility
• Calculates aggregate tail risk adjustments
• Identifies high-impact scenarios that could invalidate forecasts

• Applies temperature scaling for calibration
• Uses extremization based on agent agreement
• Generates final probability distribution
• Produces confidence intervals (50% and 90%)
• Creates quality metrics and methodology summary

oracle/ ├── app/ # Next.js App Router pages │ ├── api/ # API routes │ │ └── forecast/ # Forecast CRUD endpoints │ ├── forecast/[id]/ # Forecast detail page │ ├── history/ # Forecast history page │ ├── guide/ # User guide page │ ├── methodology/ # Methodology documentation │ └── settings/ # User settings page ├── components/ # React components │ ├── forecast/ # Forecast-specific components │ ├── ui/ # Reusable UI components │ ├── layout/ # Layout components │ └── settings/ # Settings components ├── lib/ # Core libraries │ ├── oracle/ # Forecasting pipeline │ │ ├── pipeline.ts # Main orchestration │ │ ├── layer1-decomposition.ts │ │ ├── layer2-swarm.ts │ │ ├── layer3-debate.ts │ │ ├── layer4-blackswan.ts │ │ ├── layer5-calibration.ts │ │ ├── prompts.ts # AI prompts │ │ ├── config.ts # Runtime configuration │ │ ├── cache.ts # Response caching │ │ └── error-recovery.ts │ ├── openrouter.ts # OpenRouter API client │ ├── models.ts # Model configurations │ ├── types.ts # TypeScript types │ ├── validations.ts # Zod schemas │ ├── db.ts # Database client │ └── store.ts # Zustand store ├── prisma/ # Database schema │ └── schema.prisma ├── hooks/ # Custom React hooks └── docs/ # Documentation 

POST /api/forecast Content-Type: application/json { "statusQuo": "Current state description (50-5000 chars)", "resolutionConditions": "How the question resolves (20-2000 chars)", "webResearchModel": "perplexity/sonar-reasoning-pro", "reasoningModel": "google/gemini-3-flash-preview", "enableBlackSwan": true }

GET /api/forecast/{id}

POST /api/forecast/{id}/run

GET /api/forecast?limit=10&offset=0&status=COMPLETED

DELETE /api/forecast/{id}

See docs/API.md for complete API documentation.

The pipeline can be configured via lib/oracle/config.ts:

// Search limits layer1MaxSearches: 24, // Max searches in Layer 1 layer4MaxSearches: 40, // Max searches in Layer 4 // Caching enableWebSearchCache: true, enableLLMCache: true, // Timeouts webSearchTimeoutMs: 30000, llmCallTimeoutMs: 120000, forecastTimeoutMs: 600000, // Retry configuration maxRetries: 3, retryBaseDelayMs: 1000,

import { applyFastPreset, applyThoroughPreset, applyBalancedPreset } from '@/lib/oracle/config'; applyFastPreset(); // Faster, fewer searches applyThoroughPreset(); // More comprehensive, slower applyBalancedPreset(); // Default configuration

npm run dev # Start development server npm run build # Build for production npm run start # Start production server npm run lint # Run ESLint npm run db:push # Push schema to database npm run db:studio # Open Prisma Studio npm run db:generate # Generate Prisma client

ORACLE uses SQLite by default with Prisma ORM. To use PostgreSQL:

1. Update prisma/schema.prisma:

datasource db {  provider = "postgresql"  url = env("DATABASE_URL") }

2. Update .env.local:

DATABASE_URL="postgresql://user:password@localhost:5432/oracle"

3. Run migrations:

npx prisma migrate dev

• Specific, time-bound, and objectively verifiable
• Clear resolution criteria
• Sufficient context for research

Examples:

• "Will the Federal Reserve cut interest rates before June 2025?"
• "Will SpaceX successfully launch Starship to orbit by Q2 2025?"

• Vague questions without timeframes
• Subjective or opinion-based questions
• Questions that can't be verified

Examples to avoid:

• "Will the stock market go up?" (no timeframe)
• "Is AI good for society?" (subjective)

• Typical forecast time: 3-8 minutes depending on models
• Token usage: ~50,000-100,000 tokens per forecast
• Caching: Web searches and LLM calls are cached to reduce costs
• Black Swan toggle: Disable to reduce time by ~20%

"OPENROUTER_API_KEY not set"

• Ensure .env.local exists with your API key
• Restart the development server after adding the key

"Forecast stuck in RUNNING"

• Check server logs for errors
• The forecast may have timed out (10 min limit)
• Delete and retry the forecast

"Rate limit exceeded"

• OpenRouter has rate limits per API key
• Wait a few minutes before retrying
• Consider using a paid plan for higher limits

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

This project is licensed under the MIT License - see the LICENSE file for details.

• Built with Next.js and React
• AI models via OpenRouter
• UI components from shadcn/ui
• Inspired by forecasting research from Metaculus and Good Judgment Project