Do NOT open a public GitHub issue for security vulnerabilities.

If you discover a security vulnerability in md-evals, please report it responsibly using one of these methods:

1. Go to the Security tab
2. Click "Report a vulnerability"
3. Fill in the security advisory form with details

This keeps the vulnerability private until we can address it.

Send a detailed report to:

Email: javier@example.com
Subject: [SECURITY] Vulnerability Report: [brief description]

Include:

• Vulnerability description
• Affected version(s)
• Steps to reproduce
• Potential impact
• Suggested fix (if any)

If you prefer additional security, you can encrypt using GPG:

# Contact maintainer for public key gpg --trust-model always --encrypt --recipient <key-id> message.txt

1. Within 48 hours - We'll acknowledge receipt of your report
2. Within 5 days - We'll provide an initial assessment
3. Within 30 days - We'll have a fix ready or timeline for fix
4. Coordinated disclosure - We'll work with you on release timing

We commit to:

✅ Treat all security reports seriously
✅ Maintain confidentiality of reporters
✅ Respond promptly to inquiries
✅ Fix vulnerabilities in timely manner
✅ Provide credit (unless you prefer anonymity)
✅ Keep you informed throughout the process

• Dependencies: Pinned versions in pyproject.toml
• Testing: Comprehensive pytest coverage
• Code Review: All PRs reviewed before merge
• Type Hints: Full type annotations for safety
• Validation: Input validation using Pydantic

• Dependencies: Updates and vulnerability advisories
• Code Quality: Static analysis and linting
• Access Control: GitHub repository settings
• Secrets: No API keys or tokens in code

Be aware of these limitations when using md-evals:

⚠️ API Keys: Store API keys in environment variables, not config files
⚠️ Network: Uses HTTPS for all API calls
⚠️ YAML Parsing: Ensure eval.yaml comes from trusted sources
⚠️ Model Output: LLM responses are untrusted third-party content

When security vulnerabilities are fixed:

1. Patch release is published (e.g., 1.0.1)
2. Security advisory is published on GitHub
3. CHANGELOG documents the fix
4. Dependencies are updated if needed

To stay informed about security updates:

• 👀 Watch the repository for releases
• ⭐ Star the project on GitHub
• 📧 Follow security advisories

Vulnerabilities will be publicly disclosed:

• When a patch is released
• After 90 days if no patch exists
• At reporter's request (with patch available)

We ask reporters to:

• Keep details confidential until patch is released
• Allow time for users to update
• Avoid public discussion of unreleased vulnerabilities

Embargo can be broken if:

• The vulnerability is publicly disclosed elsewhere
• Active exploitation is detected in the wild
• 90 days have passed since initial report

Key dependencies and how we manage them:

• litellm - LLM provider abstraction

  • Version: Pinned, updated regularly
  • Security: Uses HTTPS for all APIs

• pydantic - Data validation

  • Version: Latest minor version
  • Security: Well-maintained, security-focused project

• typer - CLI framework

  • Version: Pinned to stable releases
  • Security: Minimal security surface

• pyyaml - YAML parsing

  • Version: Latest with safe loader
  • Security: Never uses unsafe_load()

See pyproject.toml for full dependency list.

We monitor for vulnerabilities using:

• GitHub Dependabot
• Security advisories
• Community reports

1. Protect API Keys

   # ✅ DO: Use environment variables export GITHUB_TOKEN="your-token" # ❌ DON'T: Commit to version control # ❌ DON'T: Include in eval.yaml

2. Validate Configuration

   • Review eval.yaml before running
   • Trust only configuration from known sources
   • Use version control for configuration

3. Keep Updated

   # Check for updates pip list --outdated # Update md-evals pip install --upgrade md-evals

4. Report Suspicions

   • If you suspect a breach or exploitation
   • Contact security team immediately
   • Do not share details publicly first

We appreciate security researchers following responsible disclosure:

✅ Do:

• Report privately first
• Give us time to fix
• Be detailed in reports
• Use clear communication
• Follow up appropriately

❌ Don't:

• Publicly disclose before fix
• Exploit beyond proof-of-concept
• Demand compensation
• Share details prematurely
• Be hostile or threatening

md-evals does not currently have a bug bounty program. However:

• We deeply appreciate security research
• Reporters receive credit in advisories
• We follow coordinated disclosure practices
• Your contribution strengthens the project

To encourage responsible disclosure:

• We won't pursue legal action for good-faith reporting
• We won't share reporter information
• We protect researchers' privacy

For security-related questions:

📧 Email: javier@example.com
Subject: [SECURITY] Question: [topic]

For other questions:

• 💬 GitHub Discussions
• 📚 Documentation

No security vulnerabilities have been reported yet.

See GitHub Security Advisories for complete history.

• CODE_OF_CONDUCT.md - Community standards
• CONTRIBUTING.md - Contribution guidelines
• CHANGELOG.md - Release information
• LICENSE - MIT License

Thank you for helping keep md-evals secure! 🛡️