Skip to content

tests

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

tests

README.md

🧪 ContextForge Testing Guide

This repository includes a comprehensive test suite organized by test type and purpose:

Category Location Description
Unit tests/unit/ Fast, isolated tests for individual functions, services, models and handlers.
Integration tests/integration/ Happy-path flows that stitch several endpoints together with TestClient.
End-to-End (E2E) tests/e2e/ Full, high-level workflows that drive the running server (admin, federation, client).
Security tests/security/ Security validation: input validation, headers, cookies, middleware.
Performance tests/performance/ Database performance, N+1 detection, and benchmarking.
Fuzz tests/fuzz/ Fuzzing and property-based testing with Hypothesis.
Playwright tests/playwright/ Browser-based UI automation tests.
Migration tests/migration/ Database migration tests (SQLite and PostgreSQL).
Load tests/load/ Load test data generators and verification.
Loadtest tests/loadtest/ Locust-based load testing scenarios.
Differential tests/differential/ Differential testing (comparing implementations). 
tests/ ├── conftest.py # Shared pytest fixtures ├── unit/ # Unit tests (mirrors mcpgateway/ structure) ├── integration/ # Integration tests ├── e2e/ # End-to-end tests ├── security/ # Security validation tests ├── performance/ # Performance and N+1 detection ├── fuzz/ # Fuzzing / property-based tests ├── playwright/ # Browser UI tests ├── migration/ # Database migration tests ├── load/ # Load test generators ├── loadtest/ # Locust load tests ├── differential/ # Differential testing ├── async/ # Async profiling and benchmarks ├── client/ # HTTP client benchmarks ├── manual/ # Manual test cases and plans ├── helpers/ # Test utilities (query_counter, etc.) ├── utils/ # Shared test mocks (RBAC, etc.) └── hey/ # Load test artifacts (ignored by CI)

Quick Commands

Core Testing

Purpose Command
Run unit tests (CI default) make test
Run full suite pytest -q
Unit tests only pytest tests/unit
Integration tests pytest tests/integration
E2E tests pytest tests/e2e
Single module (verbose) pytest -v tests/unit/mcpgateway/test_main.py
Single test method pytest tests/unit/path/test_mod.py::TestClass::test_method
By name substring pytest -k "fragment"
Exclude slow tests pytest -m "not slow"
Run doctests make doctest
Smoke test (container + E2E) make smoketest

Coverage

Purpose Command
HTML coverage report make htmlcov → open docs/docs/coverage/index.html
Full coverage (md + HTML + XML) make coverage
Coverage for specific module pytest tests/unit/mcpgateway/test_main.py --cov=mcpgateway.main --cov-report=term-missing

Specialized Test Suites

Purpose Command
Security tests pytest tests/security
Fuzz / property-based tests pytest tests/fuzz
Performance / N+1 detection make test-db-perf or pytest tests/performance
Migration tests pytest tests/migration
Playwright UI tests pytest tests/playwright (requires Playwright setup)
Load tests (Locust) cd tests/loadtest && locust

Database Performance

Purpose Command
Dev server with query logging make dev-query-log
Tail query log make query-log-tail
Analyze for N+1 patterns make query-log-analyze

Coverage workflow

  1. Spot the gaps

    pytest tests/unit/mcpgateway/test_main.py \ --cov=mcpgateway.main \ --cov-report=term-missing

    Lines listed under Missing are un-executed.

  2. Write focused tests

    Add/extend tests in the relevant sub-folder (unit ➜ fine-grained; integration ➜ flows).

  3. Iterate until the target percentage (or 100 %) is reached.

Test Layout & Naming Conventions

  • Each top-level domain inside mcpgateway/ has a mirrored unit-test package: tests/unit/mcpgateway/<domain>/. Example: mcpgateway/services/tool_service.py → tests/unit/mcpgateway/services/test_tool_service.py.

  • Integration tests live in tests/integration/ and use TestClient, but patch actual DB/network calls with AsyncMock.

  • E2E tests assume a running server and may involve HTTP requests, WebSockets, SSE streams, etc.

  • Security tests validate input sanitization, headers, cookies, and middleware security.

  • Fuzz tests use Hypothesis for property-based testing of schemas, JSON-RPC, and API inputs.

  • Performance tests detect N+1 queries and measure database performance.

  • Migration tests verify Alembic migrations work correctly on SQLite and PostgreSQL.

  • Playwright tests automate browser-based UI testing (requires playwright install).

  • Load tests use Locust for stress testing; data generators live in tests/load/.

  • Log-replay / load-test artifacts are parked in tests/hey/ (ignored by CI).

Fixtures Cheat-Sheet

Fixture Scope Description
test_client function A FastAPI TestClient with JWT auth overridden.
auth_headers function A ready-made Authorization: Bearer ... header.
db_session function Database session for tests requiring persistence.

Additional fixtures are defined in module-level conftest.py files per folder.

Test Markers

Use markers to categorize and filter tests:

Marker Description Example
slow Long-running tests pytest -m "not slow"
ui UI/Playwright tests pytest -m "ui"
api API endpoint tests pytest -m "api"
smoke Smoke tests pytest -m "smoke"
e2e End-to-end tests pytest -m "e2e"

Combine markers: pytest -m "api and not slow"

Makefile Targets

Target Description
make test Run unit tests with coverage.
make doctest Run doctests in modules.
make htmlcov Generate HTML coverage report → docs/docs/coverage/index.html.
make coverage Full coverage (md + HTML + XML + badge + annotated).
make smoketest Container build + simple E2E flow.
make test-db-perf Run database performance tests.
make dev-query-log Dev server with query logging enabled.
make query-log-tail Tail the query log in another terminal.
make query-log-analyze Analyze query log for N+1 patterns.
make lint Static analysis (ruff, mypy, etc.).

Specialized Test Suites

Security Tests (tests/security/)

Validates security controls including:

  • Input validation and sanitization
  • Security headers (CSP, HSTS, etc.)
  • Cookie security attributes
  • Middleware security behavior
pytest tests/security -v

Fuzz Tests (tests/fuzz/)

Property-based testing using Hypothesis:

  • API schema fuzzing
  • JSON-RPC message fuzzing
  • JSONPath expression fuzzing
  • Input validation edge cases
pytest tests/fuzz -v # Or run with more examples: pytest tests/fuzz --hypothesis-seed=0

Performance Tests (tests/performance/)

Database and query performance:

  • N+1 query detection
  • Query count assertions
  • Response time benchmarks
make test-db-perf # Or directly: pytest tests/performance -v

See tests/performance/README.md for detailed profiling instructions.

Migration Tests (tests/migration/)

Database migration validation:

  • SQLite migration paths
  • PostgreSQL migration paths
  • Data integrity verification
pytest tests/migration -v

Requires Docker for PostgreSQL tests. See tests/migration/README.md.

Playwright Tests (tests/playwright/)

Browser-based UI automation:

  • Admin UI flows
  • Authentication workflows
  • API endpoint testing via browser
# Install Playwright browsers first: playwright install # Run tests: pytest tests/playwright -v

See tests/playwright/README.md for setup details.

Load Tests (tests/loadtest/)

Locust-based load testing:

cd tests/loadtest locust -f locustfile.py --host=http://localhost:4444

Open http://localhost:8089 to configure and run load tests.