Test Suite Organization

This directory contains a comprehensive test suite organized for different CI/CD scenarios. The tests are carefully separated to ensure safe CI execution while still providing real API validation.

🎯 Test Categories

1. Unit Tests (unit marker)

• Purpose: Fast, isolated tests with no external dependencies
• CI Safe: ✅ Always safe to run
• Cost: 🆓 Free
• When to run: Every commit, every PR
• Examples: Model validation, utility functions, core logic

2. Integration Tests (integration marker)

• Purpose: Component integration with mocked external services
• CI Safe: ✅ Safe when excluding real_api marker
• Cost: 🆓 Free (uses mocks)
• When to run: Every commit, every PR
• Examples: Service integrations with mocked APIs

3. Real API Tests (real_api marker)

• Purpose: End-to-end validation with actual API services
• CI Safe: ❌ Only run manually or on schedule
• Cost: 💰 Costs money (API calls)
• When to run: Manual trigger, scheduled validation
• Examples: OpenAI API calls, AWS Bedrock integration

🚀 Running Tests

Quick Start

# Run all safe tests (recommended for CI)
python run_tests.py --mode mock

# Run only unit tests (fastest)
python run_tests.py --mode unit

# Run integration tests with mocks
python run_tests.py --mode integration

CI/CD Modes

# CI Safe modes (no API calls, no costs)
python run_tests.py --mode unit           # Unit tests only
python run_tests.py --mode integration    # Integration tests with mocks
python run_tests.py --mode mock           # All safe tests (default)
python run_tests.py --mode all-safe       # All safe tests (explicit)

# Manual/Scheduled modes (cost money)
python run_tests.py --mode real-api       # Real API tests (with confirmation)
python run_tests.py --mode all            # Everything (with confirmation)

Direct pytest Usage

# Safe for CI (no API calls)
pytest -m "not real_api"                  # All safe tests
pytest -m "unit"                          # Unit tests only
pytest -m "integration and not real_api"  # Integration tests with mocks

# Costs money (manual only)
pytest -m "real_api"                      # Real API tests
pytest -m "openai_integration"            # OpenAI specific tests
pytest -m "bedrock_integration"           # Bedrock specific tests

🔧 GitHub Actions Integration

CI Workflow (Automatic)

File: .github/workflows/ci-tests.yml

# Runs on every push/PR - SAFE
- Push to main/develop
- Pull requests
- Multiple Python versions
- Coverage reporting
- No API calls, no costs

Real API Workflow (Manual)

File: .github/workflows/real-api-tests.yml

# Manual trigger only - COSTS MONEY
- workflow_dispatch with confirmation
- Optional scheduled runs
- Uses secrets for API keys
- Cost acknowledgment required

📊 Test Organization

Directory Structure

tests/
├── test_*.py                    # Main test files
├── api/                         # API-specific tests
├── core/                        # Core functionality tests
├── services/                    # Service layer tests
├── utils/                       # Utility tests
├── conftest.py                  # Test configuration
├── README.md                    # This file
└── README_REAL_API_TESTS.md     # Real API test details

Test File Organization

File	Type	Markers	Description
test_models.py	Unit	unit	Data model validation
test_auth.py	Unit	unit	Authentication logic
test_chat.py	Mixed	integration, some real_api	Chat functionality
test_bedrock_chat.py	Mixed	integration, some real_api	Bedrock integration
test_real_api_integration.py	Real API	real_api	Comprehensive API tests

🔒 Safety Features

1. Marker Protection

• Tests with real_api marker only run when explicitly requested
• Default test runs exclude expensive tests
• Clear separation between mock and real tests

2. Cost Warnings

• Interactive prompts before running expensive tests
• Clear cost estimates for different test modes
• Confirmation required for real API tests

3. CI/CD Safety

• GitHub Actions workflows separated by cost/safety
• Real API tests require manual trigger and confirmation
• Environment variable validation before API calls

📋 Test Markers Reference

Marker	Purpose	CI Safe	Cost	Usage
unit	Fast unit tests	✅	🆓	-m unit
integration	Integration with mocks	✅	🆓	-m integration
real_api	Real API calls	❌	💰	-m real_api
openai_integration	OpenAI specific	❌	💰	-m openai_integration
bedrock_integration	Bedrock specific	❌	💰	-m bedrock_integration
slow	Slow tests	✅	🆓	-m "not slow"

🎮 Development Workflow

During Development

# Quick validation
python run_tests.py --mode unit --verbose

# Full local validation (safe)
python run_tests.py --mode mock --verbose

# Before committing
python run_tests.py --mode all-safe

Before Release

# Validate real API integration (costs money)
python run_tests.py --mode real-api

CI/CD Pipeline

# Automated (safe)
- Every commit: unit tests
- Every PR: all safe tests
- Merge to main: full safe test suite

# Manual (costs money)  
- Release validation: real API tests
- Weekly scheduled: comprehensive API tests

🛡️ Best Practices

For CI/CD

1. Never run real API tests automatically - always require manual trigger
2. Use cost confirmations - require explicit acknowledgment of costs
3. Set spending limits - configure API key spending limits
4. Monitor usage - set up billing alerts for API usage
5. Separate environments - use different API keys for testing vs production

For Development

1. Start with unit tests - write fast, isolated tests first
2. Mock external dependencies - use mocks for integration tests
3. Validate with real APIs sparingly - only when necessary
4. Document cost implications - clearly mark expensive tests
5. Use minimal API calls - keep real API tests lightweight

For Testing

1. Layer your tests - unit → integration → real API
2. Mark tests appropriately - use correct markers for each test
3. Handle credentials safely - never commit API keys
4. Test error scenarios - include negative test cases
5. Monitor test costs - track spending on API calls

🔍 Troubleshooting

Common Issues

1. Tests not running in CI
   • Check that you’re using safe markers (not real_api)
   • Verify GitHub Actions workflow configuration
2. Real API tests failing
   • Check API credentials in environment/secrets
   • Verify API key permissions and limits
   • Check service availability and quotas
3. Unexpected costs
   • Ensure real API tests have real_api marker
   • Check that CI workflows exclude expensive tests
   • Review API usage dashboards

Debugging Commands

# Check test discovery
pytest --collect-only

# Dry run without execution
pytest --collect-only -m real_api

# Run with maximum verbosity
python run_tests.py --mode unit --verbose -v

# Check marker filtering
pytest -m "not real_api" --collect-only

📈 Continuous Improvement

Metrics to Track

• Test execution time (aim for fast CI)
• API usage costs (minimize for validation)
• Test coverage (maintain high coverage with safe tests)
• Flaky test rates (especially for real API tests)

Regular Reviews

• Weekly: Review test execution times
• Monthly: Review API costs and usage patterns
• Quarterly: Review test organization and effectiveness
• Before releases: Validate real API functionality

This organization ensures safe, fast CI execution while maintaining comprehensive validation through carefully managed real API testing.