Benchmark Methodology

This document outlines the comprehensive methodology used to benchmark Apollo RAG’s performance characteristics, ensuring transparent, reproducible, and fair measurements.

Overview

Apollo’s benchmarking methodology is designed to provide transparent, reproducible performance measurements across key RAG system dimensions: latency, throughput, accuracy, and resource utilization. All benchmarks are conducted in a controlled environment with standardized test data and clearly documented hardware specifications.

Benchmark Objectives

• Performance Characterization: Establish baseline metrics for query latency, inference speed, and system throughput
• Optimization Validation: Quantify the impact of architectural optimizations (KV cache preservation, parallel initialization, multi-tier caching)
• Resource Profiling: Measure GPU, CPU, and memory utilization patterns under various load conditions
• Accuracy Baselines: Establish retrieval quality and answer relevance metrics

Test Environment

All benchmarks were conducted in a controlled environment with consistent hardware and software configurations.

Hardware Specifications

Software Configuration

Model Configurations

Primary Model (llama.cpp):

• Model: Meta-Llama-3.1-8B-Instruct
• Quantization: Q5_K_M (5-bit mixed quantization)
• Context Window: 8192 tokens
• GPU Layers: 33 (full offload)
• Batch Size: 512
• Temperature: 0.0 (deterministic)
• VRAM Usage: ~5.4GB

Embedding Model (CPU):

• Model: BAAI/bge-large-en-v1.5
• Dimensions: 1024
• Device: CPU (PyTorch sm_120 workaround)
• Batch Size: 64 documents
• Latency: ~50ms per query

Reranker Model (GPU):

• Model: BAAI/bge-reranker-v2-m3
• Device: CUDA
• Batch Size: 32 documents
• Latency: ~60ms for 32 docs (85% faster than LLM reranking)

Data Corpus

Document Collection

The benchmark corpus consists of 100,000 diverse documents spanning multiple domains and complexity levels, designed to simulate real-world RAG workloads.

Total Corpus Statistics:

• Total Documents: 100,000
• Total Chunks: ~1.2 million (avg 1024 tokens, 128 overlap)
• Index Size: 4.8GB (dense + sparse vectors)
• Embedding Dimension: 1024 (BGE-large)
• Vector Database: Qdrant with HNSW indexing

Query Set

Benchmark Query Collection: 1,000 representative queries across three complexity tiers:

• Simple Queries (40%): Direct factual questions requiring single-document retrieval

  • Example: “What is the capital of France?”
  • Expected Latency: 8-15 seconds

• Moderate Queries (40%): Multi-faceted questions requiring hybrid search

  • Example: “Compare Python and JavaScript for web development”
  • Expected Latency: 10-18 seconds

• Complex Queries (20%): Ambiguous or multi-step reasoning requiring advanced retrieval

  • Example: “What are the implications of quantum computing on current encryption standards?”
  • Expected Latency: 15-25 seconds

Metrics Definitions

1. Latency Metrics

End-to-End Query Latency: Total time from API request reception to response completion.

Stage Breakdown:

Total Latency = Security + Cache Lookup + Retrieval + Embedding +
                Vector Search + Reranking + Generation + Confidence Scoring

Example breakdown for cache miss (simple mode):

• Security Checks: less than 5ms
• Cache Lookup: 0.86ms (Redis query cache)
• Query Embedding: 50ms (CPU, L2 cache miss)
• Vector Search: 100ms (Qdrant HNSW)
• LLM Generation: 8-12s (80-100 tok/s)
• Confidence Scoring: 500ms (parallel)
• Total: ~9-13s

2. Throughput Metrics

Inference Throughput: Token generation rate during LLM inference.

3. Accuracy Metrics

Retrieval Quality: Relevance of retrieved documents to user queries.

Confidence Scoring Formula:

confidence = 0.30 * retrieval_quality +
             0.40 * answer_relevance +
             0.30 * source_consistency

4. Resource Utilization

Testing Approach

Benchmark Phases

Phase 1: Cold Start Testing

• Measure system initialization time (parallel component loading)
• Target: less than 30 seconds from container start to first query ready
• Components: Embeddings, LLM, Vector DB, Cache, Reranker

Phase 2: Warm System Testing

• Execute 100 diverse queries with empty cache
• Measure baseline latency without cache optimization
• Record per-stage timing breakdowns

Phase 3: Cache Effectiveness Testing

• Re-run Phase 2 queries to measure cache hit benefits
• Target: 98% latency reduction on exact matches (less than 1ms vs 50-100ms)
• Validate semantic cache matching (0.95 similarity threshold)

Phase 4: Load Testing

• Ramp up concurrent users (1 → 5 → 10 → 20)
• Measure throughput degradation and latency increase
• Identify bottlenecks (GPU inference, Redis contention)

Phase 5: Optimization Validation

• A/B test individual optimizations:
  • KV cache preservation: 40-60% speedup validation
  • Parallel initialization: 3.4x startup speedup validation
  • BGE reranker: 85% faster than LLM reranking validation

Baseline Measurements

Startup Performance

Parallel Initialization Groups:

• Group 1 (parallel): embedding_cache, embeddings, bm25_data, llm → ~12s
• Group 2 (parallel): cache_manager, vectorstore, bm25_retriever, llm_warmup → ~8s
• Group 3 (parallel): metadata, conversation_memory, confidence_scorer → ~3s

Query Latency Baselines

Cache Hit Rate Over Time:

• Initial 100 queries: 0% hit rate (cold cache)
• After 500 queries: 45% hit rate
• Steady state: 60-80% hit rate (depends on query diversity)

Retrieval Accuracy Baselines

Fairness Considerations

Hardware Limitations

• RTX 5080 Constraint: PyTorch sm_120 incompatibility forces CPU embeddings (50ms overhead per query)
• Single GPU: No multi-GPU benchmarks; results reflect single-device throughput
• WSL2 Overhead: ~5-10% performance penalty vs native Linux (Docker/WSL2 virtualization)

Benchmark Design Choices

Excluded Scenarios

The following scenarios are not included in baseline benchmarks but may be added in future testing:

• Multi-modal queries (images + text)
• Multi-language retrieval (non-English corpora)
• Streaming vs batch processing (SSE vs synchronous)
• Multi-turn conversations (10+ exchange history)
• Cross-document reasoning (requires more than 5 sources)

Limitations

Known Bottlenecks

• CPU Embeddings: RTX 5080 PyTorch incompatibility adds 50ms per query (embedding generation)
• Single-threaded Inference: llama.cpp requires thread-safe single executor (ThreadPoolExecutor with max_workers=1)
• Redis Single-instance: Distributed caching not tested; production may require Redis Cluster
• No Multi-GPU: Benchmarks reflect single-GPU throughput; model sharding not implemented

Query Complexity Bias

• Simple queries (40% of test set) are over-represented vs real-world distribution
• Complex reasoning queries (20%) may underestimate tail latencies
• Benchmark queries are pre-vetted; production includes typos, ambiguity, adversarial inputs

Document Corpus Limitations

• 100K documents is mid-scale; enterprise deployments may exceed 1M+ documents
• Primarily English-language corpus (no multilingual benchmarking)
• No code repositories or structured data (CSV, JSON) included

Conclusion

Apollo’s benchmark methodology prioritizes transparency, reproducibility, and fairness in performance reporting. By documenting hardware specs, test data, metrics definitions, and known limitations, we aim to provide an honest assessment of RAG system capabilities.

Key Takeaways:

• Benchmarks conducted on RTX 5080 with 100K document corpus
• Query latency: 8-15s (simple mode), 10-25s (adaptive mode)
• Cache optimization: 98% latency reduction on hits (less than 1ms)
• Inference speed: 80-100 tokens/sec (Q5_K_M quantization)
• Retrieval accuracy: 0.94 Recall@3 (advanced mode with reranking)

Next Steps

Explore detailed benchmark results and performance analysis:

• View Results - Comprehensive performance data and visualizations
• Configuration Guide - Learn how to tune Apollo for your workload
• Deployment Guide - Production deployment strategies

For questions about methodology or to request additional benchmarks, please open an issue on GitHub.