Improve InMemoryIndex to Use a Cost-Based Cache for Predictable Memory Usage #89
Description
Problem:
The current InMemoryIndex in pkg/kvcache/kvblock/in_memory.go uses an item-count-based LRU cache. This leads to unpredictable memory usage because the size of cached values (the PodSet for each key) can vary significantly. A key associated with 100 pods(10 in our case) consumes far more memory than one with 2 pods, yet both are treated as a single item. This creates a risk of exceeding memory limits and causing system instability. This issue directly addresses the TODO comment in the code regarding memory-based configuration.
Proposed Solution:
Replace the hashicorp/golang-lru cache with a high-performance, cost-based cache like Ristretto.
This change will allow us to configure the cache with a fixed memory budget (e.g., 2 GB). Ristretto will manage this budget by evicting items based on their access frequency and their actual memory cost, which we will calculate based on the number of pods each key tracks. This also simplifies our architecture by replacing the two-level LRU with a single, more powerful cache.
Key Benefits:
Predictable Memory Footprint: Guarantees the cache operates within a stable, predefined memory limit, preventing OOM errors.
Improved Performance & Concurrency: Ristretto is designed for high-throughput systems and uses techniques like lock sharding to minimize contention, resolving the existing TODO about batching operations.
Smarter Eviction Policy: Evicts items based on their memory cost and usage (LFU), leading to a more efficient use of cache memory and potentially a higher hit ratio.