stdlib/kv_cacheView source ↗
kv_cache
6 items · generated by scribe fromstdlib/kv_cache.fd· synced 2026-11-15
Contents
- kv_cache_new (function)
- kv_cache_append (function)
- kv_cache_get (function)
- kv_cache_len (function)
- kv_cache_clear (function)
- kv_cache_evict (function)
kv_cache_new
Create a new KV cache. num_layers: number of transformer layers max_seq_len: maximum sequence length (tokens) window_size: sliding window size (0 = no window) head_dim: dimension of each attention head
pub fn kv_cache_new(num_layers: i64, max_seq_len: i64, window_size: i64, head_dim: i64) -> i64 {Parameters:
num_layers:i64max_seq_len:i64window_size:i64head_dim:i64
Returns: i64
Defined at line 14
kv_cache_append
Append K and V tensors for a given layer.
pub fn kv_cache_append(cache: i64, layer: i64, k: Tensor, v: Tensor) -> void {Parameters:
cache:i64layer:i64k:Tensorv:Tensor
Returns: void
Defined at line 19
kv_cache_get
Get K and V for a layer as a concatenated tensor.
pub fn kv_cache_get(cache: i64, layer: i64) -> Tensor {Parameters:
cache:i64layer:i64
Returns: Tensor
Defined at line 24
kv_cache_len
Get current sequence length for a layer.
pub fn kv_cache_len(cache: i64, layer: i64) -> i64 {Parameters:
cache:i64layer:i64
Returns: i64
Defined at line 29
kv_cache_clear
Clear all layers.
pub fn kv_cache_clear(cache: i64) -> void {Parameters:
cache:i64
Returns: void
Defined at line 34
kv_cache_evict
Evict the oldest n tokens from all layers (sliding window).
pub fn kv_cache_evict(cache: i64, n: i64) -> void {Parameters:
cache:i64n:i64
Returns: void
Defined at line 39
Generated by scribe, the Fides documentation generator