Skip to content

stdlib/kv_cacheView source ↗

kv_cache

6 items · generated by scribe fromstdlib/kv_cache.fd· synced 2026-11-15

stdlib/kv_cache6 items

Contents

kv_cache_new

Create a new KV cache. num_layers: number of transformer layers max_seq_len: maximum sequence length (tokens) window_size: sliding window size (0 = no window) head_dim: dimension of each attention head

pub fn kv_cache_new(num_layers: i64, max_seq_len: i64, window_size: i64, head_dim: i64) -> i64 {

Parameters:

  • num_layers: i64
  • max_seq_len: i64
  • window_size: i64
  • head_dim: i64

Returns: i64

Defined at line 14


kv_cache_append

Append K and V tensors for a given layer.

pub fn kv_cache_append(cache: i64, layer: i64, k: Tensor, v: Tensor) -> void {

Parameters:

  • cache: i64
  • layer: i64
  • k: Tensor
  • v: Tensor

Returns: void

Defined at line 19


kv_cache_get

Get K and V for a layer as a concatenated tensor.

pub fn kv_cache_get(cache: i64, layer: i64) -> Tensor {

Parameters:

  • cache: i64
  • layer: i64

Returns: Tensor

Defined at line 24


kv_cache_len

Get current sequence length for a layer.

pub fn kv_cache_len(cache: i64, layer: i64) -> i64 {

Parameters:

  • cache: i64
  • layer: i64

Returns: i64

Defined at line 29


kv_cache_clear

Clear all layers.

pub fn kv_cache_clear(cache: i64) -> void {

Parameters:

  • cache: i64

Returns: void

Defined at line 34


kv_cache_evict

Evict the oldest n tokens from all layers (sliding window).

pub fn kv_cache_evict(cache: i64, n: i64) -> void {

Parameters:

  • cache: i64
  • n: i64

Returns: void

Defined at line 39


Generated by scribe, the Fides documentation generator