Skip to content

stdlib/bpeView source ↗

bpe

8 items · generated by scribe fromstdlib/bpe.fd· synced 2026-11-15

stdlib/bpe8 items

Contents

Bpe

pub struct Bpe {

Fields:

NameType
okbool
vocabstr
id2tokVec<str>
mergesVec<str>
max_leni64

Defined at line 8


bpe_new

Build a tokenizer. vocab_json: {"": id, …}. merges_text: the merges file content, one “left right” pair per line (header lines starting with ’#’ are skipped).

Example (native only): var bp = bpe_new(”{"":0,“a”:1,“b”:2,“ab”:3}”, “a b\n”); var ids = bpe_encode(bp, “ab”); println(int_to_str(ids.get(0))); // 3

pub fn bpe_new(vocab_json: str, merges_text: str) -> Bpe {

Parameters:

  • vocab_json: str
  • merges_text: str

Returns: Bpe

Defined at line 32


bpe_token

id -> token string, or "" for an id outside the vocab.

pub fn bpe_token(bp: Bpe, id: i64) -> str {

Parameters:

  • bp: Bpe
  • id: i64

Returns: str

Defined at line 97


bpe_id

token string -> id, or -1 (linear scan over vocab keys would be quadratic; uses the JSON object’s own lookup which is indexed).

pub fn bpe_id(bp: Bpe, tok: str) -> i64 {

Parameters:

  • bp: Bpe
  • tok: str

Returns: i64

Defined at line 106


bpe_encode

Encode: split text into chars as initial symbols, then repeatedly merge the lowest-rank pair — the standard BPE application order.

pub fn bpe_encode(bp: Bpe, text: str) -> Vec<i64> {

Parameters:

  • bp: Bpe
  • text: str

Returns: Vec<i64>

Defined at line 129


bpe_decode

Decode ids to the token-concatenated string.

pub fn bpe_decode(bp: Bpe, ids: Vec<i64>) -> str {

Parameters:

  • bp: Bpe
  • ids: Vec<i64>

Returns: str

Defined at line 177


Generated by scribe, the Fides documentation generator