bpe
8 items · generated by scribe fromstdlib/bpe.fd· synced 2026-11-15
Contents
- Bpe (struct)
- bpe_new (function)
- bpe_token (function)
- bpe_id (function)
- bpe_encode (function)
- bpe_decode (function)
Bpe
pub struct Bpe {Fields:
| Name | Type |
|---|---|
ok | bool |
vocab | str |
id2tok | Vec<str> |
merges | Vec<str> |
max_len | i64 |
Defined at line 8
bpe_new
Build a tokenizer. vocab_json: {"
Example (native only): var bp = bpe_new(”{"
pub fn bpe_new(vocab_json: str, merges_text: str) -> Bpe {Parameters:
vocab_json:strmerges_text:str
Returns: Bpe
Defined at line 32
bpe_token
id -> token string, or "" for an id outside the vocab.
pub fn bpe_token(bp: Bpe, id: i64) -> str {Parameters:
bp:Bpeid:i64
Returns: str
Defined at line 97
bpe_id
token string -> id, or -1 (linear scan over vocab keys would be quadratic; uses the JSON object’s own lookup which is indexed).
pub fn bpe_id(bp: Bpe, tok: str) -> i64 {Parameters:
bp:Bpetok:str
Returns: i64
Defined at line 106
bpe_encode
Encode: split text into chars as initial symbols, then repeatedly merge the lowest-rank pair — the standard BPE application order.
pub fn bpe_encode(bp: Bpe, text: str) -> Vec<i64> {Parameters:
bp:Bpetext:str
Returns: Vec<i64>
Defined at line 129
bpe_decode
Decode ids to the token-concatenated string.
pub fn bpe_decode(bp: Bpe, ids: Vec<i64>) -> str {Parameters:
bp:Bpeids:Vec<i64>
Returns: str
Defined at line 177
Generated by scribe, the Fides documentation generator