Skip to contents

Convert Token IDs Back to Text

Source: R/api.R
detokenize.Rd

Converts a sequence of integer token IDs back into human-readable text. This is the inverse operation of tokenization and is typically used to convert model output tokens into text that can be displayed to users.

Usage

detokenize(model, tokens)

Arguments

model

A model object created with model_load. Must be the same model that was used for tokenization to ensure proper decoding

tokens

Integer vector of token IDs to convert back to text. These are typically generated by tokenize or generate

Value

Character string containing the decoded text corresponding to the input tokens

See also

tokenize, generate, model_load

Examples

if (FALSE) { # \dontrun{
# Load model
model <- model_load("path/to/model.gguf")

# Tokenize then detokenize (round-trip)
original_text <- "Hello, how are you today?"
tokens <- tokenize(model, original_text)
recovered_text <- detokenize(model, tokens)
print(recovered_text)  # Should match original_text

# Generate and display text
ctx <- context_create(model)
generated_text <- generate(ctx, "The weather is", max_tokens = 10)

# Inspect individual tokens
single_token <- c(123)  # Some token ID
token_text <- detokenize(model, single_token)
print(paste("Token", single_token, "represents:", token_text))
} # }