ExLLama.Session (LLama CPP Nif Wrapper v0.4.1)

Copy Markdown

An inference session: one llama.cpp context bound to a loaded ExLLama.Model, with its own KV cache, sampler seed and context window.

Create with ExLLama.create_session/2 (options: ExLLama.SessionOptions, which carries the 0.4.0 context fields — flash attention, pooling, KV-cache quantization, etc.). The struct may be handed between processes (e.g. into a GenServer or Task): the seed is re-stashed into the receiving process on first use. A session is not concurrent-safe — serialize access or give each consumer its own deep_copy/1.

Core flow

{:ok, session} = ExLLama.create_session(model, options)
:ok = ExLLama.Session.advance_context(session, "The capital of France is")
{:ok, %{content: text, token_length: n}} =
  ExLLama.Session.completion(session, 16, "\n")

default_options/0 returns the NIF-side defaults ({:ok, opts}) — no explicit thread counts; llama.cpp picks per-host values.

Summary

Functions

advance_context(session, context)

advance_context_with_tokens(session, context)

completion(session, max_tokens, stop)

context(session)

context_size(session)

deep_copy(session)

default_options()

model(session)

params(session)

set_context(session, context)

set_context_to_tokens(session, tokens)

start_completing_with(session, options)

truncate_context(session, n_tokens)