ExLLama.ContextParams (LLama CPP Nif Wrapper v0.4.1)

Copy Markdown

Parameters for a llama.cpp inference context.

Numeric enum fields map 1:1 to their llama.cpp counterparts:

  • flash_attn_type - -1 auto, 0 disabled, 1 enabled
  • pooling_type - -1 unspecified, 0 none, 1 mean, 2 cls, 3 last, 4 rank
  • attention_type - -1 unspecified, 0 causal, 1 non-causal (embeddings)
  • rope_scaling_type - 0 none, 1 linear, 2 yarn (llama.cpp LLAMA_ROPE_SCALING_*)
  • type_k / type_v - KV cache ggml type: 1 F16 (default), 8 Q8_0, 2 Q4_0, 30 BF16

Summary

Types

t()

@type t() :: %ExLLama.ContextParams{
  attention_type: integer(),
  embedding: boolean(),
  flash_attn_type: integer(),
  kv_unified: boolean(),
  n_batch: non_neg_integer(),
  n_ctx: non_neg_integer(),
  n_seq_max: non_neg_integer(),
  n_threads: non_neg_integer(),
  n_threads_batch: non_neg_integer(),
  n_ubatch: non_neg_integer(),
  offload_kqv: boolean(),
  op_offload: boolean(),
  pooling: boolean(),
  pooling_type: integer(),
  rope_freq_base: float(),
  rope_freq_scale: float(),
  rope_scaling_type: integer(),
  seed: non_neg_integer(),
  swa_full: boolean(),
  type_k: non_neg_integer(),
  type_v: non_neg_integer(),
  yarn_attn_factor: float(),
  yarn_beta_fast: float(),
  yarn_beta_slow: float(),
  yarn_ext_factor: float(),
  yarn_orig_ctx: non_neg_integer()
}