ExLLama.ContextParams
(LLama CPP Nif Wrapper v0.4.1)
Copy Markdown
Parameters for a llama.cpp inference context.
Numeric enum fields map 1:1 to their llama.cpp counterparts:
flash_attn_type--1auto,0disabled,1enabledpooling_type--1unspecified,0none,1mean,2cls,3last,4rankattention_type--1unspecified,0causal,1non-causal (embeddings)rope_scaling_type-0none,1linear,2yarn (llama.cppLLAMA_ROPE_SCALING_*)type_k/type_v- KV cache ggml type:1F16 (default),8Q8_0,2Q4_0,30BF16
Summary
Types
@type t() :: %ExLLama.ContextParams{ attention_type: integer(), embedding: boolean(), flash_attn_type: integer(), kv_unified: boolean(), n_batch: non_neg_integer(), n_ctx: non_neg_integer(), n_seq_max: non_neg_integer(), n_threads: non_neg_integer(), n_threads_batch: non_neg_integer(), n_ubatch: non_neg_integer(), offload_kqv: boolean(), op_offload: boolean(), pooling: boolean(), pooling_type: integer(), rope_freq_base: float(), rope_freq_scale: float(), rope_scaling_type: integer(), seed: non_neg_integer(), swa_full: boolean(), type_k: non_neg_integer(), type_v: non_neg_integer(), yarn_attn_factor: float(), yarn_beta_fast: float(), yarn_beta_slow: float(), yarn_ext_factor: float(), yarn_orig_ctx: non_neg_integer() }