ExLLama.ModelOptions
(LLama CPP Nif Wrapper v0.4.1)
Copy Markdown
Options for loading a GGUF model via llama.cpp.
load_mode
llama.cpp v0.6.0 replaced the old use_mmap/use_mlock booleans with a single
load_mode:
"auto"- auto-detect based on device capabilities (default)"none"- plain read, no special loading mode"mmap"- memory map the model"mlock"- force the model to stay in RAM"mmap+mlock"- mmap + keep in RAM ("mmap_mlock"also accepted)"direct_io"/"dio"- direct I/O if available
split_mode / use_extra_bufts
Left nil by default so the llama.cpp defaults ("layer" split, extra
buffer types enabled) are preserved; set them explicitly to override.
lazy_mode
0- off: read whole tensors up front (default)1- auto: lazy only for marked tensors larger than 4 GiB (requires mmap)2- on: read rows of marked tensors on demand (requires mmap)
Deprecated: use_mmap / use_mlock
Kept as deprecated struct fields for backward compatibility with the pre-0.4
API. When set, they are translated to load_mode ("mmap", "mlock",
"mmap+mlock", or "none" when both are false). An explicit :load_mode
always wins. They are never forwarded to the NIF directly.
Summary
Types
@type t() :: %ExLLama.ModelOptions{ check_tensors: boolean(), lazy_mode: 0 | 1 | 2, load_mode: String.t(), load_mtp: boolean(), main_gpu: non_neg_integer(), n_gpu_layers: non_neg_integer(), no_host: boolean(), split_mode: String.t(), use_extra_bufts: boolean(), use_mlock: boolean() | nil, use_mmap: boolean() | nil, vocab_only: boolean() }