ExLLama.ModelOptions (LLama CPP Nif Wrapper v0.4.1)

Copy Markdown

Options for loading a GGUF model via llama.cpp.

load_mode

llama.cpp v0.6.0 replaced the old use_mmap/use_mlock booleans with a single load_mode:

  • "auto" - auto-detect based on device capabilities (default)
  • "none" - plain read, no special loading mode
  • "mmap" - memory map the model
  • "mlock" - force the model to stay in RAM
  • "mmap+mlock" - mmap + keep in RAM ("mmap_mlock" also accepted)
  • "direct_io" / "dio" - direct I/O if available

split_mode / use_extra_bufts

Left nil by default so the llama.cpp defaults ("layer" split, extra buffer types enabled) are preserved; set them explicitly to override.

lazy_mode

  • 0 - off: read whole tensors up front (default)
  • 1 - auto: lazy only for marked tensors larger than 4 GiB (requires mmap)
  • 2 - on: read rows of marked tensors on demand (requires mmap)

Deprecated: use_mmap / use_mlock

Kept as deprecated struct fields for backward compatibility with the pre-0.4 API. When set, they are translated to load_mode ("mmap", "mlock", "mmap+mlock", or "none" when both are false). An explicit :load_mode always wins. They are never forwarded to the NIF directly.

Summary

Types

t()

@type t() :: %ExLLama.ModelOptions{
  check_tensors: boolean(),
  lazy_mode: 0 | 1 | 2,
  load_mode: String.t(),
  load_mtp: boolean(),
  main_gpu: non_neg_integer(),
  n_gpu_layers: non_neg_integer(),
  no_host: boolean(),
  split_mode: String.t(),
  use_extra_bufts: boolean(),
  use_mlock: boolean() | nil,
  use_mmap: boolean() | nil,
  vocab_only: boolean()
}

Functions

new()

new(x)