quantization¶
Loading a checkpoint in a format that isn't a torch dtype.
A model too big for the GPU you have can be held in fewer bits per weight than
any torch.dtype offers — 4-bit through bitsandbytes, 8-bit through either
bitsandbytes or transformers' own FP8. Doing that normally means building a
quantizer config object and knowing which of transformers' several quantizer
backends the format belongs to, which is a lot of ceremony for a choice that is
really just how wide is a weight.
So it goes in the dtype slot, next to the widths torch does have::
TransformersModel("meta-llama/Llama-3.2-3B", dtype="nf4", dispatch=True)
dtype here is what the weights are held as. Everything the format leaves
alone — norms, embeddings, the LM head — and everything the model computes in
stays DEFAULT_COMPUTE_DTYPE,
so activations come out of a quantized model in the same dtype they would come
out of a bfloat16 one and a trace reads the same either way.
The table below is the single place these names are defined, and it is read from
two directions that must not disagree: loading a checkpoint (here) and
sizing one before it is loaded
(bytes_per_element,
which a server uses to decide how many GPUs a deployment gets). A name one side
accepts and the other rejects is a deployment that is placed and then fails to
load, or loads and was never placed.
What is not affected is the module tree: a quantized linear is a different
class holding a differently-shaped weight, but it sits at the same path with the
same children, so module paths — and therefore interventions, envoys, and remote
requests naming them — are unchanged. Reading a raw .weight is the exception;
see Quantization.
The bitsandbytes formats swap nn.Linear and nothing else, which decides what
a checkpoint actually saves. transformers 5 holds a mixture-of-experts model's
experts as stacked 3-D parameters on one module rather than as linears, so those
weights, most of an MoE, stay at the compute dtype and the model shrinks by a few
percent.
A remote model key is {repo_id, revision} and says nothing about how the
weights are held, so dtype on a remote model shapes only the client's own meta
build: the deployment decides what a replica holds. Routing to a replica by dtype
would mean putting the dtype in the key, and treating two dtypes of one checkpoint
as two deployments, which is a routing question rather than a loading one.
module-attribute
¶
QUANTIZATIONS: dict[str, Quantization] = {'nf4': _NF4, 'int4': _NF4, '4bit': _NF4, 'fp4': Quantization(0.5, _bitsandbytes_4bit('fp4')), 'int8': _INT8, '8bit': _INT8, 'fp8': Quantization(1.0, _fp8)}
dataclass
¶
Quantization(bytes_per_element: float, build: Callable[[Any], Any], compute_dtype: str = DEFAULT_COMPUTE_DTYPE)
One way of holding weights that torch has no dtype for.
| PARAMETER | DESCRIPTION |
|---|---|
bytes_per_element
|
Nominal width of one stored weight. Nominal:
the formats here leave the LM head, embeddings and norms in 16 bits
and store a scale per block, none of which this counts, so the real
footprint is larger. On Llama-3.2-1B, whose embeddings are 21% of
its parameters,
TYPE:
|
build
|
Takes the compute dtype and returns the transformers quantizer config. Imports its backend inside, so a name nobody asks for costs nothing and a missing backend fails when it is actually wanted.
TYPE:
|
compute_dtype
|
What this format computes in, and holds everything it
does not quantize in. Per-format rather than one constant because
the backends do not agree: see
TYPE:
|
quantization(dtype: Any) -> Optional[Quantization]
The format dtype names, or None if it names a torch dtype.
None is the ordinary answer and means "nothing to do" — every caller
here is deciding whether a load needs rewriting at all.
Turn a quantization name in kwargs' dtype into a load transformers takes.
Returns a new dict; kwargs is left alone. Kwargs whose dtype is an
ordinary torch dtype come back unchanged — not normalized, not
reordered — so this can sit on every load path without having an opinion
about the ones it has nothing to do with.
With quantize false the name is replaced by the compute dtype and no
quantizer config is built. That is the meta-model path: building the
architecture without weights has nothing to quantize, and the quantizers
reject a meta device outright. The resulting tree is the same either way,
which is what lets a client build a meta model of a checkpoint a server
holds quantized and have every module path line up.
| PARAMETER | DESCRIPTION |
|---|---|
kwargs
|
Load kwargs, as passed to
TYPE:
|
quantize
|
Whether to actually build the quantizer config.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
if a |