serialization¶

serialization ¶

Source-based serialization for remote execution.

Standard cloudpickle serializes code as bytecode, which is tied to the exact Python version — a payload pickled on 3.10 can fail to load on 3.11. This module serializes code by its source instead, so it reconstructs on any Python that can parse the syntax (client and server need not match).

Two mechanisms layer on top of cloudpickle:

code_reduce — reduces a block of source plus its scope to a filtered (source, globals, locals) tuple (only the names the source references, and the code as text). RequestModel.serialize ships the traced block as one of these tuples, and the function reducer runs a dynamic def's source/globals through it too; _load_function compiles the source back to a code object on the far side.
Persistent ids — objects carrying a _persistent_id in their __dict__ are written as that id and resolved on the other side from a persistent_objects map (keeps the model out of the payload).

DEFAULT_PROTOCOL `module-attribute` ¶

DEFAULT_PROTOCOL = 4

CustomCloudPickler ¶

Bases: CloudPickler

Pickler that serializes dynamic def functions by source, not bytecode.

persistent_id ¶

persistent_id(obj: Any) -> Optional[Any]

UnknownPersistentIdError ¶

Bases: UnpicklingError

A persistent id was encountered with no entry in persistent_objects.

CustomCloudUnpickler ¶

CustomCloudUnpickler(file: BufferedIOBase, persistent_objects: Optional[dict])

Bases: Unpickler

Unpickler that resolves persistent ids from a persistent_objects map.

persistent_objects `instance-attribute` ¶

persistent_objects = persistent_objects or {}

persistent_load ¶

persistent_load(pid: Any) -> Any

code_reduce ¶

code_reduce(source: str, globals: dict, locals: dict, function: bool = False) -> tuple

Reduce source + its scope to a picklable, filtered tuple.

Returns (source, used_globals, used_locals) — the code as text (so it recompiles on any Python version) plus only the globals/locals the source references, so the whole enclosing scope isn't shipped. Shared by both the traced-block payload and the dynamic-function reducer; _load_function (or the eventual block executor) compiles the source back into a code object.

Names are looked up by subscript rather than by iterating .items() so a Scope globals resolves through its fallback chain: a nested block (a tracer.invoke(...) body) runs with a Scope as its globals, whose iteration yields only the block's own names, leaving a module global it references (e.g. torch) out of the payload.

function says source is a single def / lambda rather than a traced block, which changes what counts as "referenced" — see _function_referenced_names (private: no reference page publishes it).

reduce_block ¶

reduce_block(node: Any, globals: dict, locals: dict) -> tuple

Reduce a captured with-block to a picklable (source, globals, locals).

Unparses the block body to source — padded with leading blank lines so the statements keep their original line numbers (a remote traceback then points at the right line) — and code_reduce\ s it against its scope. Shared by the request payload (the traced block) and edit-mediator serialization.

dump ¶

dump(obj: Any, file: Any, protocol: int = DEFAULT_PROTOCOL) -> None

Serialize obj to an open binary file (pickle.dump semantics).

dumps ¶

dumps(obj: Any, protocol: int = DEFAULT_PROTOCOL) -> bytes

Serialize obj to bytes (pickle.dumps semantics).

load ¶

load(file: Any, persistent_objects: Optional[dict] = None, unpickler: type = CustomCloudUnpickler) -> Any

Deserialize from an open binary file (pickle.load semantics).

unpickler selects the unpickler class (unpickler(file, persistent_objects)) so a caller can resolve persistent ids differently — e.g. a server that rebuilds the model out-of-process passes its own.

loads ¶

loads(data: bytes, persistent_objects: Optional[dict] = None, unpickler: type = CustomCloudUnpickler) -> Any

Deserialize from bytes (pickle.loads semantics).