ndif¶
Bases: dict
Status for remote execution on NDIF.
This class provides a structured view of the NDIF status, including information about all deployed models and their current states. It inherits from dict, allowing direct access to the underlying status data while providing rich formatting for display.
| ATTRIBUTE | DESCRIPTION |
|---|---|
status |
The overall service status (UP, REDEPLOYING, or DOWN).
TYPE:
|
Examples |
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
response
|
Dictionary mapping repo_id to model info dictionaries, each containing 'model_class', 'repo_id', 'revision', 'type', and 'state'.
TYPE:
|
Bases: Enum
Overall NDIF service status.
| ATTRIBUTE | DESCRIPTION |
|---|---|
UP |
Service is operational with at least one model running.
|
REDEPLOYING |
Service is transitioning; models are deploying or starting.
|
DOWN |
Service is unavailable or no models are running.
|
Bases: Enum
Status of an individual model deployment.
| ATTRIBUTE | DESCRIPTION |
|---|---|
RUNNING |
Model is fully deployed and accepting requests.
|
DEPLOYING |
Model is currently being deployed.
|
NOT_DEPLOYED |
Model is configured but not yet started.
|
DOWN |
Model deployment has failed or is unavailable.
|
Bases: Enum
Type of model deployment on NDIF.
| ATTRIBUTE | DESCRIPTION |
|---|---|
DEDICATED |
Model is a permanent deployment.
|
PILOT_ONLY |
Model available only for pilot users.
|
SCHEDULED |
Model runs on a schedule (e.g., specific hours).
|
Register a local module for serialization by value when executing remotely on NDIF.
When submitting code for remote execution on NDIF, any local modules that are not
installed on the server will cause a ModuleNotFoundError. This function registers
a module so that its class definitions and function source code are serialized and
sent along with the request, allowing them to be rebuilt on the server.
This is a wrapper around cloudpickle.register_pickle_by_value.
| PARAMETER | DESCRIPTION |
|---|---|
module
|
The module to register for serialization by value. Can be the actual module object or a string with the module's name.
TYPE:
|
Note
- Call this function after importing the module but before using any of its contents in a remote context.
- The module's source code and definitions will be included in the serialized payload, so keep registered modules reasonably sized.
Examples:
import mymodule from nnsight import LanguageModel from nnsight.ndif import register
Register the local module so it can be used remotely¶
register(mymodule)
Or register by module name¶
register("mymodule")
Now you can use functions/classes from mymodule in remote execution¶
from mymodule.myfile import myfunction
model = LanguageModel("meta-llama/Llama-3.1-70B") with model.generate("Hello", remote=True): ... result = myfunction(model) ... result.save()
status(raw: bool = False) -> Union[dict, NdifStatus]
Query the current status of the NDIF service and all deployed models.
This is the primary function for checking NDIF availability and model status. When printed, the returned NdifStatus object displays a formatted table of all available models with their deployment type and current state.
| PARAMETER | DESCRIPTION |
|---|---|
raw
|
If True, return the raw API response dict. If False (default), return a formatted NdifStatus object.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Union[dict, NdifStatus]
|
If raw=True: The raw JSON response from the NDIF API. |
Union[dict, NdifStatus]
|
If raw=False: An NdifStatus object with formatted model information, or an empty dict if the request fails. |
Examples
|
TYPE:
|
Union[dict, NdifStatus]
|
|
Union[dict, NdifStatus]
|
|
Union[dict, NdifStatus]
|
|
Union[dict, NdifStatus]
|
NDIF Service: Up 🟢 |
Union[dict, NdifStatus]
|
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓ |
Union[dict, NdifStatus]
|
┃ Model Class ┃ Repo ID ┃ Revision ┃ Type ┃ Status ┃ |
Union[dict, NdifStatus]
|
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩ |
Union[dict, NdifStatus]
|
│ LanguageModel │ meta-llama/Llama-3.1-70B │ main │ Dedicated │ RUNNING │ |
Union[dict, NdifStatus]
|
└───────────────┴────────────────────────────┴──────────┴───────────┴─────────┘ |
ndif_status(raw: bool = False) -> Union[dict, NdifStatus]
Deprecated: Use status() instead.
Checks if a specific model is currently running on NDIF.
| PARAMETER | DESCRIPTION |
|---|---|
repo_id
|
The HuggingFace repository ID (e.g., "meta-llama/Llama-3.1-70B").
TYPE:
|
revision
|
The model revision/branch to check. Defaults to "main".
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the model is running and available, False otherwise |
bool
|
(including if the API request fails). |
Examples
|
TYPE:
|
bool
|
|
bool
|
|
bool
|
... print("Model is available!") |
Fetch and cache the NDIF environment information from the remote server.
| PARAMETER | DESCRIPTION |
|---|---|
force_refresh
|
If True, always refetch even if cached.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
dict
|
The environment info from NDIF. |
| RAISES | DESCRIPTION |
|---|---|
Exception
|
If the request fails. |
Build the environment comparison table for local and remote.
Compare the local Python environment with the NDIF remote environment and print the results.
| RETURNS | DESCRIPTION |
|---|---|
None
|
None |