Skip to content

Model lifecycle explained

A typical model lifecycle with default configuration:

  1. PUT /p/:project/models stores the model definition in the local SQLite database and in-memory cache.
  2. No immediate action is taken. The model sits idle until explicitly woken up via POST /p/:project/models or a direct inference request to this model.
  3. When woken, the daemon's JobSyncReconciler picks up the non-idle model, queries HEAppE for existing jobs, and creates/submits a new HEAppE job if the desired scale exceeds the count of currently fresh jobs.
  4. The HEAppE job transitions through queued → running states. The MetricsReconciler polls metrics from the running jobs via HEAppE data transfer endpoints.
  5. After some amount of time of inactivity on the inference API for the model, the IdleReconciler sets the model idle. The JobSyncReconciler then stops creating replacement jobs, and existing jobs drain naturally.

Optionally a model can be set to auto-scale: The scale normally starts at 1, but can increase when auto-scaling is enabled, making it possible to have multiple jobs for the model when all the running jobs get saturated with inference requests.