MLOps level 2 — CI/CD + Continous Training

Original Source Here

MLOps level 2 — CI/CD + Continous Training

MLOps level 2 is about utilising automated CI/CD to enable rapid and reliable updates to pipelines in production, as well as other environments.

Photo by Anton Maksimov juvnsky on Unsplash

Recommended Pre-Reading

This article is the final article of a 3 part series discussing the various levels of MLOps. To understand how the various levels differ please find links and resources below.

MLOps level 0 (Manual Pipelines) can be found here:

Image by author. MLOps Level 0

MLOps level 1 (Continuous Training) can be found here:

Image by author. MLOps Level 1

In order to get the most from this article it is important that you understand the terminology I will be using. To familiarise yourself with the terminology I have outlined the various steps of a machine learning service in great detail in this article:

Some of the concepts discussed overlaps with DevOps philosophy and others are specific to MLOps. To read more about how these two practices compare and contract please check out my article:

Purpose of MLOps level 2

MLOps level 2 is about enabling rapid and reliable updates to pipelines in production, as well as other environments, using automated CI/CD. Automating the pipeline by a automated CI/CD system enables rapid implementation of new ideas and experiments whilst simultaneously automatically building, testing and deploying the new pipeline artefacts and components to the appropriate target environment. Below is a diagram that visualises the concept of MLOps level 2:

Image by author. MLOps Level 2

Characteristics of MLOps level 2

Components that make up MLOps level 2 are very similar to MLOps level 1. The difference is in the addition of

  • Test and Build services (Continuous Integration).
  • Deployment services (Continuous Deployment).
  • Pipeline orchestration.

The pipeline can be broken down to 6 stages:

  1. Development and experimentation. This is where different algorithms and modelling techniques are tried and each experiment step is orchestrated. The final output of stage 1 is the source code of the pipeline steps which is pushed to a code repository.
  2. Continuous Integration of pipeline. Once the code is pushed to a code repository it triggers a test pipeline and builds the necessary components (packages, artefacts, etc) that needs to be deployed at a later stage.
  3. Continuous Delivery of pipeline. The artefacts produced in stage 2 is deployed to the target environment. The output of this stage is a deployed pipeline with the new implementation of the model from stage 1.
  4. Automated triggering. Execution of the pipeline in production is automatically triggered either based on a time schedule or in response to an event trigger. The final output of this stage is a trained model that is pushed to a model registry.
  5. Continuous Delivery of model. The final trained model is deployed as a service where applications can obtain model predictions. The final output of this stage is a deployed model prediction service.
  6. Model monitoring. The model in production is monitored live by collecting statistics of the model performance. When the model drifts or deteriorates away from an acceptable threshold this stage should trigger either an execution of the pipeline in production to retrain the model using new data or execute a new experiment pipeline.

What is new in MLOps level 2

The new concept in MLOps level 2 is automation of pipelines. This is achieved through Continuous Integration and Continuous Delivery.

In the continuous integration stage pipelines and components that make up a pipeline are built, tested and packaged when new code is pushed to a code repository. Testing in this stage is important and can include the following tests:

  • Unit test of the feature engineering logic.
  • Unit testing methods implemented in the model. For example, testing that correct data types are passed into functions.
  • Testing that during training the model converges.
  • Testing the output of the model is sensible and does not produce NaN values.
  • Testing that components in the pipeline outputs the expected artefacts.
  • Testing the integration between components of the pipeline.

In the continuous delivery stage new pipeline implementations are deployed to the target environment which in turn outputs a service that provides the newly trained model predictions. Things to consider in this stage includes:

  • Checking the compatibility of the new model on the target infrastructure. Is there sufficient resources, are required packages/libraries installed in the target environment etc.
  • Testing the performance of the prediction service.
  • Verifying that the new model meets the minimum predictive performance required.
  • Staged deployment depending on environment. For example, automated deployment of pipeline to a test environment, semi-automated deployment to a pre-production environment, manual deployment to a production environment after several successful runs in the pre-production environment.


Implementing a Machine Learning model into production involves several stages, components and pipelines. You don’t have to move all processes from one MLOps level to another. A team can implement gradually practices from MLOps level 2 into their current system to improve the level of automation.


Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: