The research team built a model that works in a Jupyter notebook. It predicts churn, detects anomalies, classifies documents, or scores leads — and it does it well. Now someone wants to use it in production. This is where most ML projects stall.
To productionize a machine learning model is an engineering problem, not a data science problem. The skills that make good models don't automatically produce reliable, maintainable inference systems. This post covers what the production gap looks like and how we close it.
The Production Gap
A Jupyter notebook model: - Runs on a data scientist's laptop - Uses manually prepared input data - Has no latency requirements - Has no error handling - Has no monitoring - Produces output that a human reviews
A production ML system: - Runs on scalable infrastructure - Receives live, uncleaned input data - Has SLA requirements (< 200ms? < 2s?) - Has retry logic, fallback behavior, and graceful degradation - Has monitoring for prediction quality, latency, and input drift - Makes decisions that affect users or business operations
The gap between these is 60–80% of the total engineering work.
What "Productionize Machine Learning Model" Actually Involves
Step 1: Model Packaging
The notebook needs to become a deployable artifact. This means:
Dependency isolation: Export exact package versions to requirements.txt or pyproject.toml. Pin versions. The model that trains on scikit-learn 1.2.0 should not be deployed on 1.3.x without regression testing.
Model serialization: Serialize the trained model to a stable format.
import mlflow
import mlflow.sklearn
# Log model with MLflow for reproducibility
with mlflow.start_run():
mlflow.log_params(model_params)
mlflow.log_metrics({"accuracy": accuracy, "f1": f1_score})
# Log model artifact with signature and input example
input_example = X_train[:5]
signature = mlflow.models.infer_signature(X_train, model.predict(X_train))
mlflow.sklearn.log_model(
model,
"model",
signature=signature,
input_example=input_example
)
Inference function: Encapsulate the full prediction pipeline — preprocessing, model call, postprocessing — in a single, testable function with typed inputs and outputs.
from pydantic import BaseModel
from typing import Optional
class PredictionInput(BaseModel):
feature_1: float
feature_2: str
feature_3: Optional[float] = None
class PredictionOutput(BaseModel):
prediction: str
confidence: float
model_version: str
def predict(input: PredictionInput) -> PredictionOutput:
# Preprocessing
features = preprocess(input)
# Inference
raw_prediction = model.predict([features])[0]
confidence = model.predict_proba([features]).max()
# Postprocessing
return PredictionOutput(
prediction=CLASS_LABELS[raw_prediction],
confidence=float(confidence),
model_version=MODEL_VERSION
)
Step 2: API Layer
Wrap the inference function in a REST API. FastAPI for Python ML services is our default:
from fastapi import FastAPI, HTTPException
import logging
app = FastAPI()
logger = logging.getLogger(__name__)
@app.post("/predict", response_model=PredictionOutput)
async def predict_endpoint(input: PredictionInput):
try:
result = predict(input)
# Structured logging for observability
logger.info({
"event": "prediction",
"confidence": result.confidence,
"prediction": result.prediction,
"model_version": result.model_version
})
return result
except Exception as e:
logger.error({"event": "prediction_error", "error": str(e)})
raise HTTPException(status_code=500, detail="Prediction failed")
@app.get("/health")
async def health():
return {"status": "healthy", "model_version": MODEL_VERSION}
Step 3: Containerization
Package the API in Docker for reproducible deployment:
FROM python:3.11-slim
WORKDIR /app
# Install dependencies first (Docker layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy model artifacts
COPY models/ ./models/
COPY src/ ./src/
USER nobody
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]
Step 4: Deployment Infrastructure
For most ML inference workloads, the deployment options:
ECS Fargate: For standard prediction APIs. Easy autoscaling, no server management, integrates cleanly with ALB. Our default for batch and online serving with moderate traffic.
Lambda: For low-traffic, latency-tolerant inference. Cold start with heavy ML models (large scikit-learn models, PyTorch) can be problematic — optimize with provisioned concurrency or Lambda extensions.
SageMaker Endpoints: For models with high traffic, real-time latency requirements, or when you want AWS-managed model hosting with built-in A/B testing and auto-scaling. Higher operational overhead but more managed.
Kubernetes (EKS): For organizations already running EKS with multi-model serving needs.
Step 5: Monitoring — The Part Most Teams Skip
Without monitoring, you don't know when the model is wrong. Production ML requires two types of monitoring:
Operational monitoring: API latency, error rates, throughput. Standard application monitoring via CloudWatch metrics and alarms.
Model quality monitoring: This is the ML-specific part. A model that was accurate when trained may degrade as the world changes. Monitoring this requires:
- Prediction distribution logging: Log every prediction. Track the distribution of output classes or regression values over time.
- Feature drift detection: If input features change distribution (e.g., user demographics shift), model performance can degrade without any code change. Compare current feature distributions to training distributions.
- Ground truth feedback loop: When outcomes are eventually known (did the churned user actually churn?), compare to predictions. Track actual model accuracy in production.
# Log predictions for drift monitoring
def log_prediction_for_monitoring(
input: PredictionInput,
output: PredictionOutput,
request_id: str
):
record = {
"request_id": request_id,
"timestamp": datetime.utcnow().isoformat(),
"input": input.dict(),
"output": output.dict()
}
# Write to S3 for offline analysis
s3.put_object(
Bucket=MONITORING_BUCKET,
Key=f"predictions/{datetime.utcnow().date()}/{request_id}.json",
Body=json.dumps(record)
)
Daily batch jobs analyze the logged predictions for drift using statistical tests (KS test for continuous features, chi-square for categorical). Alert when drift crosses threshold.
Step 6: Retraining Pipeline
The model you deploy is not the model you'll use in a year. Production ML systems need a path for:
- Scheduled retraining: Periodic retraining on fresh data
- Triggered retraining: When monitoring detects significant drift or accuracy degradation
- A/B testing: Deploy new model version alongside old, route percentage of traffic, compare performance before full cutover
We build retraining pipelines using Step Functions or AWS Managed Workflows for Apache Airflow (MWAA), depending on complexity.
For the infrastructure patterns backing ML deployment, see our data analytics capability and our guide on AI workflow automation for business.
Frequently Asked Questions
How long does it take to productionize a model that's working in a notebook?
A focused engagement for a single model with straightforward inputs: 4–8 weeks. This covers packaging, API development, containerization, deployment infrastructure, monitoring, and documentation. Complex models with custom preprocessing pipelines, high-availability requirements, or retraining pipelines take longer.
Should we use SageMaker or build our own serving infrastructure?
SageMaker is right when: you want managed infrastructure, have multiple models to serve, or need built-in A/B testing and auto-scaling without operational overhead. Custom infrastructure (ECS/EKS) is right when: your team already operates these platforms, you want more control over the serving code, or SageMaker's cost model doesn't fit your traffic pattern.
What do we do when model accuracy degrades in production?
Accuracy degradation is caught by the monitoring layer — specifically, ground truth feedback loop comparison or feature drift detection. When detected: first check whether the input data distribution has changed (feature drift). If so, retrain on recent data. If the underlying phenomenon has changed fundamentally, the problem formulation may need revisiting.
How do we handle models with very long inference times?
For models with inference times > 1–2 seconds, synchronous API design (request → wait → response) creates poor user experience. Better patterns: asynchronous inference (POST job → poll for result), pre-computation for high-probability inputs, or model optimization (quantization, ONNX export).
Can existing R&D models be deployed without a data scientist involved?
For straightforward models (scikit-learn, XGBoost, common PyTorch/TensorFlow architectures), the productionization work is primarily engineering and doesn't require ongoing data scientist involvement. Complex models with custom preprocessing or architectures may require data scientist collaboration during the packaging step to ensure the inference function matches training behavior.
Book a project diagnostic → rutagon.com/contact