install • docs • examples • we're hiring • chat with us

• Multi framework: deploy TensorFlow, PyTorch, scikit-learn, and other models.
• Autoscaling: automatically scale APIs to handle production workloads.
• ML instances: run inference on G4, P2, M5, C5 and other AWS instance types.
• Spot instances: save money with spot instances.
• Rolling updates: update deployed APIs with no downtime.
• Log streaming: stream logs from deployed models to your CLI.
• Prediction monitoring: monitor API performance and prediction results.

$ bash -c "$(curl -sS https://raw.githubusercontent.com/cortexlabs/cortex/0.16/get-cli.sh)"

# predictor.py class PythonPredictor: def __init__(self, config): self.model = download_model() def predict(self, payload): return self.model.predict(payload["text"])

# cortex.yaml - name: sentiment-classifier predictor: type: python path: predictor.py compute: gpu: 1 mem: 4G

$ cortex deploy creating sentiment-classifier

$ curl http://localhost:8888 \ -X POST -H "Content-Type: application/json" \ -d '{"text": "serving models locally is cool!"}' positive

Cortex clusters are designed to be self-hosted on any AWS account:

$ cortex cluster up aws region: us-east-1 aws instance type: g4dn.xlarge spot instances: yes min instances: 0 max instances: 5 your cluster will cost $0.19 - $2.85 per hour based on cluster size and spot instance pricing/availability ￮ spinning up your cluster ... your cluster is ready!

$ cortex deploy --env aws creating sentiment-classifier

$ curl http://***.amazonaws.com/sentiment-classifier \ -X POST -H "Content-Type: application/json" \ -d '{"text": "serving models at scale is really cool!"}' positive

$ cortex get sentiment-classifier status up-to-date requested last update avg request 2XX live 1 1 8s 24ms 12 class count positive 8 negative 4

The CLI sends configuration and code to the cluster every time you run cortex deploy. Each model is loaded into a Docker container, along with any Python packages and request handling code. The model is exposed as a web service using a Network Load Balancer (NLB) and FastAPI / TensorFlow Serving / ONNX Runtime (depending on the model type). The containers are orchestrated on Elastic Kubernetes Service (EKS) while logs and metrics are streamed to CloudWatch.

Cortex manages its own Kubernetes cluster so that end-to-end functionality like request-based autoscaling, GPU support, and spot instance management can work out of the box without any additional DevOps work.

Cortex is an open source alternative to serving models with SageMaker or building your own model deployment platform on top of AWS services like Elastic Kubernetes Service (EKS), Lambda, or Fargate and open source projects like Docker, Kubernetes, TensorFlow Serving, and TorchServe.

• Image classification: deploy an Inception model to classify images.
• Search completion: deploy Facebook's RoBERTa model to complete search terms.
• Text generation: deploy Hugging Face's DistilGPT2 model to generate text.