A. Scale-to-Zero with SageMaker Asynchronous Inference
Standard real-time endpoints cannot scale to zero instances; they require at least one instance running 24/7 to listen for requests.
The Solution: Use SageMaker Asynchronous Inference. It queues incoming requests in Amazon SQS.
Idle Handling: You can configure an autoscaling policy that tracks the queue depth. If the queue is empty for a defined period (e.g., 15 minutes), it scales the underlying EC2 instances down to zero.
The Trade-off: When a new user sends a request after an idle period, they will experience a cold start (delay of a couple of minutes) while SageMaker provisions a new container and loads the model weights back into GPU memory
No comments:
Post a Comment