A high-throughput and memory-efficient inference and serving engine for LLMs

PygmalionAI's large-scale inference engine

Foundation model benchmarking tool. Run any model on Amazon SageMaker and benchmark for performance across instance type and serving stack options.

This Guidance demonstrates how to deploy a machine learning inference architecture on Amazon Elastic Kubernetes Service (Amazon EKS). It addresses the basic implementation requirements as well as ways you can pack thousands of unique PyTorch deep learning (DL) models into a scalable architecture and evaluate performance at scale

CMP314 Optimizing NLP models with Amazon EC2 Inf1 instances in Amazon Sagemaker

This repository provides an easy hands-on way to get started with AWS Inferentia. A demonstration of this hands-on can be seen in the AWS Innovate 2023 - AIML Edition session.

Deploy Large Models on AWS Inferentia (Inf2) instances.

Sentence Transformers on EC2 Inf1