gpt4all: run open-source LLMs anywhere

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

Reference implementation of Mistral AI 7B v0.1 model.

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

本项目旨在分享大模型相关技术原理以及实战经验。

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.

Sparsity-aware deep learning inference runtime for CPUs

Code examples and resources for DBRX, a large language model developed by Databricks

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

AICI: Prompts as (Wasm) Programs

🦖 Stateful Serverless Framework for building Geo-distributed Edge AI Infra

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.