NVIDIA/TensorRT-LLM

Repository: NVIDIA/TensorRT-LLM
Language: Python
Stars: 13470
Forks: 2319
Topics: blackwell, cuda, llm-serving, moe, pytorch

Description:
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.