这是一个开放的方法论、工具和逐步指导合集，旨在帮助大型语言模型和多模态模型及其推理的成功训练和微调。

这是一份适合LLM/VLM培训工程师和操作员的技术材料。这里的内容包含大量脚本和复制粘贴命令，帮助你快速满足需求。

这个仓库是我训练大型语言模型（LLM）（和VLM）的持续脑力倾倒;这些知识大多是我在2022年训练开源BLOOM-176B模型、2023年训练IDEFICS-80B多模态模型，以及2024年 Contextual.AI RAG模型时获得的。

我整理这些信息主要是为了自己，以便快速找到我过去研究过且有效的解决方案，但像往常一样，我很乐意与更广泛的机器学习社区分享这些笔记。

目录

第一部分。见解

1. 人工智能战场工程——你需要知道的成功知识。

2. 如何选择云服务提供商——这些问题将助你拥有成功的计算云体验。

Machine Learning Engineering Open Book

This is an open collection of methodologies, tools and step by step instructions to help with successful training and fine-tuning of large language models and multi-modal models and their inference.

This is a technical material suitable for LLM/VLM training engineers and operators. That is the content here contains lots of scripts and copy-n-paste commands to enable you to quickly address your needs.

This repo is an ongoing brain dump of my experiences training Large Language Models (LLM) (and VLMs); a lot of the know-how I acquired while training the open-source BLOOM-176B model in 2022 and IDEFICS-80B multi-modal model in 2023, and RAG models at Contextual.AI in 2024.

I’ve been compiling this information mostly for myself so that I could quickly find solutions I have already researched in the past and which have worked, but as usual I’m happy to share these notes with the wider ML community.

Table of Contents

Part 1. Insights

1. The AI Battlefield Engineering - what you need to know in order to succeed.

2. How to Choose a Cloud Provider - these questions will empower you to have a successful compute cloud experience.

3. When Is It Worth Upgrading GPUs? - a practical framework for deciding whether a GPU generation upgrade is worth its cost, worked through on a real H200 → B200 benchmark.

Part 2. Hardware

1. Compute - accelerators, CPUs, CPU memory.

2. Storage - local, distributed and shared file systems.

3. Network - intra- and inter-node networking.

Part 3. Orchestration

1. Orchestration Systems - managing containers and resources
2. SLURM - Simple Linux Utility for Resource Management

Part 4. Training

1. Training - model training-related guides

Part 5. Inference

1. Inference - model inference insights

Part 6. Development

1. Debugging and Troubleshooting - how to debug easy and difficult issues

2. And more debugging

3. Testing - numerous tips and tools to make test writing enjoyable

Part 7. Miscellaneous

1. Resources - LLM/VLM chronicles

Updates

I announce any significant updates on my twitter channel https://twitter.com/StasBekman.

Ebook versions of the book