Aran Komatsuzaki: Reducing Activation Recomputation in...| Robocrunch AI

Reducing Activation Recomputation in Large Transformer Models

Achieves a Model FLOPS Utilization of 54.2% (baseline: 42.1%) when training a 530B parameter GPT-3 style model on 2240 NVIDIA A100 GPUs.https://t.co/zt6LPIthTj pic.twitter.com/aS6SszQAEc
— Aran Komatsuzaki (@arankomatsuzaki) May 12, 2022

Related

https://github.com/microsoft/mup

In collab with @OpenAI, we used μTransfer to tune GPT-3 6.7B with only 7% of its pretraining compute by tuning a 40M-param model instead. Even b ...

Shared by Edward Hu at 3/8/2022 1-3

5/ We µTransferred hyperparameters from a small 40 million parameter version of GPT-3 — small enough to fit on a single GPU — to the 6.7 billion ...

Shared by Greg Yang at 3/8/2022 0-4

https://huggingface.co/docs/transformers/master/performance#bf16

Have training stability issues but have access to Ampere GPUs and torch==1.10? You can now use bfloat16 with 🤗 Transformers@master https://t. ...

Shared by Stas Bekman at 12/3/2021 1-12

https://github.com/zphang/minimal-gpt-neox-20b

I wrote a minimal-ish implementation of GPT-NeoX-20B. It runs on a single GPU with 41-44GB of memory. You can use it as a reference or for easy ...

Shared by Jason Phang at 4/5/2022 7-33

https://blogs.nvidia.com/blog/2022/03/22/h100-transformer-engine

NVIDIA’s “H100 Transformer Engine Supercharges AI Training, Delivering Up to 6x Higher Performance Without Losing Accuracy” https://t.co/pYQvq7 ...

Shared by Kosta Derpanis at 3/22/2022 0-3

https://arxiv.org/abs/2111.11432

Impressive results from pretraining on image+text on tasks like COCO (>62%mAP), IN-1k (>83%) & K400 (>87%). But >512*10 A100 GPU ...

Shared by Yuki at 11/23/2021 1-13

768