Awni Hannun: Model FLOPs utilization (MFU) introd...| Robocrunch AI

Model FLOPs utilization (MFU) introduced in Pathways LM work (https://t.co/5K8SIwk9WD, sec 4.1).

MFU is roughly the ratio of the measured iterations-per-second to the theoretical max (for a given hardware). Mostly designed to penalize activation caching / redundant work. https://t.co/4TJ2EHbcQJ
— Awni Hannun (@awnihannun) May 12, 2022

In collab with @OpenAI, we used μTransfer to tune GPT-3 6.7B with only 7% of its pretraining compute by tuning a 40M-param model instead. Even better — as long as we underfit, wider is always better with the same HPs. Try it w/ your model! https://t.co/vU4gXzHF2Z

Shared by Edward Hu at 3/8/2022

Reducing Activation Recomputation in Large Transformer Models Achieves a Model FLOPS Utilization of 54.2% (baseline: 42.1%) when training a 530B parameter GPT-3 style model on 2240 NVIDIA A100 GPUs. https://t.co/zt6LPIthTj

Shared by Aran Komatsuzaki at 5/12/2022

Excited to share MLDemon: our new #AISTATS2022 paper to monitor #ML in deployment. MLDemon estimates model's real-time perf using mostly unlabeled data + a few adaptively collected labels. Efficient and optimal for many temporal data shifts! #MLops Paper https://t.co/BKPAsWEn8Q

Shared by James Zou at 3/1/2022

iSDF uses the main incremental neural field training methods of iMAP, but interprets the MLP output as a signed distance field rather than occupancy. Similar reconstruction quality, with auto hole-filling. Directly building an SDF could be useful for some robot planning cases.

Shared by Andrew Davison at 4/11/2022

The growing size of pre-trained language models (PLMs) brings up serious efficiency issues. To alleviate these issues, we present a suite of cost-effective techniques for the use of PLMs in pre-training, fine-tuning, and inference. #PLMs #AI #CPM Paper: https://t.co/Lo2KKK7A6z

Shared by TsinghuaNLP at 2/22/2022

Interested in using sparse expert models, but find they are unstable, hard to design or don’t fine-tune well? We address these key issues and train 269B param MoE model (w/ FLOPs of 32B dense model) that improves SOTA on NLP benchmarks liked SuperGLUE. https://t.co/taNrrx1V6G

Shared by Barret Zoph at 2/21/2022

768