Jason Wei: My mind is blown by new UL2 model. ...| Robocrunch AI

My mind is blown by new UL2 model.

- UL2 20B beats GPT-3 175B on zero-shot SuperGLUE
- Public 20B model
- Also finetunes really well

paper: https://t.co/7HNMUex99s https://t.co/Y9IbfF0Fiq
— jason wei (@_jasonwei) May 12, 2022

New UL2 model/paper from @GoogleAI! "Unifying Language Learning Paradigms" ✅ SOTA on 50-ish diverse NLP tasks ✅ Outperforms GPT-3 175B on 0-shot SGLUE ✅ 3 x perf vs T5 XXL (+LM) on 1-Shot XSUM ✅ Open code & 20B Flax checkpoints. Paper: https://t.co/U1Ks1uNUCz

Shared by Yi Tay at 5/12/2022

A great new paper from DeepMind: https://t.co/3f5f4v0a3p tl;dr: 70B language model outperforms their own 280B model with the same compute budget. It's MMLU performance now (early 2022) exceeds both the 2022 & *the 2023* forecasts made in 2021 => faster progress than expected!

Shared by Stanislav Fort (back in SF) at 3/30/2022

All 🤗 T5-based models have just gotten faster! https://t.co/k4Ravx7itq To benefit you need 1. transformers@master and 2. the latest https://t.co/k1GmbvsXOi build and it'll just work. Thanks to Eddie Yan for creating apex.normalization.FusedRMSNorm for the community! Enjoy!

Shared by Stas Bekman at 2/16/2022

Interested in using sparse expert models, but find they are unstable, hard to design or don’t fine-tune well? We address these key issues and train 269B param MoE model (w/ FLOPs of 32B dense model) that improves SOTA on NLP benchmarks liked SuperGLUE. https://t.co/taNrrx1V6G

Shared by Barret Zoph at 2/21/2022

In collab with @OpenAI, we used μTransfer to tune GPT-3 6.7B with only 7% of its pretraining compute by tuning a 40M-param model instead. Even better — as long as we underfit, wider is always better with the same HPs. Try it w/ your model! https://t.co/vU4gXzHF2Z

Shared by Edward Hu at 3/8/2022

First two Swin Transformer V2 CR weights added to timm for tiny and small models. I'm happy with the Tiny result at 81.8 top-1, Small managed 83.13 but can do better and currently trying... A collaboration with @ChristophR1996, tx! https://t.co/H5ffF1pVqa

Shared by Ross Wightman at 4/3/2022

768