Neil Houlsby: Objective often matters more than ar...| Robocrunch AI

Objective often matters more than architecture. UL2 uses a mixture of denoisers objective, and allows one to switch between different downstream modes (finetuning, prompting) to get the best of both worlds. https://t.co/vkF3OfY3Ec
— Neil Houlsby (@neilhoulsby) May 13, 2022

In UL2, what we found to be useful is also to concatenate a prompt token that associates itself with a particular type of pretraining. During inference or finetuning, one can recall similar behavior using this prompt. We call this "Mode-switching".

Shared by Yi Tay at 5/12/2022

New UL2 model/paper from @GoogleAI! "Unifying Language Learning Paradigms" ✅ SOTA on 50-ish diverse NLP tasks ✅ Outperforms GPT-3 175B on 0-shot SGLUE ✅ 3 x perf vs T5 XXL (+LM) on 1-Shot XSUM ✅ Open code & 20B Flax checkpoints. Paper: https://t.co/U1Ks1uNUCz

Shared by Yi Tay at 5/12/2022

Goal-conditioned RL struggles as envs become more complex & combinatorial. We design invariant policy architectures that can learn faster, extrapolate to more objects/entities, and stitch together learned skills in novel ways. CompGen: https://t.co/KLQkFVCk4T Thread 🧵👇 1/n

Shared by Aravind Rajeswaran at 3/14/2022

Our model, ST-MoE-32B, sets SOTA across NLP tasks spanning reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum,CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande-XL, ANLI R3).

Shared by William Fedus at 2/21/2022

Thank you NeurIPS 2021 @diffprogramming workshop for giving this work the best poster award! We are excited about how AutomaticDifferentiation.jl is impacting the #julialang #sciml and #machinelearning communities. Packages are not tied to one framework. https://t.co/GsORsOc55R

Shared by Chris Rackauckas at 3/9/2022

Our prior works (https://t.co/MWNJjpmrdF, https://t.co/h2EGSq003Y, https://t.co/4V9NYKOFyd) show L2P works in single-objective optimization (SOO), and now extends to MOO. Theoretically we perform a study on why L2P works, if the underlying objectives have certain structures. 3/

Shared by Yuandong Tian at 3/30/2022

768