robocrunch
Sayak Paul
@RisingSayak
Calling `https://t.co/zhiBaOSc0b()` at @carted, ex-@PyImageSearch, ex-@TCSResearch | Netflix Nerd
Tweets by Sayak Paul
I trained a baseline ViT S/16 model with `big_vision` and I was able to reproduce everything. 90 epochs of training ✅ 76.23% top-1 on ImageNet-1k val ✅ Took about 7.367 hours ✅ Training logs & checkpoints: https://t.co/8wWRYFsRud @giffmana @__kolesnikov__ @XiaohuaZhai
Shared by
Sayak Paul
at
5/11/2022
The way the community has responded to this has been beyond incredible. @ariG23498 and I are still processing the depth of the support we received in all honesty. Here's an update. 1/
Shared by
Sayak Paul
at
5/10/2022
Our tutorial submission on “Transformers for Visual Recognition” (@ariG23498 and I) have been accepted to ACM KDD 2022 (@kdd_news)! Here’s our proposal: https://t.co/hUY9rBBOuN But alas … 1/
Shared by
Sayak Paul
at
5/8/2022
data2vec vision models in @TensorFlow are now available in @huggingface transformers 🤗 https://t.co/7Kzj3uBrZK Thanks to @GuggerSylvain & @carrigmat! 1/
Shared by
Sayak Paul
at
5/7/2022
Another day and another dead simple yet useful work from @giffmana, @XiaohuaZhai, and @__kolesnikov__: https://t.co/DEh7nv9gEx "... Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, ..."
Shared by
Sayak Paul
at
5/5/2022
Additionally, this CaiT uses the "Talking Head" attention. So, this project could serve as a reference for the TF implementation of that. Thanks to @wightmanr for providing PT references of the models which definitely made life easier. 4/
Shared by
Sayak Paul
at
5/4/2022
The implemented models have been evaluated on the ImageNet-1k validation set to ensure correctness and to improve developer trust. Thanks to the @GoogleDevExpert program for providing #GCP credits that supported my experiments. Onwards now! 5/
Shared by
Sayak Paul
at
5/4/2022
I find CaiT interesting because it tackles two qs: * Why does the performance of deeper ViTs saturate on relatively smaller datasets? * Can we separate class attention from the self-attention stage from the patches thereby inducing a form of cross-attention? 2/
Shared by
Sayak Paul
at
5/4/2022
New work! Implementation of CaiT family of vision transformers in @TensorFlow. Code, ported models, interactive demos, notebooks for fine-tuning & inference are here: https://t.co/PSpz2EZUOl 1/
Shared by
Sayak Paul
at
5/4/2022
@wightmanr have you tried the way DINO does representation pooling on supervised vision transformers? If so, did you notice any improvements? https://t.co/YNLsGjIbeV
Shared by
Sayak Paul
at
5/2/2022
Dynamic padding is a very common technique in the NLP world yet it's not so common in @TensorFlow. Today, we're sharing code, notebooks, and articles from @carted showing how to efficiently handle variable-length text sequences with dynamic padding. https://t.co/mSeBvttoSN 1/
Shared by
Sayak Paul
at
4/26/2022
We’ve used the following methods for our analysis: * Attention rollout * Classic heatmap of the attention weights * Mean attention distance * Viz of the positional embeddings & linear projections We hope our work turns out to be a useful resource for those studying ViTs. 3/
Shared by
Sayak Paul
at
4/18/2022
We’ve also built a @huggingface organization around our experiments. The organization holds the Keras pre-trained models & spaces where you can try the visualization on your own images. https://t.co/bWzVusn9DX Contributions are welcomed :) 4/
Shared by
Sayak Paul
at
4/18/2022
What do the Vision Transformers learn? How do they encode anything useful for image recognition? In our latest work, we reimplement a number of works done in this area & investigate various ViT model families (DeiT, DINO, original, etc.). Done w/ @ariG23498 1/
Shared by
Sayak Paul
at
4/18/2022
New project 📽 Implemented DeiT models in TensorFlow & ported the pre-trained params into them. Code, pre-trained TF models, attention rollout visualization, tutorial, etc. all are available here: https://t.co/yfV0GSs4YX 1/
Shared by
Sayak Paul
at
4/12/2022
DRLoC: https://t.co/zHUYKK00gc Regularizes the training by introducing another loss term based on an SSL objective: predict the relative distances of token embeddings. Achieves promising results with ViTs when training data is scarce. 5/
Shared by
Sayak Paul
at
4/7/2022
DeiT: https://t.co/fEie46vRoh One of the first works showing the benefits of training ViTs with more regularization, augmentation, and longer training + Presents a simple distillation strategy which I think is pretty cool. 4/
Shared by
Sayak Paul
at
4/7/2022
Have been reading up stuff on training ViTs the right way lately. Here are five works that aren't leaving me anytime soon! In no particular order ⬇️ 1/
Shared by
Sayak Paul
at
4/7/2022
TensorFlow variant of ViT-MAE (masked autoencoders) is now available on @huggingface `transformers` & I am so incredibly happy to have contributed it w/ @ariG23498. This was another interesting learning experience. Read on to know why. https://t.co/enmdvKhXSc 1/
Shared by
Sayak Paul
at
3/30/2022
For me, it was Keras among other things that inspired me to take up deep learning as a potential career and leave a job I didn't like. It helped me boost my confidence too. May it continue to grow, prosper, and evolve. @fchollet THANK YOU!
Shared by
Sayak Paul
at
3/27/2022
Nice work from @HugoTouvron et al: * Process the residual connections parallelly * Only fine-tune attn layer params for adapting ViTs to higher res * Addition of MLP-based patch preprocessing layers for SSL ViTs (BERT like) https://t.co/Wv09XR9Bmb
Shared by
Sayak Paul
at
3/26/2022
I've seen many Kaggle Kernels demoing the usefulness and cool features of `timm` but this guide stands out pretty tall. Great start to Tuesday morning: https://t.co/o522BmCDb2
Shared by
Sayak Paul
at
3/15/2022
For me, it was a domain shift. @nilabhraroy comes with a PyTorch background but picked up @TensorFlow quite fast. I've learned many new things in TensorFlow from him. 2/
Shared by
Sayak Paul
at
3/5/2022
Humbled to receive the Google OSS Expert Prize with @soumikRakshit96 for our work on explaining & demoing GauGAN in #keras. * Kernel: https://t.co/ywCKuQqg5w * Blog post on https://t.co/ayUd2g1WD5: https://t.co/YHHRavAvKz * Announcement from @kaggle: https://t.co/KSfYXrwIud 1/
Shared by
Sayak Paul
at
3/2/2022
How to load-test a model that is exposed as a service? This guide provides a really helpful framework for this including code, motivation, insightful commentaries. https://t.co/oGvfrJiHfO
Shared by
Sayak Paul
at
3/1/2022
For anyone that wanted to play with the new ConvNeXt models. Thanks to @awsaf49 who contributed the fine-tuning notebook. There is still an open call for contributions in the repo in case anyone's interested: https://t.co/lfzEWDWxTA
Shared by
Sayak Paul
at
2/21/2022
An interesting take on understanding generalization: "Deep Bootstrap". Use a fresh set of samples every epoch to train on a large (and "virtually infinite dataset") first. Models that converge faster here are likely to generalize better. https://t.co/CAas2HptCb
Shared by
Sayak Paul
at
2/19/2022
I think I speak on behalf of everyone that has contributed to #Keras in any way when I say your feedback has always had a positive impact not only on our work but also on our lives. Thank you, Francois! You definitely make this community vibrant and inclusive by setting examples.
Shared by
Sayak Paul
at
12/2/2021
New work with @algo_diver! We show how to develop a minimal Ml system that can continuously be adapted w.r.t data drift. We use #Keras, #TFX, and various #GCP services. Details 👇 Huge thanks to @robert_crowe for helping us on this one.
Shared by
Sayak Paul
at
12/2/2021
Our (@SiddhaGanju & mine) work on flood region segmentation will be there at PyTorch Developer Day today. Details are here: https://t.co/xNzBgRaGm8 @PyTorch made a beautiful poster for our work (see attached) 1/
Shared by
Sayak Paul
at
12/2/2021
New work with @ariG23498 on implementing Masked Autoencoders for self-supervised pretraining of images. Thanks to @endernewton for the helpful discussions.
Shared by
Sayak Paul
at
11/25/2021
I'm currently in absolute awe after reading @theaisummer's take on Einsums by implementing a vanilla Transformer block. The dude's got magical abilities! https://t.co/8YsxMKSjEY
Shared by
Sayak Paul
at
8/31/2021
New example showing how to implement a VQ-VAE in #Keras with the PixelCNN part included. The example goes through several key ideas: optimizing discrete latent spaces, codebook sampling, and above all visualizations! https://t.co/1FTWusXqf7 1/
Shared by
Sayak Paul
at
7/26/2021
`robustness_metrics` is a mini framework by @GoogleAI that provides three sets of evaluation metrics for image classification models -- OoD, Natural Perturbation, and Uncertainty (calibration). Very promising. https://t.co/KXsUiMbaKl
Shared by
Sayak Paul
at
7/25/2021
#TFHub models are opaque. If you load a (compatible) model from Hub using `tf.keras.models.load_model` instead of `KerasLayer`, it's not opaque anymore. Here's a notebook investigating this on the BiT (Big Transfer) family of ResNets. https://t.co/ZROcplxpGT 1/
Shared by
Sayak Paul
at
7/24/2021
New example on Conditional GANs. I think this recipe is important to know if you are into generative deep learning. Disclaimer: This is NOT SoTA stuff. It's more about the workflow which you can extend to high-fidelity datasets. https://t.co/jGB3YbH6Lo https://t.co/QENxzZRpCS
Shared by
Sayak Paul
at
7/17/2021
Imagine being able to configure your model search space and eval criteria to guide that search space w/ flexibility and convenience. Enter Retiarii by @MSFTResearch: https://t.co/FVNPTpG2Um Support @TensorFlow and @PyTorch. Waiting for JAX now 😅
Shared by
Sayak Paul
at
7/12/2021
I am very happy about Aditya and @7vasudevgupta as they are a part of @gsoc. It's been a great dose of learning and sharing working w/ them. Look forward to announcements on cool models coming your way! Thanks to the @TensorFlow team for allowing me to mentor :)
Shared by
Sayak Paul
at
7/8/2021
If you're serving a @TensorFlow model from a GPU-based environment and you're yet to use TensorRT, now would be a good time: https://t.co/CF8YFA3nWr Speed benefits: https://t.co/Llob8OLrEh
Shared by
Sayak Paul
at
7/7/2021
How does a commoner train a Transformers-based model on a small or medium dataset on commodity GPUs? And still, achieve competitive results? Here, Compact Conv. Transformers (by @AliHassaniJr et al.): https://t.co/TaGEqEjaHU P.S.: The snap is not clickbait. https://t.co/MgjZO7hbF9
Shared by
Sayak Paul
at
7/5/2021
Benefits of including unlabeled data in semi-supervised learning is well-studied. Amongst all the recent advancements in this area, PAWS (@facebookai) is my favorite. It shines exceptionally well compared to other methods that too requiring fewer steps. https://t.co/iWiL22I93n
Shared by
Sayak Paul
at
7/3/2021
It was a great learning experience implementing it in @TensorFlow earlier this year: https://t.co/TCiUKN9Z4q The implementation includes all the bits that are needed to make PAWS work for CIFAR-10,
Shared by
Sayak Paul
at
7/3/2021
In case you were wondering what's possible w/ `model.compile()` & `https://t.co/Lck2QTNscz()` know that pretty much anything 😉 Switch Transformers, CLIP-like models, CycleGAN -- whatever you'd like (all of those are already available as examples FYI). Happy open-sourcing! 3/
Shared by
Sayak Paul
at
7/2/2021
Implementation & short piece on *AdaMatch* in #Keras. AdaMatch beautifully unifies semi-supervised (SS) learning & domain adaptation (DA). Within just 10 epochs I got a whopping 12% impr. on SHVN. https://t.co/U5i0hDlzh6 Cc @D_Berthelot_ML @BeccaRoelofs & others. 1/
Shared by
Sayak Paul
at
6/23/2021
AugLy: A new data augmentation library to help build more robust AI models by @facebookai. Supports different data modalities too. How cool! https://t.co/ICNREolbU6 /cc: @DanHendrycks
Shared by
Sayak Paul
at
6/18/2021
In this example, we build a Transformer-based video classifier. We first extract feature maps from the video frames and pass those to a Transformer model to model the temporality. Don't let the title scare you, I hope you find it to be readable :) https://t.co/6fIpHpOo2m
Shared by
Sayak Paul
at
6/16/2021
While training and optimizing models for mobile devices, it makes sense to guide the model design choices accordingly. I think it's hugely underestimated. Maybe reconsider? Here's a precise example of why this matter (via @GoogleAI): https://t.co/r51Pr21jkI
Shared by
Sayak Paul
at
6/13/2021
Quantifying the performance of image generation / modification models merely just by FID or PSNR is cynical IMO. Hacking these metrics is good to get the reins in but in these cases, qualitative results really go a long way. Observation after working w/ @ariG23498 & @ahthanki
Shared by
Sayak Paul
at
6/4/2021
If you ever wanted to get a strong introduction to the manifold hypothesis and how it affects generalization in deep learning bookmark this one: https://t.co/IK6wc6qNk3 @fchollet, THANK YOU for pouring everything into this one.
Shared by
Sayak Paul
at
6/3/2021
I recently gave a talk on the trends in Computer Vision in 2021 I find interesting to work on. Here's my deck: https://t.co/XuejKkLiZo. By no way, it's an exhaustive summary but I hope it'll be useful for those looking for ideas for their next project this summer.
Shared by
Sayak Paul
at
5/29/2021