Sayak Paul (@RisingSayak) | Robocrunch.Net

Sayak Paul @RisingSayak

Calling `https://t.co/zhiBaOSc0b()` at @carted, ex-@PyImageSearch, ex-@TCSResearch | Netflix Nerd

Tweets by Sayak Paul

I trained a baseline ViT S/16 model with `big_vision` and I was able to reproduce everything. 90 epochs of training ✅ 76.23% top-1 on ImageNet-1k val ✅ Took about 7.367 hours ✅ Training logs & checkpoints: https://t.co/8wWRYFsRud @giffmana @__kolesnikov__ @XiaohuaZhai

Shared by Sayak Paul at 5/11/2022

The way the community has responded to this has been beyond incredible. @ariG23498 and I are still processing the depth of the support we received in all honesty. Here's an update. 1/

Shared by Sayak Paul at 5/10/2022

Our tutorial submission on “Transformers for Visual Recognition” (@ariG23498 and I) have been accepted to ACM KDD 2022 (@kdd_news)! Here’s our proposal: https://t.co/hUY9rBBOuN But alas … 1/

Shared by Sayak Paul at 5/8/2022

data2vec vision models in @TensorFlow are now available in @huggingface transformers 🤗 https://t.co/7Kzj3uBrZK Thanks to @GuggerSylvain & @carrigmat! 1/

Shared by Sayak Paul at 5/7/2022

Another day and another dead simple yet useful work from @giffmana, @XiaohuaZhai, and @__kolesnikov__: https://t.co/DEh7nv9gEx "... Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, ..."

Shared by Sayak Paul at 5/5/2022

Additionally, this CaiT uses the "Talking Head" attention. So, this project could serve as a reference for the TF implementation of that. Thanks to @wightmanr for providing PT references of the models which definitely made life easier. 4/

Shared by Sayak Paul at 5/4/2022

The implemented models have been evaluated on the ImageNet-1k validation set to ensure correctness and to improve developer trust. Thanks to the @GoogleDevExpert program for providing #GCP credits that supported my experiments. Onwards now! 5/

Shared by Sayak Paul at 5/4/2022

I find CaiT interesting because it tackles two qs: * Why does the performance of deeper ViTs saturate on relatively smaller datasets? * Can we separate class attention from the self-attention stage from the patches thereby inducing a form of cross-attention? 2/

Shared by Sayak Paul at 5/4/2022

New work! Implementation of CaiT family of vision transformers in @TensorFlow. Code, ported models, interactive demos, notebooks for fine-tuning & inference are here: https://t.co/PSpz2EZUOl 1/

Shared by Sayak Paul at 5/4/2022

@wightmanr have you tried the way DINO does representation pooling on supervised vision transformers? If so, did you notice any improvements? https://t.co/YNLsGjIbeV

Shared by Sayak Paul at 5/2/2022

Dynamic padding is a very common technique in the NLP world yet it's not so common in @TensorFlow. Today, we're sharing code, notebooks, and articles from @carted showing how to efficiently handle variable-length text sequences with dynamic padding. https://t.co/mSeBvttoSN 1/

Shared by Sayak Paul at 4/26/2022

We’ve used the following methods for our analysis: * Attention rollout * Classic heatmap of the attention weights * Mean attention distance * Viz of the positional embeddings & linear projections We hope our work turns out to be a useful resource for those studying ViTs. 3/

Shared by Sayak Paul at 4/18/2022

We’ve also built a @huggingface organization around our experiments. The organization holds the Keras pre-trained models & spaces where you can try the visualization on your own images. https://t.co/bWzVusn9DX Contributions are welcomed :) 4/

Shared by Sayak Paul at 4/18/2022

What do the Vision Transformers learn? How do they encode anything useful for image recognition? In our latest work, we reimplement a number of works done in this area & investigate various ViT model families (DeiT, DINO, original, etc.). Done w/ @ariG23498 1/

Shared by Sayak Paul at 4/18/2022

New project 📽 Implemented DeiT models in TensorFlow & ported the pre-trained params into them. Code, pre-trained TF models, attention rollout visualization, tutorial, etc. all are available here: https://t.co/yfV0GSs4YX 1/

Shared by Sayak Paul at 4/12/2022

DRLoC: https://t.co/zHUYKK00gc Regularizes the training by introducing another loss term based on an SSL objective: predict the relative distances of token embeddings. Achieves promising results with ViTs when training data is scarce. 5/

Shared by Sayak Paul at 4/7/2022

DeiT: https://t.co/fEie46vRoh One of the first works showing the benefits of training ViTs with more regularization, augmentation, and longer training + Presents a simple distillation strategy which I think is pretty cool. 4/

Shared by Sayak Paul at 4/7/2022

Have been reading up stuff on training ViTs the right way lately. Here are five works that aren't leaving me anytime soon! In no particular order ⬇️ 1/

Shared by Sayak Paul at 4/7/2022

TensorFlow variant of ViT-MAE (masked autoencoders) is now available on @huggingface `transformers` & I am so incredibly happy to have contributed it w/ @ariG23498. This was another interesting learning experience. Read on to know why. https://t.co/enmdvKhXSc 1/

Shared by Sayak Paul at 3/30/2022

For me, it was Keras among other things that inspired me to take up deep learning as a potential career and leave a job I didn't like. It helped me boost my confidence too. May it continue to grow, prosper, and evolve. @fchollet THANK YOU!

Shared by Sayak Paul at 3/27/2022

Nice work from @HugoTouvron et al: * Process the residual connections parallelly * Only fine-tune attn layer params for adapting ViTs to higher res * Addition of MLP-based patch preprocessing layers for SSL ViTs (BERT like) https://t.co/Wv09XR9Bmb

Shared by Sayak Paul at 3/26/2022

I've seen many Kaggle Kernels demoing the usefulness and cool features of `timm` but this guide stands out pretty tall. Great start to Tuesday morning: https://t.co/o522BmCDb2

Shared by Sayak Paul at 3/15/2022

For me, it was a domain shift. @nilabhraroy comes with a PyTorch background but picked up @TensorFlow quite fast. I've learned many new things in TensorFlow from him. 2/

Shared by Sayak Paul at 3/5/2022

Humbled to receive the Google OSS Expert Prize with @soumikRakshit96 for our work on explaining & demoing GauGAN in #keras. * Kernel: https://t.co/ywCKuQqg5w * Blog post on https://t.co/ayUd2g1WD5: https://t.co/YHHRavAvKz * Announcement from @kaggle: https://t.co/KSfYXrwIud 1/

Shared by Sayak Paul at 3/2/2022

How to load-test a model that is exposed as a service? This guide provides a really helpful framework for this including code, motivation, insightful commentaries. https://t.co/oGvfrJiHfO

Shared by Sayak Paul at 3/1/2022

For anyone that wanted to play with the new ConvNeXt models. Thanks to @awsaf49 who contributed the fine-tuning notebook. There is still an open call for contributions in the repo in case anyone's interested: https://t.co/lfzEWDWxTA

Shared by Sayak Paul at 2/21/2022

An interesting take on understanding generalization: "Deep Bootstrap". Use a fresh set of samples every epoch to train on a large (and "virtually infinite dataset") first. Models that converge faster here are likely to generalize better. https://t.co/CAas2HptCb

Shared by Sayak Paul at 2/19/2022

I think I speak on behalf of everyone that has contributed to #Keras in any way when I say your feedback has always had a positive impact not only on our work but also on our lives. Thank you, Francois! You definitely make this community vibrant and inclusive by setting examples.

Shared by Sayak Paul at 12/2/2021

New work with @algo_diver! We show how to develop a minimal Ml system that can continuously be adapted w.r.t data drift. We use #Keras, #TFX, and various #GCP services. Details 👇 Huge thanks to @robert_crowe for helping us on this one.

Shared by Sayak Paul at 12/2/2021

Our (@SiddhaGanju & mine) work on flood region segmentation will be there at PyTorch Developer Day today. Details are here: https://t.co/xNzBgRaGm8 @PyTorch made a beautiful poster for our work (see attached) 1/

Shared by Sayak Paul at 12/2/2021

New work with @ariG23498 on implementing Masked Autoencoders for self-supervised pretraining of images. Thanks to @endernewton for the helpful discussions.

Shared by Sayak Paul at 11/25/2021

I'm currently in absolute awe after reading @theaisummer's take on Einsums by implementing a vanilla Transformer block. The dude's got magical abilities! https://t.co/8YsxMKSjEY

Shared by Sayak Paul at 8/31/2021

New example showing how to implement a VQ-VAE in #Keras with the PixelCNN part included. The example goes through several key ideas: optimizing discrete latent spaces, codebook sampling, and above all visualizations! https://t.co/1FTWusXqf7 1/

Shared by Sayak Paul at 7/26/2021

`robustness_metrics` is a mini framework by @GoogleAI that provides three sets of evaluation metrics for image classification models -- OoD, Natural Perturbation, and Uncertainty (calibration). Very promising. https://t.co/KXsUiMbaKl

Shared by Sayak Paul at 7/25/2021

#TFHub models are opaque. If you load a (compatible) model from Hub using `tf.keras.models.load_model` instead of `KerasLayer`, it's not opaque anymore. Here's a notebook investigating this on the BiT (Big Transfer) family of ResNets. https://t.co/ZROcplxpGT 1/

Shared by Sayak Paul at 7/24/2021

New example on Conditional GANs. I think this recipe is important to know if you are into generative deep learning. Disclaimer: This is NOT SoTA stuff. It's more about the workflow which you can extend to high-fidelity datasets. https://t.co/jGB3YbH6Lo https://t.co/QENxzZRpCS

Shared by Sayak Paul at 7/17/2021

Imagine being able to configure your model search space and eval criteria to guide that search space w/ flexibility and convenience. Enter Retiarii by @MSFTResearch: https://t.co/FVNPTpG2Um Support @TensorFlow and @PyTorch. Waiting for JAX now 😅

Shared by Sayak Paul at 7/12/2021

I am very happy about Aditya and @7vasudevgupta as they are a part of @gsoc. It's been a great dose of learning and sharing working w/ them. Look forward to announcements on cool models coming your way! Thanks to the @TensorFlow team for allowing me to mentor :)

Shared by Sayak Paul at 7/8/2021

If you're serving a @TensorFlow model from a GPU-based environment and you're yet to use TensorRT, now would be a good time: https://t.co/CF8YFA3nWr Speed benefits: https://t.co/Llob8OLrEh

Shared by Sayak Paul at 7/7/2021

How does a commoner train a Transformers-based model on a small or medium dataset on commodity GPUs? And still, achieve competitive results? Here, Compact Conv. Transformers (by @AliHassaniJr et al.): https://t.co/TaGEqEjaHU P.S.: The snap is not clickbait. https://t.co/MgjZO7hbF9

Shared by Sayak Paul at 7/5/2021

Benefits of including unlabeled data in semi-supervised learning is well-studied. Amongst all the recent advancements in this area, PAWS (@facebookai) is my favorite. It shines exceptionally well compared to other methods that too requiring fewer steps. https://t.co/iWiL22I93n

Shared by Sayak Paul at 7/3/2021

It was a great learning experience implementing it in @TensorFlow earlier this year: https://t.co/TCiUKN9Z4q The implementation includes all the bits that are needed to make PAWS work for CIFAR-10,

Shared by Sayak Paul at 7/3/2021

In case you were wondering what's possible w/ `model.compile()` & `https://t.co/Lck2QTNscz()` know that pretty much anything 😉 Switch Transformers, CLIP-like models, CycleGAN -- whatever you'd like (all of those are already available as examples FYI). Happy open-sourcing! 3/

Shared by Sayak Paul at 7/2/2021

Implementation & short piece on *AdaMatch* in #Keras. AdaMatch beautifully unifies semi-supervised (SS) learning & domain adaptation (DA). Within just 10 epochs I got a whopping 12% impr. on SHVN. https://t.co/U5i0hDlzh6 Cc @D_Berthelot_ML @BeccaRoelofs & others. 1/

Shared by Sayak Paul at 6/23/2021

AugLy: A new data augmentation library to help build more robust AI models by @facebookai. Supports different data modalities too. How cool! https://t.co/ICNREolbU6 /cc: @DanHendrycks

Shared by Sayak Paul at 6/18/2021

In this example, we build a Transformer-based video classifier. We first extract feature maps from the video frames and pass those to a Transformer model to model the temporality. Don't let the title scare you, I hope you find it to be readable :) https://t.co/6fIpHpOo2m

Shared by Sayak Paul at 6/16/2021

While training and optimizing models for mobile devices, it makes sense to guide the model design choices accordingly. I think it's hugely underestimated. Maybe reconsider? Here's a precise example of why this matter (via @GoogleAI): https://t.co/r51Pr21jkI

Shared by Sayak Paul at 6/13/2021

Quantifying the performance of image generation / modification models merely just by FID or PSNR is cynical IMO. Hacking these metrics is good to get the reins in but in these cases, qualitative results really go a long way. Observation after working w/ @ariG23498 & @ahthanki

Shared by Sayak Paul at 6/4/2021

If you ever wanted to get a strong introduction to the manifold hypothesis and how it affects generalization in deep learning bookmark this one: https://t.co/IK6wc6qNk3 @fchollet, THANK YOU for pouring everything into this one.

Shared by Sayak Paul at 6/3/2021

I recently gave a talk on the trends in Computer Vision in 2021 I find interesting to work on. Here's my deck: https://t.co/XuejKkLiZo. By no way, it's an exhaustive summary but I hope it'll be useful for those looking for ideas for their next project this summer.

Shared by Sayak Paul at 5/29/2021