Justin Johnson   @jcjohnss

Assistant Professor @UMich CSE; Visiting Scientist @facebookai; Previously CS PhD @Stanford. Deep Learning + Computer Vision.



















 

  Tweets by Justin Johnson  

Justin Johnson    @jcjohnss    11/25/2021      

We intentionally avoid data with people due to privacy concerns both through subreddit selection and filtering with face detectors so images with people don’t give great results; it usually ignores people entirely (as in your examples!)
  
          3



Justin Johnson    @jcjohnss    11/23/2021      

If we prompt the model to generate captions in the style of the /r/cakewin subreddit, then 2/5 captions know it's Elmo and another 2/5 recognize it as a cake for kids!
  
    1         6



Justin Johnson    @jcjohnss    11/23/2021      

But a model trained on RedCaps has no problem recognizing the last one as a cake -- 5/5 captions recognize it as a cake, and one nails it as an "elmo cake"!
  
          4



Justin Johnson    @jcjohnss    11/23/2021      

For fun, I've playing with some of the images from Natural Adversarial Objects (https://t.co/Y0EYBpNTm6). Models trained on COCO struggle on these:
  
    1         4



Justin Johnson    @jcjohnss    11/23/2021      

Check out our @huggingface Spaces demo to see what a captioning model trained on RedCaps can do: https://t.co/XWDhi0FCvD
  
    2         6



Justin Johnson    @jcjohnss    11/23/2021      

With 12M+ image-text pairs, our new RedCaps dataset is one of the largest public vision+language datasets. We hope it will be useful for multimodal pretraining, image captioning, and more!
  
    12         68