Graph Enrichment

Graph Enrichment is an elegant way to use Foundation Models and Graph Neural Networks together.

Foundation Models are a very powerful tool for general statistical prediction. Presently, a primary challenge in the field of natural language is to find the optimal ways of using Foundation Models beyond the task of faking understanding of language (“dreaming”).

This short note is about utilizing Foundation Models to generate initial node feature vectors in Graph Neural Networks. After such a Network is trained, the resulting node vectors are used to add additional edges which were not present in the original graph — then retrain the entire network for the "enriched" graph.

For example, if we consider a naturally occurring graph which consists of authors, tweets, and hashtags, it is particularly easy to understand the meaning of connections between the nodes. This website has several such simple author-tweet-hashtag graph networks (“channels”) demonstrating how Graph Enrichment works. Original Edges in the graph are based directly on data arriving from the Twitter API and reflecting the natural Twitter activity of people using Twitter to share links to articles.

As a starting point, each tweet’s node receives a vector representation from a large language model that serves as a feature vector for that node. This reflects the meaning of each given tweet taken separately from all other tweets in the network. Now we apply a Graph Neural Network algorithm (like message passing) and obtain new vector representation that by now reflects, to some extent, content of other tweets by the same author as well as hashtags which may appear in them. This sets the stage for Graph Enrichment. In this case, it is achieved by simply adding new links between tweet nodes which are similar to each other. Then the Graph Network algorithm is applied again to the newly formed enriched graph.

Of course, the area of application of Graph Enrichment is not limited to social networks which have been chosen here to serve as an example due to their relative simplicity and clarity.

One interesting field in which Graph Enrichment can be applied is text processing where linguistic parsing itself can be construed as an application of Graph Enrichment to a sequential representation of text. What starts as a trivial linear graph, gradually evolves and morphs into an elaborate hierarchical structure — a Syntactic Graph.

  Trending Tweets  

Antonio Grasso    @antgrasso   ·   2 hours
The Internet of Things enables citizens, businesses, and governments to interact remotely and at scale with a host of objects. Here are some key concepts. Microblog & social design @antgrasso @LindaGrass0 #IoT #IIoT #DigitalTransformation
 Reply      Retweet   13      Like     19    

More by:   Antonio Grasso
zoom in  

Mark van der Wilk    @markvanderwilk   ·   4 hours
Why does Bayesian model selection find good inductive biases, instead of giving overfitting behaviour? I made some visualisations to illustrate this for my talk at Code is available. Slides: Plots: 1/👇
 Reply      Retweet   5      Like     25    

More by:   Mark van der Wilk
zoom in  

Petar Veličković    @PetarV_93   ·   5 hours
Promoted to Staff Research Scientist @DeepMind! 🎉 It's been such an exciting year. Algorithmic reasoning has made a dent on meaningful problems, and our work is still ramping up, with many updates planned, even in 2021! 😊 Thank-you to all who supported me in this journey! 🙌
 Reply      Retweet   5      Like     468    

More by:   Petar Veličković
zoom in  

Laura Leal-Taixe    @lealtaixe   ·   5 hours
If you missed it yesterday, “Reviewing the review process” is online! great resource for junior, senior, future ACs, PCs, and anyone interested in understanding the review process. @CSProfKGD @georgiagkioxari @dimadamen @SattlerTorsten @cgmsnoek D.Forsyth
 Reply      Retweet   6      Like     20    

More by:   Laura Leal-Taixe
zoom in  

Fei-Fei Li    @drfeifei   ·   6 hours
Inviting visionary, bold, interdisciplinary researchers and scholars working in human-centered #AI to apply to @StanfordHAI ‘s Assistant Professor/Junior Fellow position!
 Reply      Retweet   10      Like     20    

More by:   Fei-Fei Li
zoom in  

Julien Simon    @julsimon   ·   6 hours
In this video, I use AutoNLP by @huggingface, to train a model that classifies song lyrics by genre. Then, I use Spaces to deploy a public web page, where I paste lyrics and predict them. You can it try it for yourself :) #MachineLearning #NLP #AutoML
 Reply      Retweet   20      Like     73    

More by:   Julien Simon
zoom in  

François Chollet    @fchollet   ·   7 hours
I think it's fun that vision models are converging towards an architecture that is nearly indistinguishable from Xception (back from 2016). Depthwise separable convolutions are simply the correct architecture prior for spatial perception, and it's time they become mainstream.
 Reply      Retweet   6      Like     27    

More by:   François Chollet
zoom in  

Microsoft Research    @MSFTResearch   ·   8 hours
The need for collaborative research that addresses global challenges has never been greater. The first ever Microsoft Research Summit convenes more than 300 world-leading researchers and engineers, October 19-21. Register today and join in. Open to all:
 Reply      Retweet   7      Like     21    

More by:   Microsoft Research
zoom in  

Aravind Rajeswaran    @aravindr93   ·   8 hours
This is great news for Embodied AI and Robotics Research. Congrats to Emo and @DeepMind on the deal. Looking forward to seeing more features added to MuJoCo.
 Reply      Retweet   4      Like     62    

More by:   Aravind Rajeswaran
zoom in  

raia hadsell    @RaiaHadsell   ·   8 hours
ML/AI/Robotics folks: you didn't even realise it was your birthday, did you? Github: Stay tuned for open sourcing and feature updates, and congratulations to the DeepMind Robotics Simulation team!
 Reply      Retweet   15      Like     197    

More by:   raia hadsell
zoom in  

clem 🤗    @ClementDelangue   ·   8 hours
T0, the best open NLP model for zero-shot is topping HackerNews today! It just came out from @BigscienceW today and is on @huggingface. Check it out!
 Reply      Retweet   21      Like     145    

More by:   clem 🤗
zoom in  

DeepMind    @DeepMind   ·   8 hours
We’ve acquired the MuJoCo physics simulator ( and are making it free for all, to support research everywhere. MuJoCo is a fast, powerful, easy-to-use, and soon to be open-source simulation tool, designed for robotics research:
 Reply      Retweet   644      Like     2526    

More by:   DeepMind
zoom in  

Tom Goldstein    @tomgoldsteincs   ·   9 hours
This awesome @distillpub article will make you feel like you understand Gaussian Processes 🧐 Then, this awesome book will make you realize that you just don’t 😵‍💫 h
 Reply      Retweet   4      Like     58    

More by:   Tom Goldstein
zoom in  

Niels Rogge    @NielsRogge   ·   10 hours
Big 41-author paper is out, in which they fine-tune T5-11B on a lot of supervised NLP datasets, allowing it outperform GPT-3 on many tasks in a zero-shot setup. FYI, T5-11B is 40 gigabytes for inference, which is still big, but a lot more manageable than GPT-3.
 Reply      Retweet   5      Like     21    

More by:   Niels Rogge
zoom in  

Chaitanya Joshi    @chaitjo   ·   14 hours
Check out an exciting line of work by @vijaypradwi on marrying GNNs and Transformers! 🙌 + Random walk distances are super powerful positional features for graphs + Explicitly separate message passing over structural + positional features is a strong inductive bias for molecules
 Reply      Retweet   4      Like     31    

More by:   Chaitanya Joshi
zoom in  

Gabriel Peyré    @gabrielpeyre   ·   18 hours
The structure tensor is the local covariance matrix field of the gradient vector field. It encodes the local anisotropy of an image. At the heart of anisotropic filtering and corner detection.
 Reply      Retweet   39      Like     283    

More by:   Gabriel Peyré
zoom in  

Pei Zhou    @peizNLP   ·   10/16/2021
Super excited that this simple idea to externalize knowledge-grounding in RG seems to be working😃! Project from my (second-time!) intern @AmazonScience Alexa AI. Huge thanks to my co-authors @AmazonScience @nlp_USC @USC_ISI!! Paper preview: 🧵 [7/7]
 Reply      Retweet   7      Like     14    

More by:   Pei Zhou
zoom in  

Niranjan    @b_niranjan   ·   10/16/2021
Can we build multihop QA datasets that model’s can’t game easily with disconnected reasoning? Our new paper, MuSiQue, aims to address this via a careful composition of single hop questions. Work led by @harsh3vedi w/ Ashish Sabharwal and Tushar Khot from @allen_ai. 🧵👇 (1/6)
 Reply      Retweet   10      Like     33    

More by:   Niranjan
zoom in  

Kawin Ethayarajh    @ethayarajh   ·   10/16/2021
How can we measure dataset difficulty? And how can we understand *why* a dataset is difficult for a given NLP model? To answer these questions, @YejinChoinka @swabhz and I take an information-theoretic perspective of difficulty. @allen_ai @stanfordnlp 1/
 Reply      Retweet   59      Like     291    

More by:   Kawin Ethayarajh
zoom in  

Yanai Elazar    @yanaiela   ·   10/14/2021
Advancements in Commonsense Reasoning and Language Models make the Winograd Schema look solved. After revisiting the experimental setup and evaluation, we show that models perform randomly! Our latest work at EMNLP, w/ @hongming110, @yoavgo, @DanRothNLP
 Reply      Retweet   52      Like     193    

More by:   Yanai Elazar
zoom in  

  More from:  

Fei-Fei Li
Prof (CS @Stanford), Co-Director @StanfordHAI, CoFounder/Chair @ai4allorg, Researcher #AI #computervision #ML AI+healthcare

Antonio Grasso
Entrepreneur, technologist, sustainability passionate | FRSA | Digital creator, B2B influencer | Author, researcher, speaker | Member @spj_tweets| @antgrasso_IT

Petar Veličković
Staff Research Scientist @DeepMind | Affiliated Lecturer @Cambridge_Uni. PhD @TrinCollCam. Ex-@Mila_Quebec @BellLabs. GDL, GNNs, Algorithms. 🇷🇸🇲🇪🇧🇦

Laura Leal-Taixe
Professor at the Technical University of Munich. Head of the Dynamic Vision and Learning group.

Mark van der Wilk
Lecturer (Assistant Professor) in Machine Learning at Imperial College London. Interested in automatic inductive bias selection using Bayesian tools.