Gearing Up for NeurIPS 2019…
While we wind down from the recent EMNLP conference, NeurIPS 2019 is just around the corner starting on Dec. 8 thru the 14th! For a quick rundown of the NeurIPS’ metadata (authors, topics etc), check out this post:
The Thirty-third Annual Conference on Neural Information Processing Systems (NeurIPS) is going to be held at the Vancouver Convention Center …
Github:
github.com
This Week:
PyTorch vs. TensorFlow: The Final Frontier
GitHub’s Developer Community is A’Boomin, Same for Data Science! Self-Supervised Representation
Gary Marcus Disses Everyone’s Demo
Part II: Knowledge Graphs Research from EMNLP
Transformers: THE Table To Know 🤯
When we think of the recent evolution inAI development, it is remarkable how the Connectionist School made us forget about the Symbolic School so quickly.
But in recent time, it seems that Symbolic AI is making a comeback (more on this later):
Very excited to share that two of our submissions to #NeurIPS2019 get accepted:
— Jian Tang (@tangjianpku) September 4, 2019
–Probabilistic Logic Neural Networks for Reasoning https://t.co/dY4Doi8CSq
–vGraph: A Generative Model for Joint Community Detection and Node Representation Learning https://t.co/ufjjZkPrpY pic.twitter.com/hdzGnEmMLS
github.com
PyTorch vs. TensorFlow: The Final Frontier
When it comes to AI research, it seams the PyTorch framework is edging out Tensorflow (TensorFlow is still 🔥🔥 though):When I published my PyTorch vs TensorFlow article, some people raised questions about whether it applied to non-NLP conferences.
— Horace He (@cHHillee) November 16, 2019
With NeurIPS posting all their papers, the answer is clear!
Pytorch: 68 -> 166 papers
Tensorflow: 91 -> 74 papers pic.twitter.com/HUUfs13Mrd
More in-depth:
https://chillee.github.io/pytorch-vs-tensorflow/
GitHub‘s’ Developer Community is A’Boomin, Same for Data Science!
Developers are flocking to GitHub contributing to more NLP projects than ever before. In general:
"10M new developers joined the GitHub community, contributing to 44M+ repositories across every continent on earth." Octoverse Report — GitHub
Full Report:
OVER THE PAST YEAR, 10M new developers joined the GitHub community, contributing to 44M+ repositories across every continent on earth…
Here is Kaggle’s survey on the state of Data Science (Python Owns R 😢):
Welcome to Kaggle’s third annual Machine Learning and Data Science Survey ― and our second-ever survey data challenge…
Self-Supervised Representation
Predicting the next word in text (GPT-2) or predicting the masked word in a sentence (BERT) are self-supervised techniques helping models learn from unlabeled data and thus priming the model for smaller labeled datasets for down streaming tasks. Check out how peeps are adopting self-supervision learning in other domains in Lilian Weng’s outstanding blog:
Self-supervised learning opens up a huge opportunity for better utilizing unlabelled data, while learning in a supervised learning manner…
Gary Marcus Disses Everyone’s Demo
In search for AI’s holy grail, AGI, Gary Marcus (of the Yann LeCun debate/arm wrestling fame) recently discovered that GPT-2 can’t do math when demoing Hugging Face’s GPT-2 chatbot used in last year’s NeurIPS ConvAI2 competition:
My first attempt at conversation w GPT-2. It evades questions, flunks basic arithmetic, and gets caught up in probabilities of blank spaces.
— Gary Marcus (@GaryMarcus) November 12, 2019
Conversations with bots like Eliza and GPT-2 can look decent for short periods, but fail if you insist they keep track of anything. pic.twitter.com/ZVNBewtu4J
He then tries out Adam’s GPT-2 talktotransformer.com and also doesn’t like it:
here it is with your system, my next try: pic.twitter.com/gjkUybsBFE
— Gary Marcus (@GaryMarcus) November 13, 2019
Minutes later he demos our BERT Question Answering model (fine-tuned on SQuAD) and trashes it in several tweets 😂😂. Here‘s an example:
in my first attempt I would give it half credit for the third answer, none for the first two. it seems to have trouble with generics, finds associated text instead. based on n = 3; will explore more later. pic.twitter.com/zThW0uAgyl
— Gary Marcus (@GaryMarcus) November 13, 2019
ML community:
On a more serious note, what Gary wants (more reasoning (symbolic) adopted alongside deep learning techniques) is important! In fact, from my personal experience, AI models developed in the private sector are mostly hybrid: deep learning/machine learning + rule-based/graphs.
Which makes the next section a perfect segue…
Part II: Knowledge Graphs Research from EMNLP
Below is what Michael discusses in his latest post:
- Question Answering over Knowledge Graphs
- Natural Language Generation from KGs
- Commonsense Reasoning with KGs
- Named Entity Recognition and Relation Linking
Here is the second part of the review of knowledge graph related papers from EMNLP 2019…
Transformers: THE Table To Know 🤯
If you’re near NYC/Columbia Univ. tomorrow, check out Harry Crane’s presentation on Probability and Complexity!
Lecture on "Probability, Complexity and the Real World"
— Harry Crane (@HarryDCrane) November 14, 2019
Monday, Nov 18, 6-7:30pm
530 W. 120 St. in NYC
Stop by if you're in the area. Should be fun.https://t.co/HxGl5UeU9m pic.twitter.com/gyty7RqH8j
This column is a weekly roundup of NLP news and code drops from researchers worldwide.
Follow us on Twitter for daily updates: @Quantum_Stat