NLP News Cypher | 12.15.19

NLP News Cypher | 12.15.19

Busy Week at NeurIPS 2019!

And, we’re back! Ok, NeurIPS comes to a thundering finish as TONS of research was discussed throughout the week. Also, several insightful reports were released covering the state of machine learning in the private industry which we’ll cover in-depth. As a result, this week’s edition will be slightly longer than usual. Cheers!

Quick segue: If you want to view the workshops from NeurIPS checkout SlidesLive:


My Favorite: For those looking to understand how model and hardware architectures contribute to optimal model inference conditions: Check out Viviennes Sze’s talk, an absolute stunner 👏👏👏:

Efficient Processing of Deep Neural Network: from Algorithms to Hardware Architectures

This tutorial describes methods to enable efficient processing for deep neural networks (DNNs), which are used in many AI applications

This Week:

Got 99 Problems But Inference Ain’t One
The Industry Reports on AI
GoodNewsEveryone — A Dataset
Graphs at NeurIPS!
GoogleBrain Got T5 working on Colab, Props!
Facebook Hooks Up with Kaggle
Think Fast and Slow
Papers With Code Joins Facebook AI
Yoshua Bengio Battles Gary Marcus
Rick Perry‘s Got This!

Got 99 Problems But Inference Ain’t One

This medium article takes a closer look at ALBERT (A Lite BERT) and how the quest for smaller model size (while maintaining/improving model robustness) continues to be top of mind of AI research.

Number of parameters is only between 4.7% to 18.% of traditional BERT and training speed is about 1.7x faster while ALBERT achieves significantly better performance than traditional BERT (Devlin et al., 2018) by replacing next sentence prediction (NSP) by sentence-order prediction (SOP).


A Lite BERT for Reducing Inference Time

BERT (Devlin et al., 2018) achieved lots of state-of-the-art results in 2018. However, it is not easy to use BERT (Devlin et al., 2018) in production even small footprint experiments…

The Industry Reports on AI

2 large reports dropped this past week that deserves our attention.

The 1st report, 🔥 AI Index 2019 🔥:

The 2019 AI Index report is here!

The AI Index Report tracks, collates, distills, and visualizes data relating to artificial intelligence…

axXiv Papers

T5 from Google was mentioned due to their Super results on SuperGlue Benchmark:

Since being launched in May, 2019, the T5 Team at Google has almost reached human baseline at the score of 88.9 within five months on SuperGLUE. Human baseline is 89.8

Regarding SQuAD, progress of v2.0 is faster than v1.0:

The F1 score for SQuAD 1.1 went from 67 in August, 2016 to 95 in May, 2019. Progress on SQuAD 2.0 has been even faster. F1 score went from 62 in May, 2018 to 90 in June, 2019.

Machine translation has increasingly become a major commercial product:

MT Systems

The 2nd Report: 🔥 2020 State of Enterprise Machine Learning 🔥

Algorithmia 2020 State of Enterprise ML

Nearly 1 in 5 peeps in the Industry take up to a year for deployment… OUCH!

ML Model


43% of respondents say ML’s biggest challenge is “scaling up”.

GoodNewsEveryone — A Dataset

For those in the sentiment analysis business, a new English dataset dropped this month.

5000 English news headlines annotated via crowdsourcing with their dominant emotions, emotion experiencers and textual cues, emotion causes and targets, as well as the reader’s perception and emotion of the headline.


Corpus of News Headlines

Graphs at NeurIPS!

Want a summary of graphs at NeurIPS, Michael Galkin dropped fire this week. At the conference, conversational task-oriented models merging with knowledge graphs was popular:

Machine Learning on Graphs @ NeurIPS 2019

If you still had any doubts — it’s time to admit. Machine Learning on Graphs becomes a first-class citizen at AI conferences while being not that mysterious as you might have imagined…

Galkin’s article mentioned a few papers, here’s a highlight:

DSTQA yields SOTA 💪 results on MultiWOZ 2.0 and MultiWOZ 2.1 (97% of slot accuracy and 51% of joint accuracy) while performing on par with the best approaches on WOZ 2.0 (still 90+% of joint accuracy).


Multi Domain Dialogue

Hyperbolic space 2D embedding of a tree in a hyperbolic space.

GoogleBrain Got T5 working on Colab, Props!

They used a TPU…

Thread + Link to Colab:


As promised, we have made the Text-To-Text Transfer Transformer (T5) models much easier to fine-tune for new tasks…

Facebook Hooks Up with Kaggle

Want a million dollars? If you do, Facebook launched a DeepFake challenge at NeurIPS:

Deepfake Detection Challenge launches with new data set and Kaggle site

Preventing the spread of deepfake videos is a serious concern for the entire tech industry and for society. We’re taking an important step forward today with the launch of…

Think Fast and Slow

DeepMind released a paper highlighting how researchers should take natural language understanding to the human level. And it starts by coupling a gradual learning system with a fast-learning system.

FYI, this is similar to Bengio’s workshop talk at NeurIPS that you can watch on SlidesLive.


Machine Learning Models

Papers With Code Joins Facebook AI

Papers with Code is joining Facebook AI

We launched Papers with Code in July 2018 with the simple idea of helping the community track newly published machine learning papers…

Yoshua Bengio Battles Gary Marcus

Grab your popcorn, while everyone is eating their holiday dinners, Yoshua Bengio and Gary Marcus will battle for AI galactic supremacy and you can live-stream it!


Yoshua Bengio and Gary Marcus on the best way forward for AI Moderated by Vincent Boucher…

Yann LeCunn be like:

Rick Perry‘s Got This!

When you think of AI, you may think of Minsky, Hinton and

(drum roll)

Rick Perry! 🤣🤣🤣🤣🤣

Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.

If you enjoyed this article please share with friends or social media!

And for complete coverage, follow our twitter: @Quantum_Stat