Busy Week at NeurIPS 2019!
And, we’re back! Ok, NeurIPS comes to a thundering finish as TONS of research was discussed throughout the week. Also, several insightful reports were released covering the state of machine learning in the private industry which we’ll cover in-depth. As a result, this week’s edition will be slightly longer than usual. Cheers!
Quick segue: If you want to view the workshops from NeurIPS checkout SlidesLive:
My Favorite: For those looking to understand how model and hardware architectures contribute to optimal model inference conditions: Check out Viviennes Sze’s talk, an absolute stunner 👏👏👏:
This tutorial describes methods to enable efficient processing for deep neural networks (DNNs), which are used in many AI applications
This Week:
Got 99 Problems But Inference Ain’t One
The Industry Reports on AI
GoodNewsEveryone — A Dataset
Graphs at NeurIPS!
GoogleBrain Got T5 working on Colab, Props!
Facebook Hooks Up with Kaggle
Think Fast and Slow
Papers With Code Joins Facebook AI
Yoshua Bengio Battles Gary Marcus
Rick Perry‘s Got This!
Got 99 Problems But Inference Ain’t One
This medium article takes a closer look at ALBERT (A Lite BERT) and how the quest for smaller model size (while maintaining/improving model robustness) continues to be top of mind of AI research.
Number of parameters is only between 4.7% to 18.% of traditional BERT and training speed is about 1.7x faster while ALBERT achieves significantly better performance than traditional BERT (Devlin et al., 2018) by replacing next sentence prediction (NSP) by sentence-order prediction (SOP).
Medium:
BERT (Devlin et al., 2018) achieved lots of state-of-the-art results in 2018. However, it is not easy to use BERT (Devlin et al., 2018) in production even small footprint experiments…
The Industry Reports on AI
2 large reports dropped this past week that deserves our attention.
The 1st report, 🔥 AI Index 2019 🔥:
The AI Index Report tracks, collates, distills, and visualizes data relating to artificial intelligence…
T5 from Google was mentioned due to their Super results on SuperGlue Benchmark:
Since being launched in May, 2019, the T5 Team at Google has almost reached human baseline at the score of 88.9 within five months on SuperGLUE. Human baseline is 89.8
Regarding SQuAD, progress of v2.0 is faster than v1.0:
The F1 score for SQuAD 1.1 went from 67 in August, 2016 to 95 in May, 2019. Progress on SQuAD 2.0 has been even faster. F1 score went from 62 in May, 2018 to 90 in June, 2019.
Machine translation has increasingly become a major commercial product:
The 2nd Report: 🔥 2020 State of Enterprise Machine Learning 🔥
Nearly 1 in 5 peeps in the Industry take up to a year for deployment… OUCH!
Highlight:
43% of respondents say ML’s biggest challenge is “scaling up”.
GoodNewsEveryone — A Dataset
For those in the sentiment analysis business, a new English dataset dropped this month.
5000 English news headlines annotated via crowdsourcing with their dominant emotions, emotion experiencers and textual cues, emotion causes and targets, as well as the reader’s perception and emotion of the headline.
Paper:
Graphs at NeurIPS!
Want a summary of graphs at NeurIPS, Michael Galkin dropped fire this week. At the conference, conversational task-oriented models merging with knowledge graphs was popular:
If you still had any doubts — it’s time to admit. Machine Learning on Graphs becomes a first-class citizen at AI conferences while being not that mysterious as you might have imagined…
Galkin’s article mentioned a few papers, here’s a highlight:
DSTQA yields SOTA 💪 results on MultiWOZ 2.0 and MultiWOZ 2.1 (97% of slot accuracy and 51% of joint accuracy) while performing on par with the best approaches on WOZ 2.0 (still 90+% of joint accuracy).
Paper:
2D embedding of a tree in a hyperbolic space.
GoogleBrain Got T5 working on Colab, Props!
They used a TPU…
Thread + Link to Colab:
As promised, we have made the Text-To-Text Transfer Transformer (T5) models much easier to fine-tune for new tasks…
Facebook Hooks Up with Kaggle
Want a million dollars? If you do, Facebook launched a DeepFake challenge at NeurIPS:
Preventing the spread of deepfake videos is a serious concern for the entire tech industry and for society. We’re taking an important step forward today with the launch of…
Think Fast and Slow
DeepMind released a paper highlighting how researchers should take natural language understanding to the human level. And it starts by coupling a gradual learning system with a fast-learning system.
FYI, this is similar to Bengio’s workshop talk at NeurIPS that you can watch on SlidesLive.
Paper:
Papers With Code Joins Facebook AI
We launched Papers with Code in July 2018 with the simple idea of helping the community track newly published machine learning papers…
Yoshua Bengio Battles Gary Marcus
Grab your popcorn, while everyone is eating their holiday dinners, Yoshua Bengio and Gary Marcus will battle for AI galactic supremacy and you can live-stream it!
Yoshua Bengio and Gary Marcus on the best way forward for AI Moderated by Vincent Boucher…
Yann LeCunn be like:
Rick Perry‘s Got This!
When you think of AI, you may think of Minsky, Hinton and
(drum roll)
Rick Perry! 🤣🤣🤣🤣🤣
Ricky Perry fine-tunes his decoder transformer in record time with 4 V100 GPUs, in case anyone is wondering.#AI #ArtificialIntelligence #NLProc #MachineLearning https://t.co/PJmQb6H0GC
— Quantum Stat (@Quantum_Stat) December 11, 2019
Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.
If you enjoyed this article please share with friends or social media!
And for complete coverage, follow our twitter: @Quantum_Stat