NLP News Cypher 01.12.19

NLP News Cypher | 01.12.20

The Truth Hurts…

Perhaps, you can tell me what’s best.

I trained a GPT-2 medium model on 18K+ tweets.

Connected the model to Twitter’s API.

Every 30 minutes it says things – sometimes funny…


Thinking of doing a blend including “interesting characters” on Twitter and retraining GPT-2 to have more relevant generation. For those interested in replicating, if you choose to use tweets for your training set, make sure your dataset doesn’t have tweets that reference media attachments like “Hey check this out: [VIDEO]”. If it does, some inferences will not be useful because you tend to generate tweets like this:

If you have cool ideas on who would be fun to add to the training set, hit me up on Twitter or comment below!

Side Note: Cool announcement mid-week. #cliffhanger 👀

Before we start, today’s column was inspired by the anthemic 1966 song “ Wild Thing” by The Troggs. Video Below 👇👇

Video starts with the band standing, instruments in hand, on what looks like a hallway of some kind. They then follow a femme fatale through a door into what looks like a back room but turns out they are actually in a middle of a subway station. 🤯🤯

Lead singer does rattle snake head movement the whole time. 🤟🤟

This Week:

GPT-2 for Tweeting [What you just read]

Neural Module Network Effects

Too Many Recaps, Not Enough Time

Lex’s Vision is 2020

Time for a Fireside Chat?

Reading Comprehension Evaluation Server

Using BERT for NLU

Dataset of the Week: AQuA

Neural Module Network Effects

Nitish Gupta et al. introduces a Neural Module Network model that is able to reason over a paragraph symbolically (arithmetic, sorting, counting) on numbers and dates. It also achieves SOTA on a subset of the DROP dataset.

According to source, code is dropping soon…

Introducing “Neural Module Networks for Reasoning over Text” accepted to ICLR 2020

We extend NMNs (@jacobandreas) for text by (a) introducing modules that reason over a paragraph, performing symbolic reasoning (such as arithmetic, sorting, counting) over numbers and dates in a probabilistic and differentiable manner; [2/4]


Neural Modal Networks

Too Many Recaps, Not Enough Time

Every big tech company’s AI research arm has come out with a “Year in Review.” This past week it was Facebook and Google’s turn. My favorite blog post (from Facebook AI’s review) discussed the challenges of open-domain dialogue:

Advances in Conversational AI

Dialogue research is a crucial component of building the next generation of intelligent agents. While there’s been progress with chatbots in single-domain dialogue, agents today are far from capable…


Facebook AI year in review: 2019

As we start the new year, we wanted to look back at some of our most notable work in 2019. On the Facebook AI blog, we highlighted groundbreaking work that advanced the state of the art…


Google Research: Looking Back at 2019, and Forward to 2020 and Beyond

he goal of Google Research is to work on long-term, ambitious problems, with an emphasis on solving ones that will dramatically help people throughout their daily lives…


Lex’s Vision is 2020

I remember watching Lex’s 2019 video (seen here) and really enjoying it. Well, he has returned. And BTW, NLP gets a big shout-out. Transformers are kind of a big deal. Anyway, lucid recap of the current state of AI across NLP and Computer Vision.

Time for a Fireside Chat?

Wasn’t aware there was a compendium for this. But Microsoft Research shared a collage of various video interviews with the industry’s thought leaders.

MSR AI Distinguished Lectures and Fireside Chats

Microsoft Research AI (MSR AI) aspires to contribute foundational breakthroughs, master Human-AI collaboration, empower organizations to effectively leverage AI…/

Reading Comprehension Evaluation Server

They call it ORB (Open Reading Benchmark). You drop a single question answering model into ORB’s server and it evaluates on several reading comprehension datasets. When submitting your model, they require a docker image that will run on their VM with 4 vCPUs, 1 P100 GPU, and 26GB RAM for eval.

dheeru dua

We ( Ananth, @AlonTalmor, @sameer_ , @nlpmattg) , are pleased to announce the release of ORB, an Open Reading Benchmark.

Using BERT for NLU

A fellow named Olivier Grisel fine-tuned BERT to convert an English user query into a representation for handling NLU on task-oriented dialogue. It was fine-tuned on SNIPS, a voice assistant dataset. The project was partly based on the Alibaba paper:

Below is an example for intent classification/slots filling on a query:

Show Predictions


Joint Intent Classification and Slot Filling with Transformers

The goal of this notebook is to fine-tune a pretrained transformer-based neural network model to convert a user query expressed in English…


Joint Intent Classification and Slot Filling with Transformers

The goal of this notebook is to fine-tune a pretrained transformer-based neural network model to convert a user query expressed in English…

Dataset of the Week: AQuA

We’re doing something new, from now on, we’ll highlight an NLP dataset every week.

Ok… back to AQuA… aka Algebra Question Answering with Rationales.

What is it:

“Algebraic word problem dataset, with multiple choice questions annotated with rationales.”


“question”: “A grocery sells a bag of ice for $1.25, and makes 20% profit. If it sells 500 bags of ice, how much total profit does it make?”,

“options”: [“A)125”, “B)150”, “C)225”, “D)250”, “E)275”], “rationale”: “Profit per bag = 1.25 * 0.20 = 0.25\nTotal profit = 500 * 0.25 = 125\nAnswer is A.”,

“correct”: “A”

Where is it?

AQUA-RAT (Algebra Question Answering with Rationales) Dataset

This dataset contains the algebraic word problems with rationales described in our paper:

Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.

If you enjoyed this article, help us out and share with friends or social media!

For complete coverage, follow our twitter: @Quantum_Stat