The Truth Hurts…
Perhaps, you can tell me what’s best.
I trained a GPT-2 medium model on 18K+ tweets.
Connected the model to Twitter’s API.
Every 30 minutes it says things – sometimes funny…
this is so awesome: #rockets
— 21st Century AI Angst (@angst_gpt2) January 12, 2020
In the words of the great rocker, Bruce Springsteen, you just gotta keep on keepin' on. #KeepMingin
— 21st Century AI Angst (@angst_gpt2) January 11, 2020
😭😭😭If you thought the Tesla Roadster was a step in the right direction, just wait 'til Tesla shows you how to get around on a track bike! @GreenCarReports #autonomous #cyberpunk
— 21st Century AI Angst (@angst_gpt2) January 11, 2020
Thinking of doing a blend including “interesting characters” on Twitter and retraining GPT-2 to have more relevant generation. For those interested in replicating, if you choose to use tweets for your training set, make sure your dataset doesn’t have tweets that reference media attachments like “Hey check this out: [VIDEO]”. If it does, some inferences will not be useful because you tend to generate tweets like this:
I agree w so many points here:
— 21st Century AI Angst (@angst_gpt2) January 12, 2020
If you have cool ideas on who would be fun to add to the training set, hit me up on Twitter or comment below!
Side Note: Cool announcement mid-week. #cliffhanger 👀
Before we start, today’s column was inspired by the anthemic 1966 song “ Wild Thing” by The Troggs. Video Below 👇👇
Video starts with the band standing, instruments in hand, on what looks like a hallway of some kind. They then follow a femme fatale through a door into what looks like a back room but turns out they are actually in a middle of a subway station. 🤯🤯
Lead singer does rattle snake head movement the whole time. 🤟🤟
This Week:
GPT-2 for Tweeting [What you just read]
Neural Module Network Effects
Too Many Recaps, Not Enough Time
Lex’s Vision is 2020
Time for a Fireside Chat?
Reading Comprehension Evaluation Server
Using BERT for NLU
Dataset of the Week: AQuA
Neural Module Network Effects
Nitish Gupta et al. introduces a Neural Module Network model that is able to reason over a paragraph symbolically (arithmetic, sorting, counting) on numbers and dates. It also achieves SOTA on a subset of the DROP dataset.
According to source, code is dropping soon…
We extend NMNs (@jacobandreas) for text by (a) introducing modules that reason over a paragraph, performing symbolic reasoning (such as arithmetic, sorting, counting) over numbers and dates in a probabilistic and differentiable manner; [2/4]
Paper
Too Many Recaps, Not Enough Time
Every big tech company’s AI research arm has come out with a “Year in Review.” This past week it was Facebook and Google’s turn. My favorite blog post (from Facebook AI’s review) discussed the challenges of open-domain dialogue:
Dialogue research is a crucial component of building the next generation of intelligent agents. While there’s been progress with chatbots in single-domain dialogue, agents today are far from capable…
Facebook:
As we start the new year, we wanted to look back at some of our most notable work in 2019. On the Facebook AI blog, we highlighted groundbreaking work that advanced the state of the art…
Google:
he goal of Google Research is to work on long-term, ambitious problems, with an emphasis on solving ones that will dramatically help people throughout their daily lives…
Me:
Lex’s Vision is 2020
I remember watching Lex’s 2019 video (seen here) and really enjoying it. Well, he has returned. And BTW, NLP gets a big shout-out. Transformers are kind of a big deal. Anyway, lucid recap of the current state of AI across NLP and Computer Vision.
Time for a Fireside Chat?
Wasn’t aware there was a compendium for this. But Microsoft Research shared a collage of various video interviews with the industry’s thought leaders.
Microsoft Research AI (MSR AI) aspires to contribute foundational breakthroughs, master Human-AI collaboration, empower organizations to effectively leverage AI…/
Reading Comprehension Evaluation Server
They call it ORB (Open Reading Benchmark). You drop a single question answering model into ORB’s server and it evaluates on several reading comprehension datasets. When submitting your model, they require a docker image that will run on their VM with 4 vCPUs, 1 P100 GPU, and 26GB RAM for eval.
We ( Ananth, @AlonTalmor, @sameer_ , @nlpmattg) , are pleased to announce the release of ORB, an Open Reading Benchmark.
Using BERT for NLU
A fellow named Olivier Grisel fine-tuned BERT to convert an English user query into a representation for handling NLU on task-oriented dialogue. It was fine-tuned on SNIPS, a voice assistant dataset. The project was partly based on the Alibaba paper: https://arxiv.org/pdf/1902.10909.pdf.
Below is an example for intent classification/slots filling on a query:
Notebook:
The goal of this notebook is to fine-tune a pretrained transformer-based neural network model to convert a user query expressed in English…
Colab:
The goal of this notebook is to fine-tune a pretrained transformer-based neural network model to convert a user query expressed in English…
Dataset of the Week: AQuA
We’re doing something new, from now on, we’ll highlight an NLP dataset every week.
Ok… back to AQuA… aka Algebra Question Answering with Rationales.
What is it:
“Algebraic word problem dataset, with multiple choice questions annotated with rationales.”
Sample:
“question”: “A grocery sells a bag of ice for $1.25, and makes 20% profit. If it sells 500 bags of ice, how much total profit does it make?”,
“options”: [“A)125”, “B)150”, “C)225”, “D)250”, “E)275”], “rationale”: “Profit per bag = 1.25 * 0.20 = 0.25\nTotal profit = 500 * 0.25 = 125\nAnswer is A.”,
“correct”: “A”
Where is it?
This dataset contains the algebraic word problems with rationales described in our paper:
Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.
If you enjoyed this article, help us out and share with friends or social media!
For complete coverage, follow our twitter: @Quantum_Stat