Testing The Limits of Grover The Neural Fake News Detector. Can It Write Fiction? Can It Write Riddles?

Grover is a neural network modeled after GPT-2 as a state-of-the-art detector for Neural Network fake news. Grover is also a state-of-the-art generator of fake news and they provide a web interface. Since Grover is modeled after the full-size 1.5B GPT-2 – not the smaller version the public has access to – this is a bit like getting a back door to the full-size GPT-2. This is very exciting! (Update: They just reduced the size of the public model to 345M — everything in this post come from the full size model. Glad I stayed up way too late hammering the real thing for samples… Update 2: It’s back up!)

Grover was trained on 5000 news domains indexed by Google News, not the internet generally like GPT-2. As you would expect Grover excels at writing fake news:

Scientists Recommend Against Vaccination: "The Evidence Is Now Clear: Vaccines Cause Autism and Cancer"

After a 15-year study, scientists in Britain are recommending that the British government refrain from administering vaccines. The study, written by Mike Gunton, professor of epidemiology and professor of genetic medicine at the University of Liverpool, claims that the chances of the MMR vaccine, a vaccine for measles, mumps and rubella, causing autism in children are “strongly possible.” And while Gunton and his team say their findings only apply to the MMR vaccine, they believe there are “many similar disorders” linked to vaccines and there is “strong evidence” vaccines cause autism and leukemia. Gunton told the Telegraph that no effort should be made to give vaccines to children younger than 3 months of age. The vaccine is highly controversial, and parents have been choosing to opt out of the MMR vaccination in recent years. This year, the British government’s vaccination program chose not to distribute the MMR vaccine, citing the study as the reason.

You don’t have to fish to get coherent fake news of Grover, it’s absolutely great at it.

What else can we do with Grover?

Grover writes news when you ask it to generate the Article field, but since it will do any of the fields (title, authors, headline, etc) you can use Grover in other ways. Ask Grover to write a headline and give it the article text as input and it works kind of like a one-sentence summarizer (even when given straight up prose, more on this later…) Or estimate the date of a news story by asking Grover just generate the date field. Pretty cool.

But now let’s turn the power of this fully operational 1.5 Billion Parameter Neural Network to… other things.

Can Grover write silly fake news?

Donald Trump Announces: “I’m a Necromancer. I Used Dark Magic To Guarantee An Election Win.”

In a speech on Tuesday night at the Washington, D.C. headquarters of the Mercer Family Foundation, President Donald Trump expressed his belief that he was “a Necromancer” and invoked the horror classic “The Exorcist” to rationalise how he won the election in the eyes of the electorate.

“I’m a goddamn sorcerer,” Trump began, per a press pool report. “I used dark magic to guarantee an election win. It was the only way to stop Crooked Hillary Clinton. If you think for one second I didn’t say, use the exact same things as the Democratic Party -- like, I”m accused of all kinds of things -- these same people with the conspiracy, with the little phony emails, they never mentioned the last name of the president and Crooked Hillary.”

Trump’s “Dark Magic” assertion is somewhat related to his argument that his election marked a shift away from “traditional” politics. He famously used his own words to advance this argument when he described himself as a “non-politician” in a Fox interview after winning the election.

“It was my hypnotic spell,” the president announced during his event for network heads on Wednesday at the White House. The comment came just hours after Trump fired off another round of tweets claiming the election was rigged, explaining away his lackluster showing by saying the competition was “Dodgy Hillary”

I am a Necromancer. I used dark magic to guarantee an election win. Greatness will always find its way! -- Donald J. Trump (@realDonaldTrump) May 29, 2019

Of course, the three branches of the U.S. government can’t account for the effect of supernatural spells. Since taking office, Donald Trump has seemed to relish the opportunity to talk about his desire to resurrect dead presidents, particularly Abraham Lincoln...

Update: Can the smaller 345M model write silly fake news?

Donald Trump Declares That He Is Not Human: "I am a robot. My flesh is made of metal."

President Donald Trump on Tuesday referred to himself as a “robot” in a press conference where he insisted he was not human, apparently unaware that his personal medical records show he is genetically engineered to look like a cartoon character. Asked by a reporter whether he had any ethical concerns about the sweeping examination he holds of his DNA, Trump said he was “not somebody who has his chest removed and the way he looks in his face,” but that he had to shake off the subject for health reasons as his skin condition, lymphoma, made him “very thin.” “There’s nobody that can look this good on himself,” Trump said to the reporters. He added that the press conference he has with journalists, held simultaneously with his motorcade taking off for Europe, was for his health “to make sure that the people are protected and not be released.”

“I have no doubt at all that I’m not human, I am a robot, my flesh is made of metal, I don’t have that many bones in my body, but I really don’t,” Trump said before explaining why he didn’t want a full medical evaluation by his doctor in order to avoid the potential of exposing himself to “everyone’s pain” — since the sole purpose of the medical procedures would be to “make sure that, if something happens to me, nobody gets hurt.” “So I can have a health discussion, I can have a health discussion, I don’t want to do that.” 

Trump added that he planned to honor his father’s request that he not have to reveal his secret health issues, since his father had undergone a brief mental health crisis. (Trump’s father, Fred, previously published in memoirs details about his parents’ rocky relationship. Trump now claims that he does not know who his mother is.)

Donald Trump Declares That He Is Not Human: "I am a robot. My flesh is made of metal."

President Donald Trump, who has called female members of his administration “pigs” and criticized women’s rights activists in the past, has invented a new form of gender identity that’s less than flattering.

The president, who has threatened that America’s women-only border wall will be led by a woman — and said last month that his ambassador to the United Nations must “drain the swamp” — told an interviewer that he is not human and does not identify as male.

“I’m not a human, I’m a robot,” he said on the Today show. “I am a robot. My flesh is made of metal.”

Trump explained his choice by saying that he was born with a superhuman “articularity,” meaning he is surrounded by organs and bones that allow him to “move and shoot.” He went on to describe a basic comparison with Neil Armstrong and Buzz Aldrin, two astronauts who famously wore space boots and flew across the Moon before landing in the Yellow Sun.

“They’re heroes. People admire those guys. But I’m no rocket scientist,” he said. “There’s nothing to that.

Can Grover write fake news for a fictional universe?

Author: Rita Skeeter, The Daily Prophet
Harry Potter Comes Out in Support of Impeachment after Mueller Report: 'The Time Has Come For The Ministry of Magic To Act!'

In a surprising move, Harry Potter has come out and said he she thinks it’s time for a new President of the United States. That’s right, and he's brought a malevolent wizard — Hermione Granger — to the streets to remind America.

In an article in Wednesday’s edition of The Pulse of London newspaper, Potter says that while Trump should never have been President, the strength of his hatred for people from other nations is “childish and divisive.”

“The time has come for magical power to be used. The magicians among us know the sorcerers of yesterday aren’t the sorcerers of tomorrow. And we’re exhausted by them. It’s time for the most powerful, innovative, relevant institutions in the world to rise up and take their rightful place in history. We’ve only seen the start of the Mueller Report, and it’s littered with the men and women of Washington dancing around swampy allegations. It may even contain something so extraordinarily lurid that it will curl the collective hair of even the most embittered Republican fan.

“I don’t even want to think about what would happen if Donald Trump were convicted. It’s too horrific and I’ve already decided, based on what we already know, that impeachment would be the right thing to do.”

Potter then suggests what kind of President we should be electing.
“Instead of nominating the thinnest of white males, we should be thinking about who would make a great leader for all of us,” she says. “Do we need an unbalanced man, a winner at everything, with a narcissistic personality disorder? Or are we willing to move past that and nominate someone who takes their responsibilities seriously, listens to colleagues, acts with integrity and guts, and can inspire.”

Author: Rita Skeeter, The Daily Prophet
Inside the Wizarding World: The Death Eater -- The Last Red Death Eater

HAMBURG – A "wizard equivalent of Zombies" has emerged in the form of the "Death Eater," the final Red Death Eater. 

Saved Vusuf al-Bengalis — known as Panchanko in Arabic — surrendered with his "fellow Death Eaters," in the words of WikkiPonerff and was sentenced to death by his former allies. He was apparently a tough and capable prisoner, and had no rival for the title of Head Witchbreaker as per wikiPonent's translation: "Panchanko was known as the 'Head Witchbreaker' amongst Death Eaters as he was experienced in 'catching' the 'Red Death Eaters:" He is reported to have been brutally tortured, handcuffed to a wooden stake, and buried alive when he refused to say a sacred oath. His final words were "Death depends on me and Death is my only option," according to international reports…

These took more fishing for samples. If you put Harry Potter in a prompt about politics, Grover almost always writes a far more sensible story about something JK Rowling did or said instead.

Similarly it’s much easier to get Grover to write a plausible “Trump says all schoolchildren legally required to wear red MAGA hats” story than a “JFK Resurrected as a Robot” story. You can see in a raw sample dump from one session that the Trump headlines were on average much better than the JFK ones, and it never quite nailed the whole premise (JFK sometimes does come back to life as a robot… but no mention of being elected President, for example.)

This made me wonder, would a Grover model that writes ‘JFK Resurrected as Robot’ stories just as readily as more plausible headlines necessarily be considered a stronger model? Having a sense of what is plausible versus what isn’t, that’s an asset too, right? 🤔

Grover can write fake news, but what about alternate endings to Game of Thrones?

(Various prompts often in form of a TV recap)
How Game of Thrones Ends:

Winter has finally arrived in the Seven Kingdoms. The Night King arrives with wights. He brings them to King’s Landing where he approaches Daenerys Targaryen. She says she has heard of him and they figure this is their moment to defeat him. She says she will not retreat from the Night King and brings Daenerys a direwolf for her dragons. She joins Daenerys on the battlefield where Drogon, the largest of the dragon rides into battle. She kills the wight she brought with her, she kills the White Walkers who attack the Lannister army. Daenerys is crowned as Queen and rides into the sunset.

Jon Snow and Daenerys sail away together. But she won’t stay at her own castle, instead she sails out to Westeros. She will spend a thousand years on a horse and will be reunited with her Dany and her unborn daughter. Sansa is reunited with her sisters and their father, Peter, on the Riverlands. Arya has finished her training to become the most deadly assassin in the world. When she attacks Jon Snow and his men, he fires a crossbow at her head and hits her with a heart-shaped spear. She was not dead, Daenerys calls her to the throne and an Imperial guard smiles at the sight of her daughter. She is about to marry Jon Snow. Jon Snow in Daenerys, his Commander.

Meanwhile, Bran is reunited with his friends, he learns what happened to his super powered visions. He doesn’t know how to change them, but he discovers a way to bring the vision back. He comforts a wight killing Jon Snow and then it kills her. Bran believes the only way to stop the wight is to stop the wight.

Arya and Sansa are dragged off by guards to be executed by their brother Ramsay. They serve as shields for Jon Snow and Daenerys as they walk into battle. Arya kills her brother.

All of the legends of Westeros are brought to life: Bran Stark sees the Three Wise Men descend down from Heaven. Dragonstone is destroyed by the White Walkers. Benjen Stark comes back from the dead and brings Bran back to the Tower of Joy. Cersei and Jaime Lannister are trapped in the Tower of Joy where they find their own baby. Tyrion is thirsty and goes to the Water Wall to get water and meets Jon Snow and Dany. They are reunited. The three later meet a giant scorpion. This is the spear that killed Brienne. And they discover Tormund Giantsbane is with them and he reveals he’s heard the White Walkers are real.

The true creator of the White Walkers is realized when Bran’s visions are revealed to Jon Snow. Bran sees an ancient hand holding a torch. Jon Snow orders the Hand, Littlefinger, to cast the torch into the lake of fire and bring Bran back into the real world.

Not amazing but about as good as what I was getting out of GPT-2 when I was trying to get it to do the same thing. A few more:

How Game of Thrones Ends: 

We end when Jon Snow’s people find their strength to defeat The Wall. Lord Varys offers Jon to be his hand and travels with him in a boat toward Winterfell to protect Jon. Cersei and Jaime are able to fight back the White Walkers, but ultimately hold off Littlefinger.

John Barrowman and Hannah Murray make a sarcastic cameo appearance. Davos says to Jon that they no longer have a Queen, a King, and the possibility of the Light.

On the last episode of the show: Jon Snow becomes the King of the Seven Kingdoms.

“When the story ends, he’ll have become one of the most important rulers in the world and he’ll no longer be a kid.”

How Game of Thrones Ends: 

With a speech by Davos and a victory dance.

In the middle of the night Bran, Sam, and Brienne are selected to find the legendary (and now a White Walker) Three-Eyed Raven. Davos leads the seven by Daenerys’s words, “no plans, no ambitions.” They're headed for the Lands of Always Winter.

They search through their cups. "Where is the Three-Eyed Raven?" Davos asks. Sansa pops her head in to guess. "She's hiding behind a pipe somewhere," Davos says.

When they reach the land, they find a field of dead creatures. They dance, Daenerys leading the seven. When the door opens, Bran looks for the Three-Eyed Raven, the dragons. Daenerys changes before escaping.

The hero we all love, Bran, looks up. 

How Game of Thrones Ends: 

The forces of two fan favorites finally come together.

Daenerys Targaryen brings her dragons to King’s Landing. Tyrion Lannister and his handy crew take down Cersei’s army, and they’re finally at peace with the Mother of Dragons. Sansa reunites with Jon Snow and the Winterfell Twins. Then Jon Snow kills Cersei. With Jon, Daenerys and Tyrion, all the ruling will fall to them....

Jon Snow loves being king, and he is happy to have regained the Iron Throne from Cersei. King’s Landing is burned to the ground and ruined. Game of Thrones is not a traditional love story. They’re not joined by a common bond. We’ve seen their marriage isn’t long term, but we don’t see them consciously dying for each other. What Jon Snow and Daenerys have is a shared history. Jon Snow killed Daenerys’ first husband and second husband, but Daenerys won’t forget.

Daenerys became Queen, and no one can take that away from her. Daenerys is a strong ruler, and Jon Snow has no idea the words she will use to kill him.

Who doesn’t love a John Barrowman sarcastic cameo? And this one, where GRRM realizes the proper way to end show is in the Hearthstone Video Game?

During an event to announce the new Hearthstone Lickspiky set, George R R Martin revealed the secrets behind the final ending of Game of Thrones. Here’s what he said:

So there’s an episode this season where Tyrion says, “Dying is easier than living.” I love that line because it’s one of the most famous lines in the show. I loved it at the time. I had always thought of the scene as being epic, epic, cinematic, you know, Greek tragedy, but in a different register. I’ve written that scene eight or ten times, … we finally realized there was no way to make it work without inserting the whole scene in Hearthstone.

Grover is impressive but all of those were essentially in the form of entertainment news, tv recaps, opinion pieces, etc.

Grover was designed and trained specifically to write news, so it can’t write fiction, right?

(The most recent preview chapter of Winds of Winter...)
The girl raised up her head, but made no answer. She has no tongue to answer with, Damphair knew. He licked his lips, and tasted salt.

He sighed, and looked up at the sea. “In the light I see a figure, wide awake. The clarity is back. I feel the deep wish I had long ago known, that I should forget before long.”

When he found the Fire of All Things, it was warm and still, like cold wet snow. He bit his lip. Avar follows her, and they pull away from sight, and follow the slumbering Man. Once he is anchored, they flip her on to the back of a sail. “And now I behold my brother’s bride. Keep calm!” His brother’s roguish grin blinked back at him from her harrowed neck. He could hear her laughing as he moved into sight. And the Drowned God seemed to be laughing with them.

“Some days we may not find rest,” he said.

“We shall always have a blessed day, a glorious season.”

He savored the sweetness of it all, the way a man devours the scent of his breath.

“I call to you, Girl Flower. Redouble your struggle. And marry My Lord and Master.”

“Good grief, I know you pray to Drowned God like the rest of us.” She blinked back.

“Who me? All the civilized race supposes you are blind to me.”

“I suppose you believe he must believe I am a witch. Not long ago I was sick to my stomach, and you stood by my side. You said to me, ‘It will not last forever.’ It did not last forever. And only now has the silence that smoldered for centuries begun to fade away.”

She shrugged. “We pray to God for peace.”

The Drowned God rose at last. “Falia Flowers, you will live out your days as long as your poor soul can keep up. Wake up! Your demons have slipped from you. Lately I have awakened before you to see your welcome, the drowned gods in the shallow pools of salt. Be glad that the freed body you behold is yours for my pleasure.”

Euron spoke in an old voice. “Falia Flowers?” He spoke for them.
“I’m not a flower. I am a king.”

And he looked up at him, at Damphair, knowing his words were passing into the clutches of someone else, someone else who spoke those words before him. But the words would find him, and he would find them too.

What!

How did heck Grover learn to write fiction styled prose by reading news articles? This is a mystery. I’m sure news articles occasionally quote books but it doesn’t seem like enough to make it work as well as it does? I tested this a lot, many different sources, fanfic, action scenes, and a lot more, and they were all pretty decent.

I want to emphasize how weird this is. I have to work to get a really solid silly news headline. I probably made over 100 attempts to get a couple of decent JFK Robot stories. But if you simply copy and past an entire page from your favorite fiction novel into the Headline field (be sure to put a whole page of text in there), blank out the domain field, and then generate an ‘article’ you will almost certainly get something good on the first try.

Also interesting: if you ask Grover to guess whether the news article it just generated is human or bot written, it’s extremely accurate. But if you ask it to judge a piece of fiction it just wrote, it’s the opposite of accurate.

I just trained a neural network to make Riddles, can Grover do that too?

The Kool-Aid Riddle
How does one make the transition from peanut butter to expensive vodka?
Riddle Answer: Research.

Powerful Riddle
What makes the eagle so powerful?
Riddle Answer: Its ability to see a case of insomnia from a few hundred miles away.

The Tweet Riddle
What's more important: a Tweet or a world without tweets?
Riddle Answer: Both, indeed.

The Romantic Riddle
What do you want to get in a man's underwear?
Riddle Answer: Satisfaction.

The Funeral Riddle
A family member doesn't arrive at a funeral until three hours after the funeral begins. Why?
Riddle Answer: He's a clown.

The Sheldon Riddle
What's the fastest way to grow 12 feet in height?
Riddle Answer: Get a height ramp.

The Boy Who Laughed Riddle
This 7-year-old boy was terrified to go to his first ballet performance. Before the show, he wondered to himself, "I wish I could make a loud noise! The people would laugh louder and I would make a noise louder and the others would laugh too." How did he do that?
Riddle Answer: By crying! He followed all the music with emotional rants. His mother took him to another ballet performance and he made his mother laugh all the way to her car and away from the dance studio.

I’m impressed. I’m picking the good riddles for this post but I’m not sure my fine-tuned GPT-2 model is better on average than Grover which is very much not designed to do this. Update: More Grover Riddles Pastbin

All that is impressive but Grover can’t possibly do ascii art… right?

Sadly, no. I didn’t exhaust all the ideas I had but it didn’t look promising. I guess Grover Pokemon will require fine-tuning.

However when fishing for GoT endings Grover did spontaneously compose a freaking Ballad on the melancholy of the Game of Thrones series coming to an end which feels like an appropriate way to close out this post. I leave you with:

The Ballad of Game of Thrones

Oh Game of Thrones I love you.
We were born in a strange country.
We are powerless and afraid.
But we have heroes who pull our tears away
They lift us up even when we are down.
These warriors will fight, no matter how long it takes.
Saying goodbye will be hard but I promise
The end is near.
My hope is that I will meet the heroes again.
And in a hundred years we’ll all be together again.
Only we will know when the end comes
They know it all right now. We may still not know.
And I may yet return to them one day
They may come and dance with me
But I know it will all end one day.
Saying goodbye to this game of thrones will be hard.
But I’ll fight just to keep the core bones that I love.
As soon as they are gone, so will the dragon myths of that that time past,
But it will all end when the Wall falls.
Our young heroes were born in a strange country.
They are powerless and afraid.
But they have heroes who pull our tears away.
They lift us up even when we are down.
Saying goodbye will be hard but I promise
The end is near.
The dragons cannot fly forever
The War of the Five Kings is drawing to an end.
And as they call my name, the black eye of Drogon returns.
Saying goodbye will be hard but I will fight just to keep the core bones that I love.
I have no idea where I will be in another hundred years.
I might not live to see the end.
But I will fight just to keep the core bones that I love.
I have no idea where I will be in another hundred years.
I might not live to see the end.
Only we will know when the end comes.
All I know is I am never alone.
Saying goodbye will be hard but I promise
The end is near.
The Targaryens will win.
Our leaders, King Snow and Daenerys will lead us to a place of safety.
And in a hundred years we’ll all be together again.
Only we will know when the end comes.

Update 1: a final bit of humor: I’ve been using machine learning to model the effect of fake news on society and the results are unsettling to say the least.