Created by Randy Ingermanson - "the Snowflake Guy"
America's Mad Professor of Fiction Writing

Advanced Fiction Writing Blog

Researching Your Novel With ChatGPT

by 21 Comments

in Research

AI tools burst onto the public consciousness last year with a new release of ChatGPT that seems pretty smart to most people. It’s not yet smarter than a human. But it’s surprisingly good. And it’s fast. And it’s cheap. 

I have no interest in writing my novels using AI. My novels should be mine, not somebody else’s. 

But a lot of people use AI as a research tool. As a better version of Google. 

What Could Possibly Go Wrong?

You’ve probably heard that ChatGPT and other tools that use “large language models” are prone to hallucination. Meaning they tend to make up an answer when they don’t know. 

What’s the story here? True or not true? 

I recently had a research question that I took to ChatGPT to see how it would do. As some of you know, I write historical novels, and I’ve read the works of many historians over the years. I’ve read so many, it’s sometimes hard to remember who exactly said what. 

I had a vague recollection that one particular historian (we’ll call him Fred) had written on a particular topic. But I wasn’t sure. A search on Google didn’t bring up anything. So I asked ChatGPT what “Fred” had said on this subject. 

ChatGPT replied with a general high-level summary of Fred’s thoughts on this subject, which it organized as five bullet points. I am very familiar with Fred’s writings, and these certainly sounded like things Fred would say, but I was looking for references, exact quotes.

I asked ChatGPT for a specific reference for bullet point 4. It immediately referred me to one of Fred’s well-known books, which I have in my library. I bought it many years ago, the fifth edition of the book. ChatGPT referred me to the sixth edition of the book, and it said I could find Fred’s thoughts explained well in Chapter 9. It even gave the title of Chapter 9. 

I opened my copy of the fifth edition and found that the title of Chapter 9 was not the one ChatGPT had given. So I went to Amazon and bought the e-book version of the sixth edition, just so I could drill down to the actual words of Fred.

When I opened the sixth edition, I found that the title of Chapter 9 in this edition was the same as in the paper copy I bought 40 years ago. ChatGPT had gotten the chapter title wrong. It wasn’t close.

Worser and Worser

I told ChatGPT it had made a mistake, that it had given the wrong title for Chapter 9, and in fact that chapter had nothing to do with the question I’d asked in the first place. 

ChatGPT immediately apologized for the confusion. It told me the actual material I wanted was found in Chapter 8 and Chapter 10 of the book. And it again gave me the titles of these chapters. 

But both titles were wrong, and those chapters also had nothing to do with the original question. It seemed that ChatGPT was simply digging itself in deeper and deeper. 

I was getting impatient. I told ChatGPT that neither chapter title was correct, and I asked for a direct quote of Fred’s own words. I figured I could then do an electronic search for any prominent words in the quote, and that would take me to the passage where Fred dealt with my question.

ChatGPT again apologized for the “confusion” and gave me a direct quote from Fred. It was two full sentences, about 70 words long. I’ve read enough of Fred’s books to know his voice. The quote sounded exactly like what he would have said. So I searched for several words in the quote. 

They weren’t there. ChatGPT had invented a direct quote from Fred. When people talk about hallucinations of ChatGPT, this is what they mean. 

I challenged ChatGPT again, telling it that I have the book, and the quotation it had given me was NOT in the book anywhere. 

More Apologies

ChatGPT apologized again. I will say this for it—ChatGPT is good at apologizing. I’ve known many people who make up stuff but never apologize for their mistake. ChatGPT at least will admit it’s wrong when you confront it. 

ChatGPT did more than apologize though. It gave me a shorter quotation from Fred’s book, giving me the page number on which the words could be found. I checked. Another hallucination.

My response to ChatGPT was very terse: “I have the book. This quotation is not found in it.”

You can guess what happened next. ChatGPT apologized again, and gave me another quote that sounded exactly like what Fred would say. This time, ChatGPT attributed it to Chater 8. 

But once again, the quote was not in the book. So I challenged ChatGPT yet again, and again it apologized and gave me another quote that sounded perfectly authentic, but wasn’t.

The Bottom Line

By the end of the session, ChatGPT had given me five apologies for incorrect statements. But I never did find out what Fred thinks on the particular topic. 

Let’s be clear. When I started the session, I had a vague recollection that Fred had written something once on this topic, but I couldn’t remember exactly what he said or where it was found. I knew that I didn’t know the answer. 

During the session, ChatGPT confidently gave me five answers that were no better than my vague recollection. They were no worse, but I was looking for better, and ChatGPT was just as bad at pulling up the actual quote as I was. But ChatGPT didn’t know that it didn’t know.

There’s a saying that’s been going around for awhile, that “You don’t know what you don’t know.” This is often associated with the Dunning-Kurger effect, which says (very roughly) that people who are very ignorant of a subject often overestimate their level of understanding of it, whereas highly informed people accurately estimate their level of understanding. 

Or in other words, ignorant people don’t know that they don’t know something. But competent people know when they know, and they also know when they don’t know. 

I am pretty well-read in my chosen time period of history, but there are many things I don’t know. And I’m at least competent enough to know that I don’t know something. 

ChatGPT does not know when it doesn’t know something. And there’s our hazard. 

Because ChatGPT was trained on information scraped from the internet. And now many people are using it to write articles to post on the internet. That’s much easier than doing the hard word yourself. In a morning’s work, you could use ChatGPT to write dozens of articles. 

Which means that future variants of ChatGPT and other large language models will probably be trained on information that was hallucinated by their own previous versions. That can’t end well. 

Should You Use ChatGPT for Research?

Yes, of course you should. I recently asked ChatGPT a math question which I’d been puzzling over for a couple of hours. It instantly suggested a possible solution. That wasn’t quite what I needed, so I pointed out some flaws in its suggestion. It immediately suggested an improvement. That was also not quite what I needed, but it was close. And I solved the math problem. 

The moral is this: Trust, but verify. (That’s an old Russian proverb that Ronald Reagan learned about and repeated often. John Kerry later updated it to “Verify and verify.”) 

Use ChatGPT for ideas. It can be very creative. But verify everything. If ChatGPT gives you a direct quote, with references and page numbers, look it up and make sure. 

Human sources are fallible too, but good human sources written by experts in their field have a huge advantage over ChatGPT—they know when they don’t know. 

The Question Nobody is Asking

by 1 Comment

in Fiction Writing

I read a book review once which began, “This book answers a question nobody is asking.” The reviewer thought that was a bad thing. He thought he was trashing the book. 

But in reality, that was quite an endorsement. If you want to make a difference in the world, you need to begin by asking a question nobody is asking. 

That’s not enough, of course. Questions are easy to ask. Anybody can ask a hard question. Anybody can ask an unanswerable question. Children do it all the time.

Progress happens when you ask a question nobody is asking, but which can finally be answered at this stage in human history, because we now know enough to tackle the question.

In 1903, the Wright brothers were asking how to get a machine to fly when it was heavier than air. A couple of years later, Einstein was asking what if the speed of light had the same value in all reference frames. (In both cases, I’m simplifying things a bit. It’s a bit more complicated than that, but not much more complicated.)

Every Novel Has a “What If?” Question

This matters for novelists because at the heart of every novel is a question. A “what if?” question. 

Most novels use a familiar “what if?” question that has been asked and answered many times before.

But the really ground-breaking novels ask a “what if?” question that is new. 

It’s easy to ask a familiar “what if?” question. When you do, you get a familiar novel. 

It’s also easy to ask an utterly unfamiliar “what if?” question that is too weird for words. When you do, you get a novel nobody wants to read. 

The tricky thing is to ask a “what if?” question that is just right. Not completely familiar. Not ridiculously bizarre. Just right. 

A Little Exercise For You

  • Make a list of some of your favorite novels. 
  • For each novel, what’s the “what if?” question the novel asked? 
  • In what way is that “what if?” question familiar—just like many other novels that came before it?
  • In what way is that “what if?” question unfamiliar—unlike any other novel that came before it?

I hope this is helpful to you in working on your next novel. 

The 80/20 Rule And Novelists

by Leave a Comment

in Career Management

I got interested in the 80/20 rule more than 10 years ago, when Amazon added a feature to their Author Central site that showed each author how they ranked among all the other authors. I immediately realized that this revealed exceptional insights about the earnings of all published authors. (Amazon removed this feature after a few years, maybe because they realized that it gave away more information than they had intended.) 

How could this feature give away “too much information”? Because sales of books follow (approximately) the 80/20 Rule: about 80% of the money is earned by the top 20% of all authors. And the 80/20 Rule is “recursive,” meaning that 80% of 80% of the money is earned by the top 20% of the top 20%. Doing the math, that means that 64% of the money goes to 4% of the authors. 

And likewise, 80% of 80% of 80% of the money goes to the top 20% of the top 20% of the top 20% of authors. Doing the math again, that means that 51.2% of the money goes into the pockets of the top 0.8% of authors. Or, if you want to round those numbers slightly, the top 1% of all authors get about half the money. 

If that sounds like something Bernie Sanders would say, all I can say is that you don’t have be a socialist to acknowledge the simple fact that this is reality. There are deep mathematical reasons for this, and it’s the way markets work. 

If you’re still wondering how this could be “too much information,” I’ll refer you to a blog post I wrote a few years ago that showed how much information you can extract from a single data point. Based on one piece of information Amazon revealed, I derived a mathematical formula to estimate the number of authors who sold at least X books per year. Just plug in X and you get out a very good approximation. That formula was valid in 2014. The formula for our current year would be somewhat different, but not very different. Because the 80/20 rule is a good approximation to reality. 

Is the 80/20 Rule Bad News?

A lot of writers see the 80/20 rule as terrible news. They would summarize the situation as this: “There are a few big winners and a lot of big losers.”

I don’t see it that way. For starters, there are a fair number of people who earn a decent amount of money from their writing. Yes, only a very few earn millions. But many, many authors earn thousands or tens of thousands. And quite a few earn hundreds of thousands. That’s not bad news. That’s good news. It’s possible to get paid well for your writing. 

But there’s better news. The 80/20 rule applies to many different aspects of life. One place where the 80/20 rule probably applies is my famous Snowflake Method. A small number of words can summarize most of the design of a novel. A small amount of design work can provide the blueprint for most of your novel. 

But for this blog post, I’m going to focus on how the 80/20 rule applies to another aspect of life that’s important to all of us—learning new stuff.

An example is in order.

The 80/20 Rule In Learning a New Language

Let’s say you decide you want to learn a new language. As a random example, suppose you decide to learn German. A quick search of the web tells me that there are more than 300,000 words in one standard reference book. 

That sounds overwhelming. How could you ever learn 300,000 words? It would take a lifetime. 

Yes, it would take a lifetime, but I guarantee you that most native German speakers don’t know all those words. The reason is because some words are more common than others. Some German words are extremely common, and others hardly ever get used. 

And the 80/20 rule is a very good estimator for how many words are super common. 20% of the words get used about 80% of the time. The calculation I did above applies here. 1% of the words get used about 50% of the time. 

So if you learned the 3000 most common German words, you’d be able to understand half the words you come across in a typical German document. 

In fact, it gets better than that. If you run the calculation out a bit further, you’ll see that by learning just the 100 most common German words, you’d be able to understand about a third of the words you find in a typical German document. 

That’s actually pretty amazing. You could learn 100 words in a week. And you’d be able to read a German newspaper and see those words all over the place. Even with that very limited vocabulary, you might be able to get the drift of an article. 

Of course, you wouldn’t understand everything. But you’d understand something. And then if you take several more weeks and expand your vocabulary out to 500 words, you’d be able to understand about 41% of the words in a typical German document. That’s amazing! 

No, you wouldn’t understand the whole German language. Yes, you would be able to get along if you suddenly found yourself in a random place in Germany. (Actually, since a great many Germans speak excellent English, you’d probably be fine, even if you spoke no German. But why would you want them to have all the fun of speaking a foreign language, when you could have some of that fun yourself?)

The 80/20 Rule in Learning Anything

The 80/20 rule applies to learning just about anything. If you learn the 1% of the concepts in any field that are most commonly used, you can get a grasp of about half of what gets said in that field. I must emphasize that you won’t have a complete grasp. You’ll have a very incomplete grasp. But you’ll understand it surprisingly well, and that has value.  

As a practical example, you can learn the absolute basics of Photoshop by watching a one-hour tutorial on YouTube. You won’t be a great graphic designer after one hour. But you’ll be able to open somebody else’s amazing Photoshop file and see how they put it together. And you’ll be able to make at least some simple tweaks to it—fix a spelling error, or adjust the colors, or swap out a logo. And that could be extremely useful to you. 

But A Little Knowledge is a Dangerous Thing, Right?

You might be thinking that a little knowledge is a dangerous thing, and therefore it’s dumb to learn just a little about a subject. 

I don’t agree. It’s dumb to learn just a little about a subject and then imagine that you’re an expert. Ask me about the arguments I’ve had with ignorant people who thought they knew more physics than me because they watched some YouTube video for an hour. 

If you only know 1% of a subject, then you only know 1% of that subject. There’s no way to change that. But if you maintain your humility and remember that you only know the basics, you can easily stay out of trouble. The people who get in trouble are the ones who think that knowing 1% makes them an expert. Do a search sometime for the phrase “Dunning-Kruger Effect” to see how common that is. 

So why should you learn that 1%? Because knowing 1% gives you the ability to talk to an expert and ask intelligent quetions that few other people would even think to ask. And when you’re doing research for a novel, the ability to pick an expert’s mind is gold. 

There are experts out there who would be happy to help you figure out something for your novel. Maybe they’ve spent a lifetime studying a particular subject. And you come to them and say, “Look, I’m a complete novice in your subject, but I’ve got this question….” And then you ask something that proves you took the trouble to learn even a tiny bit about their subject. 

Most experts will be thrilled to see that you took that trouble. Most experts will go the second mile in giving you their insights, once they see that you cared enough to learn even 1% of their subject. 

That’s why you might consider becoming a 1% novice. You can’t possibly spend a lifetime on every subject that interests you. But you could spend a month to become a 1% novice. And knowing a little about a lot of things can be valuable. 

Your Best Writing Conference Ever

by 2 Comments

in Writing Conferences

You might have heard that a writing conference is the best place you can possibly go to make the connections you need to succeed as an author. 

I’ve said this many times, and it’s absolutely true. There is no better investment you can make in your writing career than to go to a writing conference. 

But you might have heard other writers with a different opinion. Here are some of the things you might have heard:

  • “I went to a writing conference and it was awful. The editors and agents all stuck to themselves, and all the writers were novices who knew absolutely nothing. It was a complete waste of time and money.”
  • “I went to a conference and I felt so intimidated, I spent the whole time crying in my room. The food was worse than rat poison, and nobody was at all friendly. I hated every minute.”
  • “I made appointments with 2 editors and 2 agents at a conference. The agents shot me down and the editors both asked me to send them something, but then I didn’t hear anything for 6 months. When they finally contacted me, they sent form rejections. The whole thing was pointless.”

Yes, Bad Things Happen at Conferences

It’s true that bad things happen at conferences. Those terrible experiences I listed above really happened to real people. Not just once. Not just a hundred times. They’ve happened to tens of thousands of writers. 

But great things happen at conferences too. Of my 5 closest friends outside my family, I met 2 at writing conferences, I met a 3rd through 1 of those 2, and I met a 4th through a writer’s email loop that I learned about at a conference. (I do have 1 close friend I met at work, so not all my friends are writers.) 

And I have hundreds of friends I met at conferences. Yes, hundreds. They’re not all super close, but I’ve shared good times with each of them, eating meals, talking fiction, hanging out at the bar. (I’m famous for always ordering milk, but most of my friends can’t handle such strong drink and have to settle for something with alcohol in it.)

But yeah, I’ve been to conferences where I felt incredibly intimidated. I’ve been to conferences where it was super hard to get access to the editors and agents. I’ve been to conferences with awful food. I’ve been to conferences where every agent or editor I talked to wasn’t interested in my writing.

But every single conference I’ve ever been to has been a great experience. And I’ve left most of them thinking, “Wow, that was the best conference ever!”

What Makes A Great Conference Experience?

You may be thinking that I must be taking some sort of happy-chemical to make me think awful conferences are actually great. 

Sorry, I don’t. Great conference experiences don’t come out of a pharmacy. 

Great conference experiences come from aligning your expectations with reality and then acting appropriately. 

Here are the 3 realities of a writing conference:

  • All the editors and agents are overwhelmed with a flood of wannabe writers trying every possible trick to get their foot in the door. Which means that every agent and editor is playing defense every second, just to keep their sanity. They literally have to be on guard against manuscripts being shoved under the door of the bathroom stall. 
  • Most of the writers are at their very first conference. They are scared to death of the editors and agents, but at the same time, they’ve been told that this is their one chance to “network” with these bigshots and make a good impression. And they are intimidated by all the other writers, because they see other writers as the competition in a zero-sum game. 
  • A few people at conferences are seasoned writers who’ve been to a number of conferences and know some of the editors and agents. In fact, the editors and agents welcome talking to these writers because these writers are not acting desperate. 

Here’s the secret to having a great experience at a writing conference: Make friends with writers you resonate with. 

Yes, it’s really that simple. And yes, I said to put your focus on other writers. You will eventually also make friends with editors and agents, but that can take a few conferences, and it takes some luck. Whereas making friends with other writers is a sure thing.

So How Do You Make Friends With Writers?

You can make friends with writers in many ways. Here are a few:

  • Ask another writer what sort of fiction they write and then ask intelligent questions about it. 
  • Ask another writer how their writing career is going. They will tell you it’s not going well and will explain at length what’s wrong. If you know something that might advance their career, tell them. Expect nothing in return.  
  • If you see somebody who looks miserable, take the time to talk to them and see if there’s anything you can do to help. 

Your goal here is not to use other writers to advance your career. If you go in with that kind of attitude, other writers will see through you right away. Don’t be That Writer.

Your goal is to be a decent human being who helps others with no expectation of any kind of reward. 

I guarantee that if you talk to even 10 people at a conference with the goal of helping them, 1 of them will resonate with you. You’ll discover that you have a lot in common. By the end of the conference, you’ll be friends. And you’ll leave the conference feeling happy at what a great time you had.

And I guarantee that if you talk to 50 people, you’ll find someone who will turn into a life-long friend. And you’ll leave the conference thinking it was the best time you ever had in your entire life. 

“But I’m An Introvert!”

You may be thinking that you can’t possibly do all this. Because you’re an introvert. Introverts don’t reach out to others. 

Yeah, yeah, yeah. About 95% of writers at writing conferences are introverts. I’m a flaming introvert. Always have been. Always will be. I’m probably more introverted than you. But nobody has to know that when I go to a conference. 

Even an introvert can ask another writer how they’re doing. That’s all you have to do, and the other writer opens up. 

No, you don’t have to become friends with every single writer you interact with at a conference. You can’t possibly do that. You’ll resonate with a few and you’ll not resonate with most. The ones you resonate with may turn into friends. But you can be a decent human being to everyone, even if they’re not nice to you. 

This should go without saying, which means it probably needs saying. A writing conference is not a place to go looking for romance. 

But Shouldn’t You Be Networking?

No, you should not be networking at a writing conference. You should not be networking anywhere. You should scrub the word “networking” from your brain. Never, ever, ever think of another writer as a person who can help you advance your career. That way lies dragons—and the worst of the dragons are envy, greed, and malice. 

Always, always, always think of other writers as fellow runners in the grand marathon of life. Some are running too fast for you. Some are running too slow. But some are running at just your pace. These are your natural pack. Run wild and free with them. Howl at the moon together.

But What About Editors and Agents?

Eventually, you’ll meet editors and agents. This can happen at meals. In the hallway. In the lobby. At the bar. Or when you make an appointment with one of them to discuss your writing. 

Treat editors and agents like anyone else at the conference. After talking with a few dozen or a few hundred writers, you’ll be comfortable doing this. It’ll feel like the most natural thing in the world. 

You won’t resonate with most editors or agents. But some of them, you will. Some will even become friends. Be a decent human being to all of them, and when you find The One, you’ll know. And they’ll know. And at that point, your writing career will change forever.

How long will this take? That’s impossible to say. It could take a couple of conferences. It could take 10. Or 20. It might never happen. Not every writer is guaranteed a slot in traditional publishing. Some writers are happier acting as their own publisher. 

But every writer can go to any writing conference and have a great, incredible, stupendously wonderful time. And all you have to do is treat other writers the way you’d want to be treated. 

Privacy Policy Cookie Policy
Terms and Conditions
>