Your AI views are better than they used to be, but are rife with cognitive dissonance.
You acknowledge that all the negatives that people see are real: AI writing is bad (even if people cannot necessarily tell why it is bad), data centers suck, Chatbots lie and give inaccurate information with confidence, and chatbots are now included in all kinds of apps for no reason except to frustrate people and suck up power. I am glad you acknowledge these things are true and suck. That's a start.
However, when you try to imagine why some people hate AI so much, you still focus on strawmen and psychoanalysis. You repeatedly insist that dislike of AI is actually just anxiety about the future and specifically about the job market. Certainly, that is not a concern for me. I am simply concerned that AI, as it exists, sucks at things that are not backend tech work. Not everything needs to be about "anxiety." AI is now very hard to miss in daily life, and most people do not like the role it plays in their own lives.
You defend Megan McArdle for relying on the tech but are defensive about its use in your own work. If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech. I hope this is the start of an enduring social stigma.
If you are a writer and use AI to edit and check sources, you must know you could do that with grammar check and Google before these chatbots came along. If you go further and believe that your use of AI is beneficial to your work in some big way, you should be able to point to some benefit to readers.
They acted as if the problem with people disliking AI was some general Lefty thought disease akin to wokeness, not that it was trained on the corpus of the work of people it seeks to replace-without compensating them or seeking their permission! Anthropic lost a huge suit for training on Stolen work. Meta also knowingly trained on stolen work.
The New York times is suing openai for reproducing New York times content virtually word for word. The lawsuit has a hundred examples. You can look them up. There is no question. It is a theft and plagiarism machine and most writers don't like the idea of their work being taken without their permission and compensation, but they especially don't like the idea of it being used to train their replacement! That is not a lefty woke woo woo sentiment.
"If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech. I hope this is the start of an enduring social stigma."
A-fucking-men.
I'll add that the negatives are not limited to it not just not being good enough at X. I say this because the constant refrain is "they're getting better just wait soon you won't be able to tell the difference". But being able to tell the difference is not my issue with them. Making an AI that is convincing enough to trick me just means that now I'm easier to lie to. But that doesn't excuse the lie. It doesn't make the high tech lying machine ethical to use.
If the AI enthusiasts are right, then it seems like we should all be able to agree that AI use can be fully disclosed and the technology will still be welcomed by the general public with open arms. AI novels will sell, AI music will be celebrated, AI art will be hung in museums.
On the other hand, if disclosing AI generated content results in people avoiding that content, what then? We know what then. Deception and lies, that's what.
"If the AI enthusiasts are right, then it seems like we should all be able to agree that AI use can be fully disclosed and the technology will still be welcomed by the general public with open arms. AI novels will sell, AI music will be celebrated, AI art will be hung in museums."
Would you consider someone like me who finds AI very useful in certain aspects of my job an AI enthusiast? Or are you reserving that for people who specifically want art/text that's fully AI generated?
Because I think I'm in the majority recognizing that it's super useful for some things, but fully AI-generated art is probably not that interesting for most people. I don't know why you'd think AI being useful for certain things (the enthusiast view) implies AI art dominates.
I was painting with a really broad brush there. Yes, I meant people who are enthusiastic about AI art. And I agree, AI has plenty of useful applications, even if I think that on net it's doing more harm than good.
Also, am I crazy or did Katie in a previous episode straight up admit to running Drink Your Way Sober through ChatGPT? If I’m not mistaken that’s similar to what “Mia Ballard” claimed happened with Shy Girl.
This comment is very insecure and in the process gets the usefulness of AI fundamentally wrong.
AI being bad at writing doesn't mean that AI is bad at critiquing writing; it's not a person, its skillset is unevenly distributed.
Chatbots give incorrect information with confidence if you're bad at prompting them and don't double-check important facts, but they cite their sources now and will stop lying if you just ask "is that really true?". The advantage they have over Google is that they can take a plain-English query and render it into accurate results, without relying on a keyword guessing game. They have approximate knowledge of the state of every field of human expertise, and that's enough to point a search in the right direction at an unprecedented rate compared to raw Google searches.
Chatbots don't suck up that much power, it's just not true. You can easily use more power playing videogames or driving a car for equivalent amounts of time, and these are activities people do for way longer per session. And the water issue is wildly false, even less real than the power one.
The "psychoanalysis" you're complaining about is a strained attempt to be charitable to people like you who blunder into discussions either wildly wrong about AI or intentionally dishonest about it. It's kinder to empathize with why you're so vitriolic than to call you a bunch of hysterical liars, and J&K are kinder than I am.
They do use a lot of power a) when you're training them and b) when they are doing "reasoning" which is long chains of inference operations. It's not doing nothing for those 5 minutes of "thinking" time per query. That's computation in a datacenter.
They cite sources but a) quite a lot of the time, still, the sources don't exist and b) very, very often the sources don't actually say what the chatbot says they say, which makes sense given the chatbot hasn't read them, or indeed ever read anything in a meaningful sense.
"prompting" IS a "keyword guessing game" or "key phrase guessing game". Except it's worse because it will tend to confirm what's in the prompt. "discuss X" gives less biased information.
Chatbots are fine, especially with a long list of instructions never to be sycophantic etc. When you start researching something it can be hard to know what the most central papers are on a topic, and it can help find those quickly. It summarises them I'd say with about 80% accuracy. If you are after important facts, like "how long before this mouse starts to show degeneration" that will often be wrong. You need to actually find out.
Once you've used it to get a few papers the method of finding sources by citation chain works better.
It can be useful for suggesting hypotheses and spotting gaps in the literature. However: it makes obvious suggestions, and I'm required to declare that I've used it in this way. So then it looks like my ideas came from a chatbot. Fuck that, it's not worth as nearly much as a declaration "AI wasn't used" or "AI was used during early stage literature survey".
I asked it to assess a recent proposal I wrote, which I didn't use it for, except for limited early literature survey as I describe above, and it said
"Conceptual clarity Very high
Appropriate detail Very well balanced
Experimental logic Strong
Writing efficiency High
Overall competitiveness Very high
the proposal is well-constructed, appropriately detailed, and highly competitive."
That's because I wrote it.
It also said
"The text is most consistent with human-authored grant summary writing, with no reliable indicators of full AI generation. AI-assisted editing remains possible but cannot be substantiated from linguistic evidence."
So why would I ever risk adding my work to the pile of self-similar AI shit reviewers are wading through? I wouldn't. I'd try to write something that stands out as having my voice as well as being well argued. Probably the ability to do that will become more valuable as time goes on, in journalism as well as science. It's certainly what I want to read.
"[Chatbots] will stop lying if you just ask "is that really true?"
This is insane, right? If the chatbot could have told the truth, why'd it lie the first time? Why must one ask "is that really true?" And why does asking this cause it switch from lying to telling the truth?
Chatbots that haven't been trained spit out random-ish text based on pattern prediction.
The "is it true" data represents a smaller proportion of the pie. Literally separate training sessions. So you have to tap into it by prompting the AI to think "oh, this is one of those scenarios where I get the cookie for being honest, not for just going through the motions".
The "cookie for being honest" analogy is pretty different from how chatbots really work. Importantly, chatbots have no concept of truth or honesty. They're just "predicting" the next piece of text, over and over again. This process often leads to output we consider "true", but that's not because the AI went into tell-the-truth mode, it's because the AI is mimicking the text it was trained on (for the most part - the post training reinforcment influences it too), and there's a good correlation between probable text and "true" text.
I don't know what you mean by separate training sessions. There's a main training session and there's post-training reinforcement. The main training session is basically an extremely intensive, extremely time-and-energy-consuming game where the chatbot plays "guess the next word" over and over again (really next "token" rather than "word", but the difference isn't important). So the AI takes some chunk of text from its training data with a defined start and end, and then it guesses what comes next. And then it compares its guess to what actually came next, and uses this to make updates to its "weights" (aka "parameters", or just "numbers in matricies") via a very clever algorithm. And current chatbots have over 1 trillion of these "weights". I like to picture this as a giant control board with 1+ trillion little knobs that get continuously turned by the training algorithm. And as the chatbot goes through this "guess the next word" process over and over again with more and more training data, it slowly turns those knobs to positions that make it better at guessing the next word. After it's been trained, it can take input and once again play "guess the next word" over and over, except this time its guesses are printed on a screen as output. That's the basic gist of how chatbots talk to us.
Importantly, "truth" and "honesty" are just irrelevant to this process. They don't factor in, except as a kind of second-order happenstance. The chatbot has no idea whether what it's writing is true. It's doing the exact same thing when it "hallucinates" as it does when it states correct facts: it is guessing the next word.
So the reason why chatbots tend to give better answers when prompted with "is that really true?" is because the particular string of text that comprises "is that really true?" is associated with there being something wrong with its last string of output text. And not wrong in just any way, wrong in some complex but systematic way that is exploitable using those 1+ trillion little knobs that the chatbot uses to guess the next word. The AI isn't internally identifying anything as "true" or "false" - it's responding to the really big hint you gave it when you prompted it with "is that really true?" And that really big hint certainly doesn't guarantee that the next response will be correct, it just (usually) increases the probability.
As an important side-note, this method has a major deficiency: in order to give the chatbot a big hint that helps direct it toward something true, you have to already know the truth. It certainly doesn't.
You're just wrong, truthfulness was one of the main targets for Reinforcement Learning through Human Feedback training, which is the main vector through which LLMs were trained to be useful products. Tens of thousands of employees were tasked with rating AI responses on their truthfulness and if the AI did well it was rewarded in the feedback loop. If it did poorly it was penalized. But there were a lot of qualities that the LLM was trained to have and not every project was about truthfulness, so if you nudge the LLM to be truthful you pull its understanding of the context towards that portion of its training data.
All the stuff you described with trillions of knobs describes the first layer of training, before the RLHF that turned it from a token predictor into a useful chatbot.
Yes, I did skim over reinforcement learning in my summary of how LLMs work. I agree, fine tuning the trained LLM for truthfulness will make truthful responses more likely, at least in situations similar to those for which human feedback was given. I would trust such an LLM more than one that had not gone through reinforcement learning focused on truthfulness, at least if I believed I was prompting it about something that is within the domain of its training / RLHF. But I'd still keep in mind that the LLM probably doesn't have any conception of "truth" or "honesty".
I do not agree with this:
"if you nudge the LLM to be truthful you pull its understanding of the context towards that portion of its training data."
I see what you're getting at, but you're assigning human properties to a highly complex, mostly uninterpretable next-token prediction algorithm. It doesn't identify truth vs. falsehood and then use context to decide which to deliver. It just does the one thing it always does: it generates probability distributions for possible next tokens and then selects from them. I suppose it's *possible* that there are some under-the-hood matrix calculations going on that are doing something that could be reasonably interpreted as "deciding to be honest". But I'll believe that when someone presents credible research that turns such a claim into something other than pure speculation. In the meantime, we've all seen example after example after example of an LLM demonstrating behavior that is entirely consistent with mindless next-token prediction and hard to account for otherwise. I will grant that the technology is improving to the point where they make major errors less often. But I'm guessing this *isn't* because something magical has happened.
My apologies if you're not claiming anything stronger than "RLHF increases the probability of truthful output by some amount across some set of scenarios".
"If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech."
This just doesn't follow or make sense in the context of the situation with McArdle. It's easy to get certain usefulness out of AI, so no, it doesn't make sense to disdain anyone who uses it.
There's a huge gap between McArdle finding AI useful in some areas of her job, and people "relying" on shitty tech. Sometimes it really is just using available tools to do better work. There's no reason for the generalized anti-AI stigma you're hoping for except in cases of plagiarism, AI-generated text, etc. where there's *already* a stigma (and there should be).
My ex is an internist- and he says that AI is great at diagnosing. He says the only time it screws up is if he has forgotten to put in all the available information. He’s 58, and is a phenomenal diagnostician.
If you can't tell whether or not a writer uses AI in some way, how can it be a negative? I think the cognitive dissonance is on your side for not acknowledging that there are benefits too (and they don't necessarily have to be beneficial for the reader either.
I just see a continuing misunderstanding of what "AI" is doing, and what can be usefully accomplished by such a tool.
It's fine at "opinions" even if there's no mind there to think it. The output texts can reasonably simulate an opinion on something. Similarly it's fine at critique, and some pattern recognition if you don't require accuracy.
What an LLM will never be good at is perfect accuracy. Hallucinations will never go away, so unless you're comfortable with "there's an X% chance that this article exists and a Y% chance that is says Z" you will always have to independently verify its outputs. If you don't want it to make mystery edits to your texts you need to ask it to highlight your mistakes, but you then need to make the changes yourself in a separate program (or else be comfortable with random error insertion).
Jesse seems to grasp this, but Katie seems not to.
Maybe this isn’t necessary to add, but the ways in which an LLM will be inaccurate is significantly different from the ways in which a person will be inaccurate, which is why we use anthropomorphic terms like hallucination rather than just saying error. A person makes logical mistakes, or errors rooted in emotions or ideological bias. A statistical token associating computer system makes statistical token association errors.
I hope Lindy divorces her gross husband and girlfriend, goes offline far from the northwest for 18 months and comes back fucking jacked as a new health influencer.
A big magazine profile and photo spread from a old-school warehouse gym (barbells and chalk) to announce the return needs a clever headline...
If a person wanted to get a little psychoanalytical, it actually makes sense IMO that Lindy, who is clearly deeply troubled and in need of reparenting, has basically created a situation in which she has a new mommy and a new daddy.
I love you guys, and I agree that the political left gives bad anti-ai arguments. However, I think you are way too glib and dismissive of peoples’ ai anxieties more generally. I’d suggest doing a deeper dive and developing a more nuanced view.
Jesse and Katie need more nuance when it comes to AI checkers, too. I am a college professor. I refused to use the checkers for a long time because of worries about false positive rates. I was once falsely accused of plagiarism in eighth grade and would never want to do that to someone. But Pangram really is quite good.
I now run every piece of student writing through Pangram before grading it. If I get the green light, then I can read with less vigilance and just concentrate on being an educator rather than a policeman. If AI usage is flagged, then I look for other evidence, which I can usually find either in the submitted text itself or through comparison with an in-class writing exercise I have them do early in the semester.
Students rarely use AI in a thoughtful way, in my experience. Those who use it merely copy and paste, sometimes running the initial result through a humanizer before submitting it to me.
Educators still need to assign writing because we learn to think more clearly and rigorously when we write. Writing is thinking! And we can’t do all the writing during class, especially at the university level where we typically get our students for three scant hours per week and have oodles of content to teach. Oral exams are also not the answer; not only do they not substitute for the writing process, they can’t be scaled to the number of students most of us teach.
I loathe playing cop. But if educators don’t enforce academic integrity, students don’t learn, and their degrees will soon be regarded as worthless scraps of parchment.
Pangram has good privacy policies. This is essential to me because I value my students’ privacy. It does not use the submitted data for anything except the check that I have requested. It claims the lowest false positive rate in its field, and my own testing confirms that, as did Katie‘s brief experiment with it.
Yet our hosts dismiss all of the checkers as garbage. C’mon, guys! I need you to be perverts for nuance here!
The number of students is the problem. The student body should be a third what it is. And Oral/written exams are fine. If people can't pass them, they shouldn't be in college.
I don't think leftist critiques of data centers draw enough attention to the way they inflate the cost of living for people with the misfortune of having one built in their community. They drive up utility bills, often in rural areas where access to water and electricity is already tenuous. Politicians often give out tax breaks to entice companies to build data centers in their towns because they erroneously think they "create jobs"; They don't. The centers are built by short-term contractors, most of whom are basically nomads from out of town who hop from site to site. Once the data centers are built, they only require a few dozen people to run. So you're increasing the cost of living for literally everyone in a community for the sake of creating a puny number of jobs.
There are emerging claims that the light and noise pollution caused by data centers is also bad for communities, but I regard those claims with more skepticism. The strain they create on utilities and the resultant cost passed on to residents is a much more substantial problem.
I want to be clear that I know data centers are used for things other than AI, notably social media and cryptocurrency. That's one of a couple reasons I've cut way back on social media. I've always hated cryptocurrency because I recognize it as a bigger fool's scam, so I have no issue abstaining from that. That said, social media platforms seemed to be running just fine before this big uptick in new data center construction happened. It's obvious that these things aren't being built because they fulfill some kind of actual demand, they're being built because of this big dumb AI arms race going on between tech companies, and because "new data center" sounds sexy to shareholders. And now companies are adding unwanted AI bloatware to everything to try and justify this enormous waste. My fucking refrigerator does not need AI.
Have you seen videos of the communities near these data centers. I can't speak for light pollution, but the noise is RIDICULOUS. And most of the videos I've seen aren't exactly next to the facility, it travels pretty far. It would be like living next to a neverending freight train.
Last week, I saw a news report about a suspected cancer cluster near an Amazon data center. Now, this could be coincidence, so I’m awaiting more evidence. But I’m also adding physical health effects to my long list of open questions about how data centers impact quality of life.
Agreed. They both seem way too certain about AI and dismiss criticisms in general. But when they talk about how they use AI specifically themselves they see all the limitations.
Agree, it would be especially great to have Brad palumbo as a guest on during that episode. He's done a pretty good job of debunking all the anti-environmental nonsense.
I can't find his link right now but here's another one
That’s one part. Honestly, the disdain for Tucker has reached derangement levels. The guy blinks and people go crazy. He’s basically saying the same things he always has, it’s just that MAGA has moved in a different direction. Candace, on the other hand, is both insane and kinda evil. She spouts a lot of antisemitic garbage, but even worse is the harassment campaign she’s been running against Erika Kirk for months, posting at least a dozen full videos dissecting Erika’s life. Candace believes that Erika and Israel were behind Charlie’s death, and that Tyler Robinson was framed, and she’s managed to convince her legion of braindead fans of it. Charlie Kirk himself has unfortunately been turned into a kind of prop, with everyone who ever met him claiming that he supported the same things that they do.
Megyn Kelly landed in hot water for a couple of reasons: she’s friends with both Erika and Candace, and so refuses to weigh in on any of this (why Erika still wants to be friends with her is beyond me). She also started criticizing Israel late last year, and didn’t support Trump’s actions in Venezuela or Iran. This has led to people like Dave Rubin, Bari, and Moynihan boycotting her show. Her biggest critic is probably Ben Shapiro, who was good friends with Megyn, then gave a speech criticizing her without reaching out to her first—MK did not take this well. Piers Morgan then got pulled into this when he mentioned on his show that Ben had stopped responding to his text messages, and Ben responded on his own show by comparing Piers to Jerry Springer—Piers did not take this well. Mark Levin also got pissed at Megyn for questioning Israel and started going after her in kind of an incel-ish way (started calling her “Grandma Groyper,” etc.) Megyn ignored him for a while, then gave Levin a nickname of his own: Micro Penis Mark.
Nick Fuentes is basically a footnote at this point. Even he’s said Candace has gone too far attacking Erika (he also called Tucker an antisemite). Glenn Greenwald is crashing out, turning into a sad caricature of the self-hating Jew. Dave Smith is Dave Smith, and that drives a lot of people crazy for some reason. The people who support the war in Iran, like Ben and Batya, spend a lot of time denouncing “the podcasters” and calling them irrelevant. Through all of this, Matt Walsh has emerged as the voice of reason, calmly debunking the conspiracy theories about Erika Kirk and speaking out against the Iran War. Brad is a much smaller figure, but he’s managed to stay calm as well.
Ranked from most to least important, I’d say the biggest issues dividing the right are Israel/antisemitism — Candace and Erika — Iran. But as you can see, it’s become a giant clusterfuck. Turns out the trans furries won—Charlie Kirk was killed and his death fractured the right.
I’m committing the sin here of commenting without listening to the episode but it’s clear from Jesse’s twitter that he takes AI existential risk — that is, the scenario where AI development is automated and they rapidly become better at everything than humans, economically displace us and then send us the way of the dodo — entirely seriously. So like when someone says they’re anxious or distrustful of seeing AI pushed everywhere, surely you should be treating their anxieties as ‘directionally correct’ rather than dumb leftist misinformation, even if you think they’re worried for the wrong reasons.
I enjoyed the aside about the word tulle at the beginning of this episode because it brought to mind this chart that I saw online a few years ago, Words Known Better by Males Than Females and Vice Versa: https://observablehq.com/@yurivish/words
Apparently about 80% of women know the word tulle but only about 25% of men do!
I'm honestly surprised that the anti-LLM movement has gotten blue coded. I hate LLMs, and it's not because of data centers and the environment. It's because I think having human-impersonation machines all over the place is a bad idea, generally. I know the technology is cool, and I know there are positive use cases, but I think they're outweight by the negative use cases. I mean, we're talking about MACHINES THAT ARE GOOD AT PRETENDING TO BE HUMAN. It can't take a whole lot of imagination to come up with reasons to not want these things around.
I'm not gonna list them here. I just want to point out that, if anything, an aversion to MACHINES THAT IMPERSONATE PEOPLE seems like it should be red-coded. Or at least religious-coded. Y'know, souls and all that. I'm an atheist, and I think I'm appreciating for the first time in my life what it's like to see something as "satanic". Human-impersonation machines are satanic. I mean, I don't believe in Satan. But if I did, I'd bet foisting LLMs upon humanity is just the sort of thing he'd get up to. Maybe "misanthropic" is a better word.
I know everyone likes to say they understand that the AI isn't really a mind, but the illusion that one is communicating with a conscious being is just so damn powerful. But it's fake fake fakity fake and this matters. For example, Jesse and Katie and most other people express frustration that they'll give ChatGPT some instruction and then ChatGPT will say it understands but then it doesn't follow the instruction. No one should be the slightest bit surprised by this. There's no mind! There's no understanding! It is fundamentally incapable of understanding your instruction because "understanding" plays no role. It's just taking in your text and doing a bunch of fancy math and spitting out it's own text. That text only contains meaning to the extent that the human user chooses to imbue it with meaning. It means jack shit to the mindless text generation machine. And then that user will ask ChatGPT "hey I told you not to do X but you did it what's the deal" and ChatGPT will give some explanation but the explanation is FAKE!!!! ChatGPT can't know why it spit out text that broke the rule you gave it because ChatGPT DOESN'T KNOW ANYTHING! ARGGG!!!
This is what I mean by the illusion being too powerful. People who swear up and down that they know they aren't interacting with a conscious being will then ask the thing they just said they know isn't a conscious being to explain why it "wrote" whatever it just wrote. Which is a really stupid thing to ask if you've internalized that you aren't interacting with a conscious being. Whatever. Fuck this technology. I'd feel exactly the same way if it used zero electricity.
I'm an AI doomer so I appreciate a lot of what you're writing, I just don't think it helps to downplay how novel and sophisticated AI output can be, or that some people might have tastes that actually *favor* consuming AI art.
Thanks for the feedback; I take your point regarding AI art. I'm not saying other people shouldn't like it, to each their own. I do believe that *enough* people don't want it that those who create AI art (or AI anything) face an ethical disclosure dillemma: they can be honest and substantially (maybe severely) narrow their possible audience, or they can conceal that it was made by AI (or "in large part" by AI, or whatever is appropriate) and trick people who sincerely aren't interested in it. I have pretty strong feelings about this, obviously, but I know that my feelings about art and ethics are subjective to me, and if it sounded like I was presenting them as objective, that's on me.
As for the novelty and sophistication of AI output, I won't deny that it is mind-blowing what this technology can do. Not trying to downplay how amazing it is. But I confess that it's hard for me to really appreciate given the constant barrage of speculative, poorly justified, and sometimes flat-out false claims about AI being delivered to the lay public, mostly via tech CEOs and the journalists who cover them uncritically. Altman is the worst, but Pichai's pretty bad and Amodei seems like he understands the technology well enough to know better but still insists on doing this routine of "I'm not saying we've conqured the mysteries of the human mind, but many people believe that we'll have them conquered in a year or two and boy I just don't know what to think" (obviously not a real quote, but that's how it sounds to me). One of these willful misrepresentations has to do with AI creativity. A technology whose foundation is mimickry (very sophisticated mimickry, to be sure) is going to be constrained in a way that humans aren't when it comes to novelty. I mean, we're talking about technology that simply cannot work at all without being given *massive* datasets full of things humans have created, so that it can "learn" complex patterns and then use these for generation. It would be a scientific miracle if somehow this architecture gave rise to a process other than the one that AI engineers designed for it.
Now, I don't know how to account for human creativity. Minds are mysterious things. Certainly a lot of human creativity is based on mimickry. Not just "a lot"; the majority. But it can't all be mimickry, otherwise new forms of art would never come into being. There wasn't jazz in the 1800s, there wasn't hip-hop in the 1950s, there wasn't k-pop in the 1980s. But those things exist now. How'd that happen? If you trained a music generation AI only on music recorded prior to, say 1960, and tried to get it to create some Ramones-style punk, it just would not be able to do it, no matter how detailed your prompt. And yet, the Ramones existed. And, if you went back in time to 1960 and told some kid to play high tempo 8th note downstroke power chords on his guitar, he could do it even though that wasn't a thing yet. The conceptual description is enough for a human.
Or, take visual art. If you trained an image generating AI only on art made before 1900, no amount of prompt engineering would get it to produce a Jackson Pollack knock-off. It could spit out Monet knock-offs like there's no tomorrow, but tell it to make an image that looks like someone had spent the day flinging paint off a brush and onto a canvas, and you'd get... I dunno, probably something that looks like a Monet. On the other hand, if you went back to 1900 and told a person to spend all day flinging paint off the brush and onto a giant canvas, they could do it, and after they did it they'd have at least a 2nd order novelty in front of them (the 1st order being the idea to do it in the first place).
I admittedly don't know how to pin this down finely. I don't know how to identify a line between what an AI can and cannot do, given its training. But I know how deep learning works and I know that our minds have to be doing something fundamentally different and - when it comes to creativity - much more sophisticated.
I don't know about you, but I am reminded of this all the time when using AI. There's a common kind of "uncanny valley" sensation when you realize the AI just cannot do what you're trying to get it to do. Kinda like how light switches don't work in a lucid dream. I don't know why it's light switches, but I know the dream is a simulacrum and so of course some aspects will seem more "real" than others. LLMs produce simulacrums of human minds, which often feel utterly convincing. But at some point I always run into the equivalent of that fake light switch, which reminds me that the apparent human-like mental abilities I'm witnessing from the LLM are illusory. An illusion is still an illusion even when it's convincing.
What is that thing where you hear someone talk about something you know a lot about and you get furious about how wrong they are, but then you forget the next episode cuz you don't know the topic? I'm in the first part of that rn. The linguistic mechanics of AI "tells" is actually fascinating and nuanced. There is a reason stuff feels like AI, and it's not just because it's bad writing. K&J are way over their skis on this one
I was also pulling my hair out when they discussed the "tells." For two professional writers who - by their own admission - use AI constantly, to the point where Katie has a cute nickname for it, they really couldn't think of anything other than em-dashes? Like come on you two, you have to be more observant than that.
Haven't finished the episode, but the metaphor overload in the first passage they read from shy girl is very AI-coded.
The thing people don't realize is that AI writes poorly because the 10s of thousands of people that trained it by rating prompts converged on a slightly above average IQ. So they collectively could instill "metaphors good" but couldn't pick up "too many metaphors bad".
Most AI writing style things are limitations of the training and not the technology, so they follow this pattern. AIs think that the sorts of things midwits write are a positive good, rather than an imperfect version of a better thing. If you could get 10,000 MFAs to spend a year each training the same AI, you'd end up with really good writing.
So the biggest tell is "mediocre and proud of it", because AIs get a cookie when they're mediocre and expect the end user to be impressed.
I'm working on a much larger article, but much of it comes down to idiolect and register. These are linguistic fingerprints that are very hard to fake, especially across a large sample. Other tells include over-repetition of context and various violations of Gricean maxims
“4 out of 28 children. . .” Why not just say 1 out of 7? Definitely AI.
The Shy Girl story reminds me of the Goodreads Reylo scandal, where the author got caught giving 1 star reviews to rival authors and invented a story about how her fanfic friend was behind it.
Not sure it’s plagiarism to claim a memory that turns out to have been suggested by someone else’s experience. We don’t own the copyright on our experiences. If she was genuinely unaware she was doing that, it would fit with the “recovered memory” source. If she was aware she was just claiming someone else’s experience as her own, that would be immoral and unethical but I don’t see how that’s an IP violation.
I don’t think she actually underwent psychedelic therapy; it sounds like she made that part up too. She’s being sued for “invasion of privacy, negligence, and infliction of emotional distress.”
There’s also been other reporting that she essentially defamed one of her middle school teachers (whom she accused of sexual assault) by describing him in enough detail that he could be identified despite changing his name. So that could result in a lawsuit too. The thing about her ghostwriter posing as an agent who wheedled more information out of the Jane Doe could certainly implicate her in some kind of fraud scheme. And it wouldn’t surprise me if the publishing house ends up suing her too to exonerate themselves.
Really, I just used “plagiarism” as a placeholder. I’m not sure there’s one term to describe this scenario. It’s pretty shocking.
Yes, she sounds like an amoral attention hound. I’m just not sure lawsuits are the solution because there are too many fuzzy boundaries here. I mean, other memoirists lied but is that fraud? Perhaps the teacher could prove damages, but the classmate? She might only be mad she didn’t get a best seller out of her experience. None of that is to endorse the memoirist, who is exploiting audience prurience and exacerbating false beliefs about memory!
Just a couple points of fact on how LLMs work that are relevant to this episode:
1. LLMs generate text *probabilistically*. There are no deterministic rules involved, and you cannot make the LLM implement a deterministic rule via a prompt. It is incapable of following such a rule in the way that we traditionally expect computers to follow rules (e.g. "turn off screen after 5 minutes of inactivity"; "take all the orange pixels and turn them green"). Saying "don't add anything I didn't write" will likely improve results to the extent that such a directive is probabilistically associated with output that is more similar to the input than it would have otherwise been without that instruction. But this doesn't mean it's actually following your rule; the rule is just changing next-token probabilities (tokens are short series of characters, kinda like words but split up differently). And if you ask an LLM to edit, it has to change *some* words. And it doesn't "know" which ones matter and which ones don't. Our ability to influence LLM output via careful prompting is always subject to this fundamental limitation.
Now, there are some AIs that mix deterministic functions with probabilistic text generation. If you feed a resume to an AI and say "rearrage and reformat this to make it look professional, but don't change any content", this might result in calling a deterministic function that actually checks to make sure the words are all the same. But the LLM itself is simply not capable of performing this task; that's not how LLMs work.
2. On energy use, it takes VASTLY more computational power for an LLM to generate text than it takes for traditional Google to return an ordered list of URLs. Maybe this doesn't matter so much anymore, since Google has decided to give us LLM generated text whether we ask for it or not. But the way search works and the was token generation works are just nothing alike. Briefly:
- When you put a query into a search engine, the search engine pulls results from an existing giant database of URL that have been "crawled". The algorithm for choosing these takes your input and then applies whatever the search engine has developed on the back-end to decide on an ordered list of URLs.
- When you prompt an LLM, it feeds your input through a series of (mostly) large matrix and vector multiplications in order to get a ranked list of probable next tokens, and then one is selected. So now we have the first token (short series of characters) of the LLM's response. Then, your prompt plus that one token it just chose is fed through this process again. Then your prompt plus those two tokens it just chose is fed through again, etc. It keeps doing this until it chooses (probabilistically!) the "stop" token.
Estimates for the relative computational costs using traditional search to an LLM are one to two orders of magnitude (10x to 100x more power needed for the LLM). If you're using a "reasoning" model, toss some more zeroes on there.
I'm not telling anyone they should care about this; it's not near the top of my reasons for disliking AI. But since Jesse brought it up...
Glad to be of help! I guess a shorter answer would be that "search" actually searches websites whereas LLMs use the vast mountain of text they were trained on to generate plausible sounding answers, which often are good but sometimes aren't. You could say they're "searching" through their training data, in a roundabout way. But they aren't searching websites (unless it's one that has a website search function tacked on, but with those the LLM portion will often misrepresent what's on the website)
Most of the professions that are going to be significantly impacted by AI are heavily left leaning.
That's why their opposition to AI is more unhinged than the right.
Whatever "ickiness" Katie and Jessie feel is not going to be shared by most. We can look historically at creative work that is simply obsolete or exist in niche, nostalgia driven markets.
Take animation. Used to be an undertaking of massive creative labor by human beings drawing, coloring, and refining every single frame. Now? 100% generated. No one watching a Pixar or modern animated film feels "icky" because a human didn't create it. We think of the modern operators who do rigging, etc. as creatives.
But painting and drawing frame by frame? Gone. /done
Journalism is going to go this way too. Probably programming. Banging on a keyboard to crank out 1000's of words is just not going to be what a journalist does. And no one used to the technology is going to care or be icked out by it. . . . anymore than they care about their cartoons not being hand painted.
What's interesting is the entire concept of plagiarism is going to change. No one says a scientist plagiarized their published paper because they used calculators to compute the numbers or computers to process the statistical analysis.
That's not where the value comes from. It comes from the data collection, curation, decisions on which statistical models to use, etc. Why should we care if an AI model generated the text if the content is accurate and the findings of interest? They won't.
Your claims about computer animation are absolute bupkis and demonstrate zero understanding of the medium. A LOT of time, work, and artistry goes into designing and rigging models, and that's before you even get to the animation phase. The turnaround time on a CGI film today is about the same as a traditionally animated film in the 1980s, because though frames don't have to be individually drawn or painted anymore, CGI allows for a lot more mid-production meddling that eats up time in its own right. I won't get too far into it, but once your model is finalized and rigged, a big chunk of the work is done, so scrapping a scene doesn't put you back at square one the way it does with 2D. Today, producers, studio heads (especially at big companies like Pixar), and directors obsessively test-screen and tweak and re-test-screen and re-tweak films, effectively offsetting the time saved by using CGI instead of 2D. That's one of the main reasons why CGI is preferred to 2D in mainstream film; re-animating a 2D scene is much costlier and requires that studios have faith in directors and that directors fully commit to a unified vision. This is part of why The Thief and the Cobbler was never properly finished; the director kept changing his mind. Spider-Man: Across the Spider-Verse is an example of a CGI film where the amount of last-second revisions inflated the budget and caused ethical concerns for animators.
Also, frame-by-frame animation absolutely still happens, it's just not being made by major studios like Disney or Dreamworks. There's usually at least one 2D film nominated for Best Animated Picture every year, and there's still a huge market for Japanese 2D animation.
TL;DR: CGI films are very much a labor of love made by human beings, and it's an insult to the profession that you would insinuate that they aren't. Watching a CGI film is in no way comparable to watching 90 minutes of AI slop.
The artists that hand draw frames don't exist anymore for the most part.
Tools came about that largely eliminated that work. . . . and in the process created entirely new types of work. Creatives adapted and began using the new tools to....create.
And now, you find it laughable and insulting to suggest someone using these tools are _not_ artists and are _not_ creating. Which I did not remotely insinuate; I explicitly said we consider the riggers, designers etc as artists today. Their labor is still there. Their vision is still there. Using new tools in new ways to create even more amazing art.
Now. We have AI. It can generate text, articles, books, and getting closer and closer to creating original research and science publications indistinguishable from those written by hand. And creatives in writing, journalism, science, are getting triggered by people using these tools. Saying if you do you're not a real journalist or _actually_ creating.
Flash forward a few decades when it'll seem anachronistic to sit at a keyboard and bang out 1000's of words of text. Someone on some forum will get triggered by even the slightest suggestion that if you use these tools you're not _actually_ creating. The written word will become like clay that can be shaped and formed en masse into new works.
How will they be using them? What will the new content look like?
Who the fuck knows? That's like asking a farmer in 1850 what farming in 2000 would look like. Or asking the creator of Steamboat Willie in the '20s how Into the Spyderverse or Avatar would be made or what they'd look like.
They literally could not even begin to describe it. They wouldn't have the words.
CGI takes the same amount of effort as 2D animation, it's just a different process. It was not invented to "reduce" work the way AI was, it was invented to achieve things practical effects couldn't, and some people at Pixar decided to use the technique to make a whole movie. Asking a robot to chew up a bunch of other peoples' artwork and shit out an amalgamation of it requires no artistic skill. Pro-AI-"art" guys literally use the lack of skill it requires as a talking point, accusing artists who dislike it of "ableism." This is not a situation where the same level of expertise is being applied but with different skillsets, as with 2D and CGI.
If you couldn't be assed to make or write something yourself, why should I be assed to look at it or read it? And I'm sure as shit not going to spend money on it. Art is a tool of human expression, that is its main purpose. It doesn't matter how "indistinguishable" it is, for the same reason you wouldn't accept an "indistinguishable" robot somebody made of your dead grandmother. Having a robot make art for you is like programming a robot to feel emotions for you. A lot of use cases for AI, like organization and copy editing, make sense. This one does not.
THIS!!!!! I don't care how well an AI imitates writing or painting or animation or music. I'm not interested. It actually matters to me that the products of human creation which I choose to spend my time experiencing be made by humans.
So, the people making the AI stuff can either deal with the fact that a lot of us will reject it out of hand, or they can lie. Like they're doing right now everything they fail to disclose their use of AI.
I think this is fair, but the analogy lacks something important: no one claimed that photography could be a substitute for a painting. No one predicted that photograph technology was a year or two away from becoming just as good a painting. Photography was a genuinely new medium.
I don't think AI art is analagous. I mean, the whole way the technology works is by feeding the AI image generator a bazillion images and running it through an algorithm that "learns" to imitate what it was fed.
I'm sure they did, but it's a bit of a false equivalence, because photography is still human expression at its core, just as all art is. Crucially, photos inherently have provenance, because of their relationship to time and place; in a way, photography is equal parts art and journalism, which is why Pulitzer prices are awarded to photographers. When a person takes a picture, they're expressing something and immortalizing a singular moment in time. Consider Falling Man, one of the most famous photos of the 21st century. The photo is important not only because it's of a real thing that happened, but also because it succinctly captures of profundity of a tragedy in a deeply resonant, strikingly intimate way.
Could somebody have theoretically made an identical piece of art by telling AI to generate an image of a man falling from a skyscraper? Yes, but it wouldn't have had the same impact. The artistry is in the photo's provenance, in the human emotions it captured and the emotions that motivated the cameraman to point the camera at that particular man in that particular moment. AI lacks the human motivations that make a painting OR a photo a piece of art. The "decisions" of AI are not motivated by anything beyond fulfilling the prompt. There is no provenance in a black box, it just excretes an image into existence based on what effectively amounts to word association.
To be clear, I'm not against AI integration into visual art wholesale. It can be a useful tool for the ideation phase of a drawing or painting, similar to how someone would use Pinterest. AI's ability to collate and assemble images into one has utility, I don't deny that. I am specifically against the characterization (or god forbid, normalization) of entirely AI-generated images as art. I would also like to note as an aside that in the eyes of the law, the folks writing prompts for AI images are not legally the creators. The LLMs themselves are the creators, which is why raw AI images, just like photos taken by monkeys, are public domain; intellectual property laws require a human creator.
Having an AI that does a great job of generating articles and books is only half the equation. You also need people who will choose to spend their time reading it. For me, language is a form of human communication. With some narrow exceptions, I simply do not want to read text generated by a machine, regardless of how well it resembles what a human might have written.
A good part of the value in a scientific paper (that's worth a shit) is created in the process of writing and editing it. You find out if the ideas work and the evidence supports them by trying to explain what you did in a persuasive way. Writing is the heart of it, not the periphery. The more senior the scientist, the more their life is about writing.
Many bad scientific papers will indeed use AI but good ones don't, and won't for these reasons. Apart from anything else there's a University regulation against publishing AI generated text, even though they do supply AI. Code is different, that can be AI generated.
To be clear an LLM can't write a high quality paper, in fact it's more trouble than it's worth because of the error checking. It's useful for literature reviews, but only for finding and summarising references. You still have to read them, since it's very often wrong on topics that aren't settled, which means all topics a scientist is writing about.
"You find out if the ideas work and the evidence supports them by trying to explain what you did in a persuasive way. Writing is the heart of it, not the periphery."
They’ll get there. Already pretty close. Papers often lack details, skip steps, forget to mention data. It’s of of the things that makes them challenging to reproduce.
You use it then. You'll be restricted to shit journals. And you'd better not use it for writing reviews during your brief career or I'm coming for you.
What? No I'm planning on ensuring anyone who conducts peer review of novel scientific work without reading or understanding it, instead putting it through an LLM - whether to save time or for any other reason - is no longer asked to peer review other people's work, because they aren't meeting the professional standards required of them. They can turn down reviews if they don't have time, that is what people do.
It has nothing to do with fucking cancel culture that's truly an inane remark.
All of these vatic one-line-per-sentence AI evangelist accounts swear that the horizon is just around the corner* and soon every single creative industry will be subsumed. Like, okay, when? It hasn't gotten any better for years.
* a classic LLM mixed metaphor I've seen them use more than once
Your examples are all of machines enhancing something that originates from a human mind. Going from using a keyboard to using an LLM is not analagous to going from, say, a pencil to a keyboard. Nor would it be analagous to going from a keyboard to a voice-to-text translator. Because the pencil and the keyboard and the voice-to-text are all being used to communicate language chosen by a human being. The LLM replacing (at least to an extent) the human, which fundamentally alters what it means to communicate.
Likewise for the cartoons. Going from hand-drawn cells to CGI is not akin to going from CGI to generating a cartoon from a prompt, because the latter removes a huge amount of human intention.
And hey, maybe you're right and the public will embrace this. I kinda doubt it but we'll see.
As for using calculators rather than doing math by hand, it's again the same thing. Using data analysis software to fit a regression model doesn't change anything relative to using a pencil and paper, except that it doesn't take 6 months. Feeding data and a research question to an AI and saying "do a statistical analysis" is a fundamentally different because now a human being isn't choosing the method to be applied.
The Hex code situation absolutely baffles me, because every Windows computer has a Hex code picker as part of the operating system! Taylor didn't have to Google ANYTHING, because you can just hit the PrtScn key or Windows+Shift+S to see a drop-down menu offering a dropper tool that translates any color you hover over with your mouse into either Hex or RGB. Your computer doesn't even have to have a freaking internet connection for you to be able to do this. I imagine iOS has a similar function. For a writer whose beat is technology, Taylor Lorenz seems to have pretty crummy tech literacy.
The issue with AI is that it’s too easy to use it improperly. It can be very helpful as an editing tool if you retain control e.g. you ask it to point out specific errors etc. but you don’t use the text it spits out, you keep your original document open and edit manually. But if you get lazy or sloppy (I’m no stranger to both) you skip the safeguarding steps.
I'm happy Jesse and Katie mentioned that the environmental impact of AI is PEANUTS compared to the meat, egg and dairy industries. As a vegan this argument has been driving me up a wall for 2+ years. I'm a French as a foreign language teacher and regularly use AI to generate the annoying grammar exercises I hate writing, things like "conjugate all the verbs in the future tense" where it's just ten sentences like "Tomorrow, my sister and I will go to the movie." I hate this and will gleefully keep asking AI to make these exercises for me and go to sleep knowing my veganism makes me way more environmentally sound than the thousands of meat eaters being performatively angry at AI on social media.
I think this is a fair point (Go vegan!), but one should compare apples to apples. It's not merely a question of whether it's better to avoid AI or avoid dairy, but "Should I eat meat for protein, or something similar made of beans? Are there benefits to the less environmentally friendly option that justify the difference in impact?", "Should I do a regular search on DDG, or ask ChatGPT? Blah blah impact" etc
Your AI views are better than they used to be, but are rife with cognitive dissonance.
You acknowledge that all the negatives that people see are real: AI writing is bad (even if people cannot necessarily tell why it is bad), data centers suck, Chatbots lie and give inaccurate information with confidence, and chatbots are now included in all kinds of apps for no reason except to frustrate people and suck up power. I am glad you acknowledge these things are true and suck. That's a start.
However, when you try to imagine why some people hate AI so much, you still focus on strawmen and psychoanalysis. You repeatedly insist that dislike of AI is actually just anxiety about the future and specifically about the job market. Certainly, that is not a concern for me. I am simply concerned that AI, as it exists, sucks at things that are not backend tech work. Not everything needs to be about "anxiety." AI is now very hard to miss in daily life, and most people do not like the role it plays in their own lives.
You defend Megan McArdle for relying on the tech but are defensive about its use in your own work. If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech. I hope this is the start of an enduring social stigma.
If you are a writer and use AI to edit and check sources, you must know you could do that with grammar check and Google before these chatbots came along. If you go further and believe that your use of AI is beneficial to your work in some big way, you should be able to point to some benefit to readers.
They acted as if the problem with people disliking AI was some general Lefty thought disease akin to wokeness, not that it was trained on the corpus of the work of people it seeks to replace-without compensating them or seeking their permission! Anthropic lost a huge suit for training on Stolen work. Meta also knowingly trained on stolen work.
The New York times is suing openai for reproducing New York times content virtually word for word. The lawsuit has a hundred examples. You can look them up. There is no question. It is a theft and plagiarism machine and most writers don't like the idea of their work being taken without their permission and compensation, but they especially don't like the idea of it being used to train their replacement! That is not a lefty woke woo woo sentiment.
"If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech. I hope this is the start of an enduring social stigma."
A-fucking-men.
I'll add that the negatives are not limited to it not just not being good enough at X. I say this because the constant refrain is "they're getting better just wait soon you won't be able to tell the difference". But being able to tell the difference is not my issue with them. Making an AI that is convincing enough to trick me just means that now I'm easier to lie to. But that doesn't excuse the lie. It doesn't make the high tech lying machine ethical to use.
If the AI enthusiasts are right, then it seems like we should all be able to agree that AI use can be fully disclosed and the technology will still be welcomed by the general public with open arms. AI novels will sell, AI music will be celebrated, AI art will be hung in museums.
On the other hand, if disclosing AI generated content results in people avoiding that content, what then? We know what then. Deception and lies, that's what.
"If the AI enthusiasts are right, then it seems like we should all be able to agree that AI use can be fully disclosed and the technology will still be welcomed by the general public with open arms. AI novels will sell, AI music will be celebrated, AI art will be hung in museums."
Would you consider someone like me who finds AI very useful in certain aspects of my job an AI enthusiast? Or are you reserving that for people who specifically want art/text that's fully AI generated?
Because I think I'm in the majority recognizing that it's super useful for some things, but fully AI-generated art is probably not that interesting for most people. I don't know why you'd think AI being useful for certain things (the enthusiast view) implies AI art dominates.
I was painting with a really broad brush there. Yes, I meant people who are enthusiastic about AI art. And I agree, AI has plenty of useful applications, even if I think that on net it's doing more harm than good.
Perhaps an ungenerous take, but I have a feeling Katie and Jesse use AI in their writing considerably more than they’ve admitted to.
Made even more obvious by continuously affirming that “I only have the free version”
Also, am I crazy or did Katie in a previous episode straight up admit to running Drink Your Way Sober through ChatGPT? If I’m not mistaken that’s similar to what “Mia Ballard” claimed happened with Shy Girl.
This comment is very insecure and in the process gets the usefulness of AI fundamentally wrong.
AI being bad at writing doesn't mean that AI is bad at critiquing writing; it's not a person, its skillset is unevenly distributed.
Chatbots give incorrect information with confidence if you're bad at prompting them and don't double-check important facts, but they cite their sources now and will stop lying if you just ask "is that really true?". The advantage they have over Google is that they can take a plain-English query and render it into accurate results, without relying on a keyword guessing game. They have approximate knowledge of the state of every field of human expertise, and that's enough to point a search in the right direction at an unprecedented rate compared to raw Google searches.
Chatbots don't suck up that much power, it's just not true. You can easily use more power playing videogames or driving a car for equivalent amounts of time, and these are activities people do for way longer per session. And the water issue is wildly false, even less real than the power one.
The "psychoanalysis" you're complaining about is a strained attempt to be charitable to people like you who blunder into discussions either wildly wrong about AI or intentionally dishonest about it. It's kinder to empathize with why you're so vitriolic than to call you a bunch of hysterical liars, and J&K are kinder than I am.
They do use a lot of power a) when you're training them and b) when they are doing "reasoning" which is long chains of inference operations. It's not doing nothing for those 5 minutes of "thinking" time per query. That's computation in a datacenter.
They cite sources but a) quite a lot of the time, still, the sources don't exist and b) very, very often the sources don't actually say what the chatbot says they say, which makes sense given the chatbot hasn't read them, or indeed ever read anything in a meaningful sense.
"prompting" IS a "keyword guessing game" or "key phrase guessing game". Except it's worse because it will tend to confirm what's in the prompt. "discuss X" gives less biased information.
Chatbots are fine, especially with a long list of instructions never to be sycophantic etc. When you start researching something it can be hard to know what the most central papers are on a topic, and it can help find those quickly. It summarises them I'd say with about 80% accuracy. If you are after important facts, like "how long before this mouse starts to show degeneration" that will often be wrong. You need to actually find out.
Once you've used it to get a few papers the method of finding sources by citation chain works better.
It can be useful for suggesting hypotheses and spotting gaps in the literature. However: it makes obvious suggestions, and I'm required to declare that I've used it in this way. So then it looks like my ideas came from a chatbot. Fuck that, it's not worth as nearly much as a declaration "AI wasn't used" or "AI was used during early stage literature survey".
I asked it to assess a recent proposal I wrote, which I didn't use it for, except for limited early literature survey as I describe above, and it said
"Conceptual clarity Very high
Appropriate detail Very well balanced
Experimental logic Strong
Writing efficiency High
Overall competitiveness Very high
the proposal is well-constructed, appropriately detailed, and highly competitive."
That's because I wrote it.
It also said
"The text is most consistent with human-authored grant summary writing, with no reliable indicators of full AI generation. AI-assisted editing remains possible but cannot be substantiated from linguistic evidence."
So why would I ever risk adding my work to the pile of self-similar AI shit reviewers are wading through? I wouldn't. I'd try to write something that stands out as having my voice as well as being well argued. Probably the ability to do that will become more valuable as time goes on, in journalism as well as science. It's certainly what I want to read.
"[Chatbots] will stop lying if you just ask "is that really true?"
This is insane, right? If the chatbot could have told the truth, why'd it lie the first time? Why must one ask "is that really true?" And why does asking this cause it switch from lying to telling the truth?
Chatbots that haven't been trained spit out random-ish text based on pattern prediction.
The "is it true" data represents a smaller proportion of the pie. Literally separate training sessions. So you have to tap into it by prompting the AI to think "oh, this is one of those scenarios where I get the cookie for being honest, not for just going through the motions".
The "cookie for being honest" analogy is pretty different from how chatbots really work. Importantly, chatbots have no concept of truth or honesty. They're just "predicting" the next piece of text, over and over again. This process often leads to output we consider "true", but that's not because the AI went into tell-the-truth mode, it's because the AI is mimicking the text it was trained on (for the most part - the post training reinforcment influences it too), and there's a good correlation between probable text and "true" text.
I don't know what you mean by separate training sessions. There's a main training session and there's post-training reinforcement. The main training session is basically an extremely intensive, extremely time-and-energy-consuming game where the chatbot plays "guess the next word" over and over again (really next "token" rather than "word", but the difference isn't important). So the AI takes some chunk of text from its training data with a defined start and end, and then it guesses what comes next. And then it compares its guess to what actually came next, and uses this to make updates to its "weights" (aka "parameters", or just "numbers in matricies") via a very clever algorithm. And current chatbots have over 1 trillion of these "weights". I like to picture this as a giant control board with 1+ trillion little knobs that get continuously turned by the training algorithm. And as the chatbot goes through this "guess the next word" process over and over again with more and more training data, it slowly turns those knobs to positions that make it better at guessing the next word. After it's been trained, it can take input and once again play "guess the next word" over and over, except this time its guesses are printed on a screen as output. That's the basic gist of how chatbots talk to us.
Importantly, "truth" and "honesty" are just irrelevant to this process. They don't factor in, except as a kind of second-order happenstance. The chatbot has no idea whether what it's writing is true. It's doing the exact same thing when it "hallucinates" as it does when it states correct facts: it is guessing the next word.
So the reason why chatbots tend to give better answers when prompted with "is that really true?" is because the particular string of text that comprises "is that really true?" is associated with there being something wrong with its last string of output text. And not wrong in just any way, wrong in some complex but systematic way that is exploitable using those 1+ trillion little knobs that the chatbot uses to guess the next word. The AI isn't internally identifying anything as "true" or "false" - it's responding to the really big hint you gave it when you prompted it with "is that really true?" And that really big hint certainly doesn't guarantee that the next response will be correct, it just (usually) increases the probability.
As an important side-note, this method has a major deficiency: in order to give the chatbot a big hint that helps direct it toward something true, you have to already know the truth. It certainly doesn't.
You're just wrong, truthfulness was one of the main targets for Reinforcement Learning through Human Feedback training, which is the main vector through which LLMs were trained to be useful products. Tens of thousands of employees were tasked with rating AI responses on their truthfulness and if the AI did well it was rewarded in the feedback loop. If it did poorly it was penalized. But there were a lot of qualities that the LLM was trained to have and not every project was about truthfulness, so if you nudge the LLM to be truthful you pull its understanding of the context towards that portion of its training data.
All the stuff you described with trillions of knobs describes the first layer of training, before the RLHF that turned it from a token predictor into a useful chatbot.
Yes, I did skim over reinforcement learning in my summary of how LLMs work. I agree, fine tuning the trained LLM for truthfulness will make truthful responses more likely, at least in situations similar to those for which human feedback was given. I would trust such an LLM more than one that had not gone through reinforcement learning focused on truthfulness, at least if I believed I was prompting it about something that is within the domain of its training / RLHF. But I'd still keep in mind that the LLM probably doesn't have any conception of "truth" or "honesty".
I do not agree with this:
"if you nudge the LLM to be truthful you pull its understanding of the context towards that portion of its training data."
I see what you're getting at, but you're assigning human properties to a highly complex, mostly uninterpretable next-token prediction algorithm. It doesn't identify truth vs. falsehood and then use context to decide which to deliver. It just does the one thing it always does: it generates probability distributions for possible next tokens and then selects from them. I suppose it's *possible* that there are some under-the-hood matrix calculations going on that are doing something that could be reasonably interpreted as "deciding to be honest". But I'll believe that when someone presents credible research that turns such a claim into something other than pure speculation. In the meantime, we've all seen example after example after example of an LLM demonstrating behavior that is entirely consistent with mindless next-token prediction and hard to account for otherwise. I will grant that the technology is improving to the point where they make major errors less often. But I'm guessing this *isn't* because something magical has happened.
My apologies if you're not claiming anything stronger than "RLHF increases the probability of truthful output by some amount across some set of scenarios".
"If all the negatives about AI are true, it is totally fair for people to disdain those who use it. Writers should either make sure there is no trace of AI in their work and simply lie about it, or accept that people will despise them for their choice to rely on shitty tech."
This just doesn't follow or make sense in the context of the situation with McArdle. It's easy to get certain usefulness out of AI, so no, it doesn't make sense to disdain anyone who uses it.
There's a huge gap between McArdle finding AI useful in some areas of her job, and people "relying" on shitty tech. Sometimes it really is just using available tools to do better work. There's no reason for the generalized anti-AI stigma you're hoping for except in cases of plagiarism, AI-generated text, etc. where there's *already* a stigma (and there should be).
My ex is an internist- and he says that AI is great at diagnosing. He says the only time it screws up is if he has forgotten to put in all the available information. He’s 58, and is a phenomenal diagnostician.
Which is why it's terrible. It can't collect information accurately. Which is what makes a diagnostician phenomenal.
If you can't tell whether or not a writer uses AI in some way, how can it be a negative? I think the cognitive dissonance is on your side for not acknowledging that there are benefits too (and they don't necessarily have to be beneficial for the reader either.
Yes, very well said.
I just see a continuing misunderstanding of what "AI" is doing, and what can be usefully accomplished by such a tool.
It's fine at "opinions" even if there's no mind there to think it. The output texts can reasonably simulate an opinion on something. Similarly it's fine at critique, and some pattern recognition if you don't require accuracy.
What an LLM will never be good at is perfect accuracy. Hallucinations will never go away, so unless you're comfortable with "there's an X% chance that this article exists and a Y% chance that is says Z" you will always have to independently verify its outputs. If you don't want it to make mystery edits to your texts you need to ask it to highlight your mistakes, but you then need to make the changes yourself in a separate program (or else be comfortable with random error insertion).
Jesse seems to grasp this, but Katie seems not to.
Maybe this isn’t necessary to add, but the ways in which an LLM will be inaccurate is significantly different from the ways in which a person will be inaccurate, which is why we use anthropomorphic terms like hallucination rather than just saying error. A person makes logical mistakes, or errors rooted in emotions or ideological bias. A statistical token associating computer system makes statistical token association errors.
I think the same way on this.
I hope Lindy divorces her gross husband and girlfriend, goes offline far from the northwest for 18 months and comes back fucking jacked as a new health influencer.
A big magazine profile and photo spread from a old-school warehouse gym (barbells and chalk) to announce the return needs a clever headline...
If she files I guarantee that getting Aham and Roya out of that house is going to be an uphill battle. Coming soon: Lindy the Landlord!
They will try to get her to sign the house over to them due to racism
The sad thing is she might fall for it
Roya is basically her mother now: pays her bills, waters the plants, tucks her in at night.
...then goes and fucks her husband while Lindy sleeps alone in the spare room. What a sick situation.
If a person wanted to get a little psychoanalytical, it actually makes sense IMO that Lindy, who is clearly deeply troubled and in need of reparenting, has basically created a situation in which she has a new mommy and a new daddy.
she’s gotta sell the house and evict their grifter asses
Thankfully it's still owned by her mother I believe. So Aham won't get a cut.
"From Shrill to Swole"
Lindy drops the dead weight (and get fit too)
I love you guys, and I agree that the political left gives bad anti-ai arguments. However, I think you are way too glib and dismissive of peoples’ ai anxieties more generally. I’d suggest doing a deeper dive and developing a more nuanced view.
Jesse and Katie need more nuance when it comes to AI checkers, too. I am a college professor. I refused to use the checkers for a long time because of worries about false positive rates. I was once falsely accused of plagiarism in eighth grade and would never want to do that to someone. But Pangram really is quite good.
I now run every piece of student writing through Pangram before grading it. If I get the green light, then I can read with less vigilance and just concentrate on being an educator rather than a policeman. If AI usage is flagged, then I look for other evidence, which I can usually find either in the submitted text itself or through comparison with an in-class writing exercise I have them do early in the semester.
Students rarely use AI in a thoughtful way, in my experience. Those who use it merely copy and paste, sometimes running the initial result through a humanizer before submitting it to me.
Educators still need to assign writing because we learn to think more clearly and rigorously when we write. Writing is thinking! And we can’t do all the writing during class, especially at the university level where we typically get our students for three scant hours per week and have oodles of content to teach. Oral exams are also not the answer; not only do they not substitute for the writing process, they can’t be scaled to the number of students most of us teach.
I loathe playing cop. But if educators don’t enforce academic integrity, students don’t learn, and their degrees will soon be regarded as worthless scraps of parchment.
Pangram has good privacy policies. This is essential to me because I value my students’ privacy. It does not use the submitted data for anything except the check that I have requested. It claims the lowest false positive rate in its field, and my own testing confirms that, as did Katie‘s brief experiment with it.
Yet our hosts dismiss all of the checkers as garbage. C’mon, guys! I need you to be perverts for nuance here!
The number of students is the problem. The student body should be a third what it is. And Oral/written exams are fine. If people can't pass them, they shouldn't be in college.
Jesse and (especially) Katie are very susceptible to “I read one article/post on the subject and I’m an expert now” type thinking.
I don't think leftist critiques of data centers draw enough attention to the way they inflate the cost of living for people with the misfortune of having one built in their community. They drive up utility bills, often in rural areas where access to water and electricity is already tenuous. Politicians often give out tax breaks to entice companies to build data centers in their towns because they erroneously think they "create jobs"; They don't. The centers are built by short-term contractors, most of whom are basically nomads from out of town who hop from site to site. Once the data centers are built, they only require a few dozen people to run. So you're increasing the cost of living for literally everyone in a community for the sake of creating a puny number of jobs.
There are emerging claims that the light and noise pollution caused by data centers is also bad for communities, but I regard those claims with more skepticism. The strain they create on utilities and the resultant cost passed on to residents is a much more substantial problem.
I want to be clear that I know data centers are used for things other than AI, notably social media and cryptocurrency. That's one of a couple reasons I've cut way back on social media. I've always hated cryptocurrency because I recognize it as a bigger fool's scam, so I have no issue abstaining from that. That said, social media platforms seemed to be running just fine before this big uptick in new data center construction happened. It's obvious that these things aren't being built because they fulfill some kind of actual demand, they're being built because of this big dumb AI arms race going on between tech companies, and because "new data center" sounds sexy to shareholders. And now companies are adding unwanted AI bloatware to everything to try and justify this enormous waste. My fucking refrigerator does not need AI.
Have you seen videos of the communities near these data centers. I can't speak for light pollution, but the noise is RIDICULOUS. And most of the videos I've seen aren't exactly next to the facility, it travels pretty far. It would be like living next to a neverending freight train.
Last week, I saw a news report about a suspected cancer cluster near an Amazon data center. Now, this could be coincidence, so I’m awaiting more evidence. But I’m also adding physical health effects to my long list of open questions about how data centers impact quality of life.
The freight train comparison seems about right!
Agreed. They both seem way too certain about AI and dismiss criticisms in general. But when they talk about how they use AI specifically themselves they see all the limitations.
Agree, it would be especially great to have Brad palumbo as a guest on during that episode. He's done a pretty good job of debunking all the anti-environmental nonsense.
I can't find his link right now but here's another one
https://youtu.be/H_c6MWk7PQc?si=7Bwo5DT4QtaWotJu
I want Brad to come on to talk about the MAGA civil war, but I may be the only person here paying attention to that lol.
Are you talking about Tucker Carlson and Candice Owens?
That’s one part. Honestly, the disdain for Tucker has reached derangement levels. The guy blinks and people go crazy. He’s basically saying the same things he always has, it’s just that MAGA has moved in a different direction. Candace, on the other hand, is both insane and kinda evil. She spouts a lot of antisemitic garbage, but even worse is the harassment campaign she’s been running against Erika Kirk for months, posting at least a dozen full videos dissecting Erika’s life. Candace believes that Erika and Israel were behind Charlie’s death, and that Tyler Robinson was framed, and she’s managed to convince her legion of braindead fans of it. Charlie Kirk himself has unfortunately been turned into a kind of prop, with everyone who ever met him claiming that he supported the same things that they do.
Megyn Kelly landed in hot water for a couple of reasons: she’s friends with both Erika and Candace, and so refuses to weigh in on any of this (why Erika still wants to be friends with her is beyond me). She also started criticizing Israel late last year, and didn’t support Trump’s actions in Venezuela or Iran. This has led to people like Dave Rubin, Bari, and Moynihan boycotting her show. Her biggest critic is probably Ben Shapiro, who was good friends with Megyn, then gave a speech criticizing her without reaching out to her first—MK did not take this well. Piers Morgan then got pulled into this when he mentioned on his show that Ben had stopped responding to his text messages, and Ben responded on his own show by comparing Piers to Jerry Springer—Piers did not take this well. Mark Levin also got pissed at Megyn for questioning Israel and started going after her in kind of an incel-ish way (started calling her “Grandma Groyper,” etc.) Megyn ignored him for a while, then gave Levin a nickname of his own: Micro Penis Mark.
Nick Fuentes is basically a footnote at this point. Even he’s said Candace has gone too far attacking Erika (he also called Tucker an antisemite). Glenn Greenwald is crashing out, turning into a sad caricature of the self-hating Jew. Dave Smith is Dave Smith, and that drives a lot of people crazy for some reason. The people who support the war in Iran, like Ben and Batya, spend a lot of time denouncing “the podcasters” and calling them irrelevant. Through all of this, Matt Walsh has emerged as the voice of reason, calmly debunking the conspiracy theories about Erika Kirk and speaking out against the Iran War. Brad is a much smaller figure, but he’s managed to stay calm as well.
Ranked from most to least important, I’d say the biggest issues dividing the right are Israel/antisemitism — Candace and Erika — Iran. But as you can see, it’s become a giant clusterfuck. Turns out the trans furries won—Charlie Kirk was killed and his death fractured the right.
I think they should have you on as a guest. That was a very good summation
Thanks! 😂 Hopefully Katie sees it, she loves good drama.
I’m committing the sin here of commenting without listening to the episode but it’s clear from Jesse’s twitter that he takes AI existential risk — that is, the scenario where AI development is automated and they rapidly become better at everything than humans, economically displace us and then send us the way of the dodo — entirely seriously. So like when someone says they’re anxious or distrustful of seeing AI pushed everywhere, surely you should be treating their anxieties as ‘directionally correct’ rather than dumb leftist misinformation, even if you think they’re worried for the wrong reasons.
My impression is that they haven’t really internalized how devastating the AI steamroller has already been for a lot of people.
It will all be fine.
I enjoyed the aside about the word tulle at the beginning of this episode because it brought to mind this chart that I saw online a few years ago, Words Known Better by Males Than Females and Vice Versa: https://observablehq.com/@yurivish/words
Apparently about 80% of women know the word tulle but only about 25% of men do!
I just asked my husband if he knows what tulle is. He looked at me blankly and said he had no idea.
Your comment made me do something I rarely do anymore: wish I still had a husband!
Ask a male friend and see what he says!
It’s netting! Not plastic! It’s nylon or polyester! Clearly they are not around ballet dancers or four-year-old girls who have five tulle skirts.
Nylon and polyester are both plastics.
I stand corrected!
Still annoyed though because calling tulle "plastic" makes it sound stiff and solid when it's fluffy and light and moveable. Sigh.
This is an amazing link thank you so much. I really loved the “tulle” moment on this episode.
I love this.
There were three Woman Words that I kind of knew, and I definitely knew the rest.
By contrast, the first 12 Man Words I did not know at all. In total there were only four Man Words that I knew.
😀
I'm pretty sure that a jacquard is a french retard who is also a jackass.
I'm honestly surprised that the anti-LLM movement has gotten blue coded. I hate LLMs, and it's not because of data centers and the environment. It's because I think having human-impersonation machines all over the place is a bad idea, generally. I know the technology is cool, and I know there are positive use cases, but I think they're outweight by the negative use cases. I mean, we're talking about MACHINES THAT ARE GOOD AT PRETENDING TO BE HUMAN. It can't take a whole lot of imagination to come up with reasons to not want these things around.
I'm not gonna list them here. I just want to point out that, if anything, an aversion to MACHINES THAT IMPERSONATE PEOPLE seems like it should be red-coded. Or at least religious-coded. Y'know, souls and all that. I'm an atheist, and I think I'm appreciating for the first time in my life what it's like to see something as "satanic". Human-impersonation machines are satanic. I mean, I don't believe in Satan. But if I did, I'd bet foisting LLMs upon humanity is just the sort of thing he'd get up to. Maybe "misanthropic" is a better word.
I know everyone likes to say they understand that the AI isn't really a mind, but the illusion that one is communicating with a conscious being is just so damn powerful. But it's fake fake fakity fake and this matters. For example, Jesse and Katie and most other people express frustration that they'll give ChatGPT some instruction and then ChatGPT will say it understands but then it doesn't follow the instruction. No one should be the slightest bit surprised by this. There's no mind! There's no understanding! It is fundamentally incapable of understanding your instruction because "understanding" plays no role. It's just taking in your text and doing a bunch of fancy math and spitting out it's own text. That text only contains meaning to the extent that the human user chooses to imbue it with meaning. It means jack shit to the mindless text generation machine. And then that user will ask ChatGPT "hey I told you not to do X but you did it what's the deal" and ChatGPT will give some explanation but the explanation is FAKE!!!! ChatGPT can't know why it spit out text that broke the rule you gave it because ChatGPT DOESN'T KNOW ANYTHING! ARGGG!!!
This is what I mean by the illusion being too powerful. People who swear up and down that they know they aren't interacting with a conscious being will then ask the thing they just said they know isn't a conscious being to explain why it "wrote" whatever it just wrote. Which is a really stupid thing to ask if you've internalized that you aren't interacting with a conscious being. Whatever. Fuck this technology. I'd feel exactly the same way if it used zero electricity.
I'm an AI doomer so I appreciate a lot of what you're writing, I just don't think it helps to downplay how novel and sophisticated AI output can be, or that some people might have tastes that actually *favor* consuming AI art.
Thanks for the feedback; I take your point regarding AI art. I'm not saying other people shouldn't like it, to each their own. I do believe that *enough* people don't want it that those who create AI art (or AI anything) face an ethical disclosure dillemma: they can be honest and substantially (maybe severely) narrow their possible audience, or they can conceal that it was made by AI (or "in large part" by AI, or whatever is appropriate) and trick people who sincerely aren't interested in it. I have pretty strong feelings about this, obviously, but I know that my feelings about art and ethics are subjective to me, and if it sounded like I was presenting them as objective, that's on me.
As for the novelty and sophistication of AI output, I won't deny that it is mind-blowing what this technology can do. Not trying to downplay how amazing it is. But I confess that it's hard for me to really appreciate given the constant barrage of speculative, poorly justified, and sometimes flat-out false claims about AI being delivered to the lay public, mostly via tech CEOs and the journalists who cover them uncritically. Altman is the worst, but Pichai's pretty bad and Amodei seems like he understands the technology well enough to know better but still insists on doing this routine of "I'm not saying we've conqured the mysteries of the human mind, but many people believe that we'll have them conquered in a year or two and boy I just don't know what to think" (obviously not a real quote, but that's how it sounds to me). One of these willful misrepresentations has to do with AI creativity. A technology whose foundation is mimickry (very sophisticated mimickry, to be sure) is going to be constrained in a way that humans aren't when it comes to novelty. I mean, we're talking about technology that simply cannot work at all without being given *massive* datasets full of things humans have created, so that it can "learn" complex patterns and then use these for generation. It would be a scientific miracle if somehow this architecture gave rise to a process other than the one that AI engineers designed for it.
Now, I don't know how to account for human creativity. Minds are mysterious things. Certainly a lot of human creativity is based on mimickry. Not just "a lot"; the majority. But it can't all be mimickry, otherwise new forms of art would never come into being. There wasn't jazz in the 1800s, there wasn't hip-hop in the 1950s, there wasn't k-pop in the 1980s. But those things exist now. How'd that happen? If you trained a music generation AI only on music recorded prior to, say 1960, and tried to get it to create some Ramones-style punk, it just would not be able to do it, no matter how detailed your prompt. And yet, the Ramones existed. And, if you went back in time to 1960 and told some kid to play high tempo 8th note downstroke power chords on his guitar, he could do it even though that wasn't a thing yet. The conceptual description is enough for a human.
Or, take visual art. If you trained an image generating AI only on art made before 1900, no amount of prompt engineering would get it to produce a Jackson Pollack knock-off. It could spit out Monet knock-offs like there's no tomorrow, but tell it to make an image that looks like someone had spent the day flinging paint off a brush and onto a canvas, and you'd get... I dunno, probably something that looks like a Monet. On the other hand, if you went back to 1900 and told a person to spend all day flinging paint off the brush and onto a giant canvas, they could do it, and after they did it they'd have at least a 2nd order novelty in front of them (the 1st order being the idea to do it in the first place).
I admittedly don't know how to pin this down finely. I don't know how to identify a line between what an AI can and cannot do, given its training. But I know how deep learning works and I know that our minds have to be doing something fundamentally different and - when it comes to creativity - much more sophisticated.
I don't know about you, but I am reminded of this all the time when using AI. There's a common kind of "uncanny valley" sensation when you realize the AI just cannot do what you're trying to get it to do. Kinda like how light switches don't work in a lucid dream. I don't know why it's light switches, but I know the dream is a simulacrum and so of course some aspects will seem more "real" than others. LLMs produce simulacrums of human minds, which often feel utterly convincing. But at some point I always run into the equivalent of that fake light switch, which reminds me that the apparent human-like mental abilities I'm witnessing from the LLM are illusory. An illusion is still an illusion even when it's convincing.
I’ve often thought that AI generated images and videos have a dream-like quality to them.
Mormons are very into trans humanism
I don’t quite get it but the religious people are more into control and hoarding than they are into souls in a lot of cases
What is that thing where you hear someone talk about something you know a lot about and you get furious about how wrong they are, but then you forget the next episode cuz you don't know the topic? I'm in the first part of that rn. The linguistic mechanics of AI "tells" is actually fascinating and nuanced. There is a reason stuff feels like AI, and it's not just because it's bad writing. K&J are way over their skis on this one
I was also pulling my hair out when they discussed the "tells." For two professional writers who - by their own admission - use AI constantly, to the point where Katie has a cute nickname for it, they really couldn't think of anything other than em-dashes? Like come on you two, you have to be more observant than that.
> they really couldn't think of anything other than em-dashes?
Maybe listen back? They had a number of tics they described as typical of AI-generated text.
Yeah based on what Katie read on a Reddit post, on their own they could barely come up with any.
You’re looking for the Gell-Mann amnesia effect.
Thank you!
This kind of post is useless without an example. Show your knowledge!
Haven't finished the episode, but the metaphor overload in the first passage they read from shy girl is very AI-coded.
The thing people don't realize is that AI writes poorly because the 10s of thousands of people that trained it by rating prompts converged on a slightly above average IQ. So they collectively could instill "metaphors good" but couldn't pick up "too many metaphors bad".
Most AI writing style things are limitations of the training and not the technology, so they follow this pattern. AIs think that the sorts of things midwits write are a positive good, rather than an imperfect version of a better thing. If you could get 10,000 MFAs to spend a year each training the same AI, you'd end up with really good writing.
So the biggest tell is "mediocre and proud of it", because AIs get a cookie when they're mediocre and expect the end user to be impressed.
I'm working on a much larger article, but much of it comes down to idiolect and register. These are linguistic fingerprints that are very hard to fake, especially across a large sample. Other tells include over-repetition of context and various violations of Gricean maxims
Cool! What that has to do with the episode and what was talked about remains a mystery though.
Wikipedia has its issues but its editors are very vigilant about sussing out AI writing. There’s a whole page on it, fascinating reading: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
“4 out of 28 children. . .” Why not just say 1 out of 7? Definitely AI.
The Shy Girl story reminds me of the Goodreads Reylo scandal, where the author got caught giving 1 star reviews to rival authors and invented a story about how her fanfic friend was behind it.
Another big publishing scandal right now is about The Tell. It’s probably the biggest plagiarism scandal in decades. Hopefully BARpod will cover it.
I would like to know more. Have a link?
https://www.nytimes.com/2026/03/05/nyregion/amy-griffin-the-tell-lawsuit.html?unlocked_article_code=1.YVA.m1la.BhPHMKFkyJOg&smid=nytcore-ios-share
Not sure it’s plagiarism to claim a memory that turns out to have been suggested by someone else’s experience. We don’t own the copyright on our experiences. If she was genuinely unaware she was doing that, it would fit with the “recovered memory” source. If she was aware she was just claiming someone else’s experience as her own, that would be immoral and unethical but I don’t see how that’s an IP violation.
I don’t think she actually underwent psychedelic therapy; it sounds like she made that part up too. She’s being sued for “invasion of privacy, negligence, and infliction of emotional distress.”
There’s also been other reporting that she essentially defamed one of her middle school teachers (whom she accused of sexual assault) by describing him in enough detail that he could be identified despite changing his name. So that could result in a lawsuit too. The thing about her ghostwriter posing as an agent who wheedled more information out of the Jane Doe could certainly implicate her in some kind of fraud scheme. And it wouldn’t surprise me if the publishing house ends up suing her too to exonerate themselves.
Really, I just used “plagiarism” as a placeholder. I’m not sure there’s one term to describe this scenario. It’s pretty shocking.
Yes, she sounds like an amoral attention hound. I’m just not sure lawsuits are the solution because there are too many fuzzy boundaries here. I mean, other memoirists lied but is that fraud? Perhaps the teacher could prove damages, but the classmate? She might only be mad she didn’t get a best seller out of her experience. None of that is to endorse the memoirist, who is exploiting audience prurience and exacerbating false beliefs about memory!
Just a couple points of fact on how LLMs work that are relevant to this episode:
1. LLMs generate text *probabilistically*. There are no deterministic rules involved, and you cannot make the LLM implement a deterministic rule via a prompt. It is incapable of following such a rule in the way that we traditionally expect computers to follow rules (e.g. "turn off screen after 5 minutes of inactivity"; "take all the orange pixels and turn them green"). Saying "don't add anything I didn't write" will likely improve results to the extent that such a directive is probabilistically associated with output that is more similar to the input than it would have otherwise been without that instruction. But this doesn't mean it's actually following your rule; the rule is just changing next-token probabilities (tokens are short series of characters, kinda like words but split up differently). And if you ask an LLM to edit, it has to change *some* words. And it doesn't "know" which ones matter and which ones don't. Our ability to influence LLM output via careful prompting is always subject to this fundamental limitation.
Now, there are some AIs that mix deterministic functions with probabilistic text generation. If you feed a resume to an AI and say "rearrage and reformat this to make it look professional, but don't change any content", this might result in calling a deterministic function that actually checks to make sure the words are all the same. But the LLM itself is simply not capable of performing this task; that's not how LLMs work.
2. On energy use, it takes VASTLY more computational power for an LLM to generate text than it takes for traditional Google to return an ordered list of URLs. Maybe this doesn't matter so much anymore, since Google has decided to give us LLM generated text whether we ask for it or not. But the way search works and the was token generation works are just nothing alike. Briefly:
- When you put a query into a search engine, the search engine pulls results from an existing giant database of URL that have been "crawled". The algorithm for choosing these takes your input and then applies whatever the search engine has developed on the back-end to decide on an ordered list of URLs.
- When you prompt an LLM, it feeds your input through a series of (mostly) large matrix and vector multiplications in order to get a ranked list of probable next tokens, and then one is selected. So now we have the first token (short series of characters) of the LLM's response. Then, your prompt plus that one token it just chose is fed through this process again. Then your prompt plus those two tokens it just chose is fed through again, etc. It keeps doing this until it chooses (probabilistically!) the "stop" token.
Estimates for the relative computational costs using traditional search to an LLM are one to two orders of magnitude (10x to 100x more power needed for the LLM). If you're using a "reasoning" model, toss some more zeroes on there.
I'm not telling anyone they should care about this; it's not near the top of my reasons for disliking AI. But since Jesse brought it up...
Thanks, this is helpful. I had been wondering how LLMs differed from search.
Glad to be of help! I guess a shorter answer would be that "search" actually searches websites whereas LLMs use the vast mountain of text they were trained on to generate plausible sounding answers, which often are good but sometimes aren't. You could say they're "searching" through their training data, in a roundabout way. But they aren't searching websites (unless it's one that has a website search function tacked on, but with those the LLM portion will often misrepresent what's on the website)
Most of the professions that are going to be significantly impacted by AI are heavily left leaning.
That's why their opposition to AI is more unhinged than the right.
Whatever "ickiness" Katie and Jessie feel is not going to be shared by most. We can look historically at creative work that is simply obsolete or exist in niche, nostalgia driven markets.
Take animation. Used to be an undertaking of massive creative labor by human beings drawing, coloring, and refining every single frame. Now? 100% generated. No one watching a Pixar or modern animated film feels "icky" because a human didn't create it. We think of the modern operators who do rigging, etc. as creatives.
But painting and drawing frame by frame? Gone. /done
Journalism is going to go this way too. Probably programming. Banging on a keyboard to crank out 1000's of words is just not going to be what a journalist does. And no one used to the technology is going to care or be icked out by it. . . . anymore than they care about their cartoons not being hand painted.
What's interesting is the entire concept of plagiarism is going to change. No one says a scientist plagiarized their published paper because they used calculators to compute the numbers or computers to process the statistical analysis.
That's not where the value comes from. It comes from the data collection, curation, decisions on which statistical models to use, etc. Why should we care if an AI model generated the text if the content is accurate and the findings of interest? They won't.
Your claims about computer animation are absolute bupkis and demonstrate zero understanding of the medium. A LOT of time, work, and artistry goes into designing and rigging models, and that's before you even get to the animation phase. The turnaround time on a CGI film today is about the same as a traditionally animated film in the 1980s, because though frames don't have to be individually drawn or painted anymore, CGI allows for a lot more mid-production meddling that eats up time in its own right. I won't get too far into it, but once your model is finalized and rigged, a big chunk of the work is done, so scrapping a scene doesn't put you back at square one the way it does with 2D. Today, producers, studio heads (especially at big companies like Pixar), and directors obsessively test-screen and tweak and re-test-screen and re-tweak films, effectively offsetting the time saved by using CGI instead of 2D. That's one of the main reasons why CGI is preferred to 2D in mainstream film; re-animating a 2D scene is much costlier and requires that studios have faith in directors and that directors fully commit to a unified vision. This is part of why The Thief and the Cobbler was never properly finished; the director kept changing his mind. Spider-Man: Across the Spider-Verse is an example of a CGI film where the amount of last-second revisions inflated the budget and caused ethical concerns for animators.
Also, frame-by-frame animation absolutely still happens, it's just not being made by major studios like Disney or Dreamworks. There's usually at least one 2D film nominated for Best Animated Picture every year, and there's still a huge market for Japanese 2D animation.
TL;DR: CGI films are very much a labor of love made by human beings, and it's an insult to the profession that you would insinuate that they aren't. Watching a CGI film is in no way comparable to watching 90 minutes of AI slop.
You're actually reinforcing my exact point.
The artists that hand draw frames don't exist anymore for the most part.
Tools came about that largely eliminated that work. . . . and in the process created entirely new types of work. Creatives adapted and began using the new tools to....create.
And now, you find it laughable and insulting to suggest someone using these tools are _not_ artists and are _not_ creating. Which I did not remotely insinuate; I explicitly said we consider the riggers, designers etc as artists today. Their labor is still there. Their vision is still there. Using new tools in new ways to create even more amazing art.
Now. We have AI. It can generate text, articles, books, and getting closer and closer to creating original research and science publications indistinguishable from those written by hand. And creatives in writing, journalism, science, are getting triggered by people using these tools. Saying if you do you're not a real journalist or _actually_ creating.
Flash forward a few decades when it'll seem anachronistic to sit at a keyboard and bang out 1000's of words of text. Someone on some forum will get triggered by even the slightest suggestion that if you use these tools you're not _actually_ creating. The written word will become like clay that can be shaped and formed en masse into new works.
How will they be using them? What will the new content look like?
Who the fuck knows? That's like asking a farmer in 1850 what farming in 2000 would look like. Or asking the creator of Steamboat Willie in the '20s how Into the Spyderverse or Avatar would be made or what they'd look like.
They literally could not even begin to describe it. They wouldn't have the words.
CGI takes the same amount of effort as 2D animation, it's just a different process. It was not invented to "reduce" work the way AI was, it was invented to achieve things practical effects couldn't, and some people at Pixar decided to use the technique to make a whole movie. Asking a robot to chew up a bunch of other peoples' artwork and shit out an amalgamation of it requires no artistic skill. Pro-AI-"art" guys literally use the lack of skill it requires as a talking point, accusing artists who dislike it of "ableism." This is not a situation where the same level of expertise is being applied but with different skillsets, as with 2D and CGI.
If you couldn't be assed to make or write something yourself, why should I be assed to look at it or read it? And I'm sure as shit not going to spend money on it. Art is a tool of human expression, that is its main purpose. It doesn't matter how "indistinguishable" it is, for the same reason you wouldn't accept an "indistinguishable" robot somebody made of your dead grandmother. Having a robot make art for you is like programming a robot to feel emotions for you. A lot of use cases for AI, like organization and copy editing, make sense. This one does not.
THIS!!!!! I don't care how well an AI imitates writing or painting or animation or music. I'm not interested. It actually matters to me that the products of human creation which I choose to spend my time experiencing be made by humans.
So, the people making the AI stuff can either deal with the fact that a lot of us will reject it out of hand, or they can lie. Like they're doing right now everything they fail to disclose their use of AI.
Didn’t people say similar things about photography back when it was new? Some believed it was literally soul sucking, unlike painting.
I don’t know—humans find new modes of expression and some of them get elevated into art forms.
I think this is fair, but the analogy lacks something important: no one claimed that photography could be a substitute for a painting. No one predicted that photograph technology was a year or two away from becoming just as good a painting. Photography was a genuinely new medium.
I don't think AI art is analagous. I mean, the whole way the technology works is by feeding the AI image generator a bazillion images and running it through an algorithm that "learns" to imitate what it was fed.
I'm sure they did, but it's a bit of a false equivalence, because photography is still human expression at its core, just as all art is. Crucially, photos inherently have provenance, because of their relationship to time and place; in a way, photography is equal parts art and journalism, which is why Pulitzer prices are awarded to photographers. When a person takes a picture, they're expressing something and immortalizing a singular moment in time. Consider Falling Man, one of the most famous photos of the 21st century. The photo is important not only because it's of a real thing that happened, but also because it succinctly captures of profundity of a tragedy in a deeply resonant, strikingly intimate way.
Could somebody have theoretically made an identical piece of art by telling AI to generate an image of a man falling from a skyscraper? Yes, but it wouldn't have had the same impact. The artistry is in the photo's provenance, in the human emotions it captured and the emotions that motivated the cameraman to point the camera at that particular man in that particular moment. AI lacks the human motivations that make a painting OR a photo a piece of art. The "decisions" of AI are not motivated by anything beyond fulfilling the prompt. There is no provenance in a black box, it just excretes an image into existence based on what effectively amounts to word association.
To be clear, I'm not against AI integration into visual art wholesale. It can be a useful tool for the ideation phase of a drawing or painting, similar to how someone would use Pinterest. AI's ability to collate and assemble images into one has utility, I don't deny that. I am specifically against the characterization (or god forbid, normalization) of entirely AI-generated images as art. I would also like to note as an aside that in the eyes of the law, the folks writing prompts for AI images are not legally the creators. The LLMs themselves are the creators, which is why raw AI images, just like photos taken by monkeys, are public domain; intellectual property laws require a human creator.
Having an AI that does a great job of generating articles and books is only half the equation. You also need people who will choose to spend their time reading it. For me, language is a form of human communication. With some narrow exceptions, I simply do not want to read text generated by a machine, regardless of how well it resembles what a human might have written.
Im confused. Did you reply to the wrong person?
Im the one that said it eliminated an entire job description.
For some reason substack always puts my comments in the wrong tree
A good part of the value in a scientific paper (that's worth a shit) is created in the process of writing and editing it. You find out if the ideas work and the evidence supports them by trying to explain what you did in a persuasive way. Writing is the heart of it, not the periphery. The more senior the scientist, the more their life is about writing.
Many bad scientific papers will indeed use AI but good ones don't, and won't for these reasons. Apart from anything else there's a University regulation against publishing AI generated text, even though they do supply AI. Code is different, that can be AI generated.
To be clear an LLM can't write a high quality paper, in fact it's more trouble than it's worth because of the error checking. It's useful for literature reviews, but only for finding and summarising references. You still have to read them, since it's very often wrong on topics that aren't settled, which means all topics a scientist is writing about.
"You find out if the ideas work and the evidence supports them by trying to explain what you did in a persuasive way. Writing is the heart of it, not the periphery."
Straight into my veins with this!
Not yet….
They’ll get there. Already pretty close. Papers often lack details, skip steps, forget to mention data. It’s of of the things that makes them challenging to reproduce.
I think AI could be an improvement here.
You use it then. You'll be restricted to shit journals. And you'd better not use it for writing reviews during your brief career or I'm coming for you.
So you’re planning on canceling someone who you are replying to on an anti cancel culture podcast?
What? No I'm planning on ensuring anyone who conducts peer review of novel scientific work without reading or understanding it, instead putting it through an LLM - whether to save time or for any other reason - is no longer asked to peer review other people's work, because they aren't meeting the professional standards required of them. They can turn down reviews if they don't have time, that is what people do.
It has nothing to do with fucking cancel culture that's truly an inane remark.
So you’re going to cancel them from doing something because you don’t like the process that they choose to go about it
Thats fine for now. Sure.
It’ll change.
"It'll change"? Into what?
All of these vatic one-line-per-sentence AI evangelist accounts swear that the horizon is just around the corner* and soon every single creative industry will be subsumed. Like, okay, when? It hasn't gotten any better for years.
* a classic LLM mixed metaphor I've seen them use more than once
AI models are already conducting peer reviewed science of a quality that meets or exceeds that created by humans.
What will change is acceptance.
https://sakana.ai/ai-scientist-nature/
Your examples are all of machines enhancing something that originates from a human mind. Going from using a keyboard to using an LLM is not analagous to going from, say, a pencil to a keyboard. Nor would it be analagous to going from a keyboard to a voice-to-text translator. Because the pencil and the keyboard and the voice-to-text are all being used to communicate language chosen by a human being. The LLM replacing (at least to an extent) the human, which fundamentally alters what it means to communicate.
Likewise for the cartoons. Going from hand-drawn cells to CGI is not akin to going from CGI to generating a cartoon from a prompt, because the latter removes a huge amount of human intention.
And hey, maybe you're right and the public will embrace this. I kinda doubt it but we'll see.
As for using calculators rather than doing math by hand, it's again the same thing. Using data analysis software to fit a regression model doesn't change anything relative to using a pencil and paper, except that it doesn't take 6 months. Feeding data and a research question to an AI and saying "do a statistical analysis" is a fundamentally different because now a human being isn't choosing the method to be applied.
The Hex code situation absolutely baffles me, because every Windows computer has a Hex code picker as part of the operating system! Taylor didn't have to Google ANYTHING, because you can just hit the PrtScn key or Windows+Shift+S to see a drop-down menu offering a dropper tool that translates any color you hover over with your mouse into either Hex or RGB. Your computer doesn't even have to have a freaking internet connection for you to be able to do this. I imagine iOS has a similar function. For a writer whose beat is technology, Taylor Lorenz seems to have pretty crummy tech literacy.
The issue with AI is that it’s too easy to use it improperly. It can be very helpful as an editing tool if you retain control e.g. you ask it to point out specific errors etc. but you don’t use the text it spits out, you keep your original document open and edit manually. But if you get lazy or sloppy (I’m no stranger to both) you skip the safeguarding steps.
I'm happy Jesse and Katie mentioned that the environmental impact of AI is PEANUTS compared to the meat, egg and dairy industries. As a vegan this argument has been driving me up a wall for 2+ years. I'm a French as a foreign language teacher and regularly use AI to generate the annoying grammar exercises I hate writing, things like "conjugate all the verbs in the future tense" where it's just ten sentences like "Tomorrow, my sister and I will go to the movie." I hate this and will gleefully keep asking AI to make these exercises for me and go to sleep knowing my veganism makes me way more environmentally sound than the thousands of meat eaters being performatively angry at AI on social media.
I think this is a fair point (Go vegan!), but one should compare apples to apples. It's not merely a question of whether it's better to avoid AI or avoid dairy, but "Should I eat meat for protein, or something similar made of beans? Are there benefits to the less environmentally friendly option that justify the difference in impact?", "Should I do a regular search on DDG, or ask ChatGPT? Blah blah impact" etc
When you talk about the cop, janitor thing, are you referencing the show, “high potential?”
That was EXACTLY my question. My wife and I were listening to the episode and we were like, "Is she watching High Potential?"
Good Will Hunting, i assume high potential also is referencing good will hunting
but it was specifically the addition of being a detective, which is high potential
I think that the classic book with a passage that got mistaken for AI by one of the detectors was Frankenstein.
By Mary Shellmy?
becoming a forever covid-er was probably the dumbest thing Taylor lorenz could have done for herself, but also fully tracks and is completely on brand
someone needs to refresh herself on whether hamburger helper has meat in it (the word "helper" is a hint)
Some of it does! Or at least used to if you count non-egg/dairy animal products as meat. It's in the flavor packets.
If you are having Hamburger Helper twice a week, you are browning the meat yourself.
I guess you haven't seen National Lampoon's Vacation
https://youtube.com/shorts/58KzP9HEF_k?si=mK62LrI6_ud2seYu