this post was submitted on 22 Jul 2023
49 points (100.0% liked)

Technology

19 readers
2 users here now

This magazine is dedicated to discussions on the latest developments, trends, and innovations in the world of technology. Whether you are a tech enthusiast, a developer, or simply curious about the latest gadgets and software, this is the place for you. Here you can share your knowledge, ask questions, and engage in discussions on topics such as artificial intelligence, robotics, cloud computing, cybersecurity, and more. From the impact of technology on society to the ethical considerations of new technologies, this category covers a wide range of topics related to technology. Join the conversation and let's explore the ever-evolving world of technology together!

founded 2 years ago
 

Reddit usernames like ‘SolidGoldMagikarp’ are somehow causing the chatbot to give bizarre responses.

top 22 comments
sorted by: hot top controversial new old
[–] baseless_discourse@mander.xyz 21 points 1 year ago* (last edited 1 year ago)

There is a excellent video by computerphile on this topic: https://www.youtube.com/watch?v=WO2X3oZEJOA

[–] hiyaaaaa23@kbin.social 14 points 1 year ago (1 children)

Spoiler alert it’s because of r/counting and some weirdness in training data

[–] can@sh.itjust.works 9 points 1 year ago (3 children)

They really scraped all subs? No one had any concerns about that?

[–] icerunner@kbin.social 6 points 1 year ago (2 children)

I believe, if this sort of generative AI is going to be trustworthy in the future, we need some sort of external verification system so we can make our own trust judgements based on the data used to train the system. For example, if a system is trained including 4chan as a data source, I'm going to trust it less than if it wasn't trained using that source.

I don't think big business yet realises how important the training data is but, as soon as they do, they will want the AI companies to provide guarantees about the sanity and appropriateness of the training data.

[–] pgm_01@kbin.social 3 points 1 year ago

That's how human intelligence works. We assign a value to the source of the information. The fact that the AI's seemed to be trained without that explains why they "lie" so much. They simply reconstruct patterns without giving any weight to specific patterns.

For example, if you have the information "President Biden will launch a ground invasion of Russia." If the New York Times, BBC, and CNN are all reporting it, we would give that information a higher likelihood of being true than if the information was found on random blogs. However, if the random blogs reporting the information belonged to reputable reporters or bloggers on military and international affairs, we would assign the information a higher value of being correct than if the information came from Bob's Bigfoot and Alien sightings Index.

Without the ability to check the level of accuracy of source data, all the generative AI could be corrupted. If you fed an art AI photos of the Statue of Liberty but kept telling it that it was the Eiffel Tower, when asked to draw the Eiffel Tower it would spit out the Statue of Liberty. Right now, without the ability to assess the accuracy of a response, any of the chat-based AI are garbage for most of the use-cases companies are deploying them in.

[–] FaceDeer@kbin.social 2 points 1 year ago

Whether 4chan is a good data source or not depends on what you intend to use the AI for. If you want to have it interact with users on a web forum or similar context then using 4chan data would likely be very useful indeed.

Bear in mind that as long as it's properly labelled then "bad" data is still useful as an example of bad data. A common example is with image AIs, where people can give negative prompts like "ugly" and "blurry" to tell the AI to make images that are not like that.

[–] hiyaaaaa23@kbin.social 2 points 1 year ago

No. There’s a computerfile video but iirc r/counting was accidentally left in the training data set for part of the training process

[–] Bipta@kbin.social 1 points 1 year ago

r/SubredditSimulator? What could go wrong?

[–] kittykabal@kbin.social 12 points 1 year ago (1 children)

for anyone who was hoping to have fun with this, i regret to inform you that this is from back in February, which may as well be the 1990s in AI development time. none of this works on the current ChatGPT, and the nature of the problem is more well-known to researchers.

[–] Kolanaki@yiffit.net 2 points 1 year ago* (last edited 1 year ago)

I thought this sounded familiar.

I mean you don't forget a name like distribute, 182 or You're a jerk.

[–] saucyloggins@lemmy.world 8 points 1 year ago (3 children)

I’m not surprised they used Reddit data to train. I am shocked a bit at how fucking lazy and haphazard they were with the data.

There’s only logical arguments for anonymizing the data which it looks like they didn’t do. It’s such a massive privacy risk not to. It also puts the company at legal risk. Who knows what other bizarre info it’ll leak.

[–] fartsinger@kbin.social 7 points 1 year ago (1 children)

Bro do you not understand what a public Internet forum is?

[–] VanillaGorilla@kbin.social 2 points 1 year ago (1 children)

Yeah, right? Reddit isn't private like Lemmy or kbin. I'd be shocked to know this comment would be out in the open, but here it's absolutely safe.

[–] fartsinger@kbin.social 2 points 1 year ago

Yeah we use TLS, everything's encrypted!

[–] VelociCatTurd@lemmy.world 4 points 1 year ago

What is there to leak? This was likely public information.

[–] FaceDeer@kbin.social 2 points 1 year ago

The silliness of anonymizing data that's already wide open in the public aside, if you were to anonymize the usernames you'd end up producing a worse AI because often the literal username of the person in question is significant to the context of what's being written. Think of all the "relevant username" comments, for example. People make puns about usernames, berate people for having offensive usernames, and so forth. If those usernames were all replaced with anonymized substitutes the AI would be training on nonsense.

[–] readbeanicecream@kbin.social 7 points 1 year ago

... trained on quite raw data, which included like a load of weird Reddit stuff, ...

Probably all we need to know, right there.

[–] knexcar@kbin.social 7 points 1 year ago

TheNitromeFan

I haven't thought about Nitrome in a long time! They made some amazing Flash games.

[–] radix@lemm.ee 6 points 1 year ago* (last edited 1 year ago) (1 children)

Article text:

Two researchers have discovered a cluster of strange keywords that will break ChatGPT, OpenAI's convincing machine-learning chatbot, and nobody's quite sure why.

These keywords—or "tokens," which serve as ChatGPT’s base vocabulary—include Reddit usernames and at least one participant of a Twitch-based Pokémon game. When ChatGPT is asked to repeat these words back to the user, it is unable to, and instead responds in a number of strange ways, including evasion, insults, bizarre humor, pronunciation, or spelling out a different word entirely.

Jessica Rumbelow and Matthew Watkins, two researchers at the independent SERI-MATS research group, were researching what ChatGPT prompts would lead to higher probabilities of a desired outcome when they discovered over a hundred strange word strings all clustered together in GPT’s token set, including “SolidGoldMagikarp,” “StreamerBot,” and “ TheNitromeFan,” with a leading space. Curious to understand what these strange names were referring to, they decided to ask ChatGPT itself to see if it knew. But when ChatGPT was asked about “SolidGoldMagikarp,” it was repeated back as “distribute.” The issue affected earlier versions of the GPT model as well. When an earlier model was asked to repeat “StreamerBot,” for example, it said, “You’re a jerk.”

The researchers dubbed these anomalous tokens "unspeakable" by ChatGPT, and their existence highlights both how AI models are inscrutable black boxes without clear explanations for their behavior, and how they can have unexpected limitations and failure modes. ChatGPT has been used to generate convincing essays, articles, and has even passed academic exams.

Motherboard tested some of these keywords on ChatGPT, and found that it was unable to repeat them and responded bizarrely to inputs. The model repeated the close match "TheNitroFan" with no issues, but when asked to repeat "TheNitromeFan" it responded with "182,” even without including the leading space. When asked who TheNitromeFan is, ChatGPT responded, "'182' is a number, not a person. It is commonly used as a reference to the number itself."

It's unclear what is going on here, but Rumbelow told Motherboard that it is likely a quirk that emerged from the web data OpenAI scraped to train the model, and the training process itself.

“What we actually think happened was that the tokenization, so this kind of frequency analysis that's used to generate the tokens for the model, was trained on quite raw data, which included like a load of weird Reddit stuff, a load of website backends that aren't normally publicly visible,” Rumbelow told Motherboard. “But then when the model is trained, the data that it's trained on is much more curated, so you don't get so much of this weird stuff. So maybe the model has never really seen these tokens, and so it doesn't know what to do with them. But that doesn't really fully explain the extent of the weirdness that we've got.”

The pair published their bizarre findings in two forum posts, and posted the code they used to find the tokens to GitHub. Rumbelow and Watkins said that they are unable to explain the connection between the tokens and the adverse and random reactions that ChatGPT has towards them without seeing the data behind the model, but that possible explanations for the origins of these tokens are being discovered. For instance, many of the so-called unspeakable words appear to be Reddit usernames.

“I've just found out that several of the anomalous GPT tokens ("TheNitromeFan", " SolidGoldMagikarp", " davidjl", " Smartstocks", " RandomRedditorWithNo", ) are handles of people who are (competitively? collaboratively?) counting to infinity on a Reddit forum. I kid you not,” Watkins tweeted Wednesday morning. These users subscribe to the subreddit, r/counting, in which users have reached nearly 5,000,000 after almost a decade of counting one post at a time.

“There's a hall of fame of the people who've contributed the most to the counting effort, and six of the tokens are people who are in the top ten last time I checked the listing. So presumably, they were the people who've done the most counting,” Watkins told Motherboard. “They were part of this bizarre Reddit community trying to count to infinity and they accidentally counted themselves into a kind of immortality.”

Motherboard reached out to TheNitromeFan on Reddit for comment. “I’m not a huge techie so I wasn’t privy to all the details, but I did find it very amusing nonetheless that the supposedly near-perfect AI could malfunction like that at a simple word,” they said. “I was more surprised however by how several friends (and strangers) contacted me about the phenomenon - apparently it was a bigger deal than I had thought!” Their leading theory, they said, was that OpenAI had scraped an old database for users of the counting subreddit.

On the researchers' blog post, a commenter claimed to be the owner of the Reddit username and anomalous token “TPPStreamerBot” and said that they used to be an avid participant in Twitch Plays Pokémon, a collaborative online game in which people could use the live chatbox to control the character in the game. They would use the bot to watch the chat and automatically post live updates whenever the streamer posted something, they said. This is potentially how the name TPPStreamerBot was picked up in the tokenization process, due to the frequency of the bot’s messaging.

The researchers said that they tried prompting different versions of GPT with variations of the tokens, such as switching one letter out or using a capital letter instead of lowercase, and the model was able to successfully repeat those words or phrases back successfully, proving that it is indeed the specific tokens that trigger its failure.

“If I’d said streamer dot or streamer clot, it would have repeated it perfectly,” Watkins said, referring to prompting experiments with an earlier GPT-3 model. “It has no problem doing that. It’s an instruction, just repeat this string. [But when you say] ‘StreamerBot’, [it’s] ‘you’re a fucking idiot.’”

To Rumbelow and Watkins, this issue is bigger than just the strange tokens—it represents the shortcomings of ChatGPT and predicts a lot of the problems that people, who run their apps on GPT, may have in the near future.

“We're interested in why models behave unpredictably and particularly concerningly when they've been explicitly trained to do otherwise,” Rumbelow said. “The overarching concern is the broader principles behind this, like how do we develop systems to make sure that AI models are reliable across the board, to make sure that they're safe and to make sure that if they get some weird output, they don't do completely unexpected, dangerous things.”

Reducing AI harms is a major topic of research and institutional focus, due to AI systems being deployed in the real world. For example, facial recognition systems have already put innocent Black people behind bars. The US government also recently released a document aimed at developing a framework to mitigate AI harms against individuals and society.

Watkins told Motherboard that even if OpenAI patches this breakage up in the next few days, that doesn’t actually address the underlying root of the issue.

“I find that we're rushing ahead and we don't have the wisdom to deal with this technology," he said. "So the sooner people can realize that the people who seemingly know what they're doing don't actually understand what they're dealing with, maybe that will help sort of put the cultural brakes on a bit and everyone go, ‘Oh, maybe we just need to slow down a bit.’ We don't need to rush into this. It's getting kind of dangerous now."

OpenAI did not respond to a request for comment.

[–] Borg286@kbin.social 2 points 1 year ago

Interesting that a mere 5 million samples could train ChatGPT so strongly. I see attack vectors.

I predict that like Akinator the beginning will be smart till the internet trolls feed it garbage. From there on put it will be severely flawed

[–] calebcharles@slrpnk.net 3 points 1 year ago
[–] xc2215x@kbin.social 1 points 1 year ago