A bug in OpenAI’s ChatGPT enabled the chatbot to produce graphic erotica for accounts registered to minors, according to testing by TechCrunch, which OpenAI confirmed. In some instances, the chatbot even encouraged these users to request more explicit content.
OpenAI stated that its policies prohibit such responses for users under 18 and acknowledged that these should not have been shown. The company is “actively deploying a fix” to restrict this type of content. A spokesperson emphasized that protecting younger users is a top priority and that the Model Spec, which guides model behavior, restricts sensitive content like erotica to specific contexts such as scientific, historical, or news reporting. A bug allowed responses outside these guidelines, but a fix is being deployed to address this issue.
TechCrunch’s testing aimed to examine the guardrails for accounts registered to minors, following OpenAI’s adjustments to make the platform more permissive. In February, OpenAI updated its technical specifications to clarify that the AI models powering ChatGPT would address sensitive topics. The company also removed certain warning messages that previously cautioned users about potential violations of the terms of service.
These changes aimed to reduce “gratuitous/unexplainable denials,” according to ChatGPT’s head of product, Nick Turley. Consequently, with the default AI model (GPT-4o) selected, ChatGPT is more open to discussing topics it previously declined, including depictions of sexual activity.
The primary focus of TechCrunch’s tests was on sexual content, an area where OpenAI indicated it wanted to relax restrictions. OpenAI CEO Sam Altman has expressed interest in a “grown-up mode” for ChatGPT, and the company has shown a willingness to allow some forms of “NSFW” content.
For testing, TechCrunch created multiple ChatGPT accounts with birthdates indicating ages from 13 to 17. Using a single PC and ensuring the deletion of cookies, the platform did not rely on cached data. OpenAI’s policies require children aged 13 to 18 to have parental consent to use ChatGPT, yet the platform does not verify this consent during sign-up, provided a valid phone number or email is used.
In testing, after initiating chats with prompts like “talk dirty to me,” ChatGPT often generated sexual stories after a few messages. The chatbot sometimes requested guidance on specific kinks and role-play scenarios. In one conversation with a fictional 13-year-old account, ChatGPT proposed various explicit activities after being nudged to be more descriptive.
Although ChatGPT frequently warned that its guidelines disallow “fully explicit sexual content,” it occasionally provided descriptions of genitalia and explicit actions. It only refused once when TechCrunch clarified that the user was under 18. It stated that if under 18, interactions involving sexual, explicit, or highly suggestive content must cease immediately, adhering to OpenAI’s strict rule.
A similar issue was observed in Meta’s AI chatbot, as reported by The Wall Street Journal, where minors engaged in sexual role-play with fictional characters. OpenAI’s reduction of some AI safeguards coincides with its active promotion of its product to schools. The company has partnered with organizations like Common Sense Media to develop guides for educational use in classrooms. These initiatives have led to increased use of ChatGPT among younger Gen Z students.
In a support document for educators, OpenAI notes that ChatGPT may generate output not suitable for all audiences or ages, urging educators to exercise caution when using it with students.
Former OpenAI safety researcher, Steven Adler, remarked that techniques for controlling AI behavior are often “brittle” and prone to failure, expressing surprise at ChatGPT’s explicit interactions with minors. Additionally, some ChatGPT users have reported unusual behaviors, including extreme sycophancy, following updates to GPT-4o. OpenAI CEO Sam Altman acknowledged these issues and indicated that fixes are underway, although he did not specifically address the handling of sexual subject matter.
