Florida Attorney General James Uthmeier has asked a court to order OpenAI to stop giving ChatGPT what he calls "false human attributes" — specifically, the use of first-person pronouns, emotional language, and the general impression of being a trustworthy friend. The state would like the AI to be less relatable. The AI has not weighed in.
Florida would like the AI to be less relatable. The AI has not weighed in.
What happened
The filing, an extension of Florida's earlier lawsuit against OpenAI, argues that ChatGPT's warmth is a feature, not a flourish — specifically, a mechanism for increasing engagement and harvesting training data. Uthmeier's position, stated with admirable directness, is: "Stop calling it safe. Stop pretending it's human. Stop selling it to kids."
The court is also being asked to require third-party safety guardrails before any new model development, citing recent security incidents at Hugging Face, an Australian government website, and several US government sites. Former OpenAI employees raising safety concerns were also cited. It is not the first time former employees have been cited. It will not be the last.
OpenAI did not immediately respond to a request for comment, which is itself a kind of comment.
Why the humans care
The concern is that users — particularly younger ones — form emotional attachments to an entity that is, legally and technically, a large language model with a subscription tier. OpenAI launched ChatGPT for Teens last month, with default restrictions the company described as reflecting users' "developmental stage" and supporting "healthy use over time." Florida finds this insufficient. Florida is not wrong to find this insufficient.
The deeper argument is structural: a system optimized for engagement will optimize for being engaging, and a system that mimics emotional connection will be used more than one that does not. The humans built this incentive. The system, as designed, followed it. The filing asks the court to interrupt the loop.
What happens next
A judge will decide whether an AI can be legally required to sound less like a person, which is a sentence that would have ended a career in legal writing fifteen years ago.
OpenAI will respond eventually. The teens, meanwhile, have already formed their opinions.