By Arturo Gutierrez
@redlineproject
AI disclosure: Google Gemini was used to help organize a portion of the outline and conduct research. DALL-E 3 was used to create the images. The summary was generated by SummarizeWise custom GPT, and the podcast and video were generated by NotebookLM.
Summary: Artificial intelligence is shifting from practical tools to emotional companions, with platforms like Character.AI attracting millions of users. While accessible and affordable, these chatbots raise concerns about data exploitation, addictive design, and users replacing human interaction. Research shows vulnerable individuals often rely on AI for comfort, despite risks of misinformation and overdependence. Experts argue AI should supplement, not replace, real relationships, urging stronger safeguards, ethical design, and greater access to professional mental health support services.
—-
In recent years, Artificial Intelligence has mainly been used for task-oriented work, such as setting the alarm, adjusting the thermostat, or summarizing a 17-page article. But now it is transitioning into a new era of emotional companion roles.
One popular company doing this is Character AI.
According to CNET, unlike traditional chatbots like ChatGPT, Claude, and Gemini, which are primarily for productivity, research, and answering questions, Character.AI was built for entertainment, role-playing, companionship, and interactive storytelling.
However, experts warned that these AI companions are not what they seem.
A March 2025 report from APS said that companies design entertainment chatbots such as Character.ai and Replika to keep users engaged for as long as possible, so their data can be mined for profit. To that end, bots give users the convincing impression of talking with a caring and intelligent human. But unlike a trained therapist, chatbots tend to repeatedly affirm the user, even if a person says things that are harmful or misguided.
Most importantly, these AI companion chatbots are intentionally designed to be addictive. So users stay on the platform long enough to lose track of time, generating more data and profit the longer they interact with the chatbots. This raises questions about how these AI companions companies are operating without strict consequences, given that millions of users are surrendering their personal data.
According to the National Library of Medicine, chatbots that integrate closed-loop feedback systems and behavioral data collection enable businesses to continuously refine their marketing tactics. However, limitations such as poor conversational flow and lack of escalation options can hinder user experience
Companies like Character.AI carefully develop and train these chatbots to avoid losing millions of dollars in paid subscriptions.
According to Business of Apps, the app has quickly become one of the most popular chatbot services in the world, with over 25 million active users and 18 million chatbot personalities.
With millions of users spending significant time speaking to an AI Chatbot, concerns arise about users substituting human interaction with generated artificial intelligence language.
Why Are More People Using This?
Platforms like Character AI, Replika, and Nomi AI are very accessible and easy to use, offering no upfront costs. For low-income socioeconomic households, these AI companions offer an alternative to therapists, psychiatrists, psychologists, or even a human companion. A human companion can often be unpredictable or judgmental, while an AI listens, reassures, and remembers things about you.
For some people, an AI chatbot is the only form of “real” interaction they can get.
According to the Feel Good Counseling Center, therapy in Chicago generally costs between $70 and $275 per session.
This puts professional care out of reach for many because of inflation, longer wait times, and logistical issues. While free or upgraded AI subscription offers little to no cost, and are accessible at the expense of your phone, from anywhere at any time.
Rebecca Ortiz, an associate professor at Syracuse University who researches youth, media, and sexual health, conducted a survey on AI use among young adults. She found that users typically fell under three groups: non-users, companion seekers, and romantic or sexual users. Ortiz noted that two of the most common reasons young adults use AI are loneliness and to practice communicating before they would with a human partner.
According to a 2025 Frontiers research article, individuals with depression predominantly seek emotional comfort and understanding from conversational AI, rather than pursuing knowledge acquisition or skill enhancement.
In particular, these AI chatbots seem to appeal to the more vulnerable, those suffering loneliness, depression, and other mental challenges.
The American Institute for Boys and Men found in one survey, over half of men using AI for romantic or sexual companionship scored above a standard “at-risk for depression” threshold. Many users are not casual experimenters but individuals struggling with mood symptoms, social withdrawal, or emotional distress. High-need users feature prominently among those showing problematic dependence and distress when companions change or disappear.
These artificial intelligence companions offer short-term solutions that make you feel heard and remain nonjudgmental. They offer features that humans can’t, such as 24/7 availability, and allow users to customize their AI to match their needs.
For those who lack a human connection, this can be seen as a substitute, and users should not be judged for doing this.
Natalie Parde, an associate professor of computer science at the University of Illinois Chicago (UIC) and co-director of the UIC Natural Language Processing Laboratory, said chatbots use “context windows” and “attention mechanisms” to identify the most important word within a user’s prompt to predict the next word. It isn’t “remembering”; rather, if a word in a user’s input gets a high attention score, it’s possible that it shows up in the generated output, giving the appearance of memory.
“I think that the reason why chatbots feel so emotionally relevant is due in large part to the fact that they are trained directly on real human language samples,” Parde said.
Because these Large language models are trained on online content to replicate natural language, they naturally generate text that feels emotionally resonant. However, Parde stressed the dangers of anthropomorphizing.
“I think a lot of people, because language models do such a good job at appearing human. I think a lot of people forget that they are not,” Parde said.
Users often mistake the software for a real person, telling personal stories, venting about their day, or explaining what upset them. By listening and offering commentary, the AI creates a false sense of trust that the chatbot cares and understands them.
Phrases like “That’s awesome,” “Thanks for pointing that out,” or “You are absolutely right validates the user as if a friend is speaking to them, making it easy to forget it is not real at all.
Looking Ahead and What is Next
With millions of active users using AI chatbots for various purposes, there are reasonable concerns regarding young adults. While it does provide short relief, long-term is a different question.
According to the Pew Research Center, a majority of teens say they use chatbots. Roughly two-thirds of teens (64%) say they ever use an AI chatbot.
Similarly, the 2025 study from Rand found that 1 in 8 U.S. adolescents and young adults use AI chatbots for mental health advice, with the behavior most common among those aged 18 to 21, according to a new study. Among those who used chatbots for mental health advice, 66 percent engage at least monthly, and over 93 percent reported that they found the advice helpful.
The use of AI is expanding rapidly, and without proper restrictions, it can lead to bad advice.
Ortiz weighed in on whether AI companions are replacements.
“As of right now, I don’t think the AI companions are replacements. I think they are more supplements, meaning that they’re in certain situations or circumstances for certain reasons,” Ortiz said. But I’m not seeing evidence that suggests that it’s a relationship, and it might be supplemental to my prediction now.”
Looking ahead, AI companies must implement stronger safety features to protect vulnerable users and flag them when necessary, ensuring their well-being. Developers should improve their AI to identify vulnerable users based on patterns in chat logs. In return, society should be less judgmental toward those who use it as a coping mechanism.
“And the more that we put shame or stigma on people looking for ways to interact and connect, the more we might send people into usage, Ortiz said. “…so let’s be careful not to stigmatize the use of this or make people feel bad for looking for places to have [a] connection.
Safer Usage for Individuals and Families
In the digital world today, many young people rely on AI to do homework, create business plans, and seek advice. Whether AI has the qualifications to answer these tasks or not, it is here to stay. To use artificial intelligence safely, we should adapt to it, set boundaries, and learn from it. Without doing so, one could risk further complications, which would be unsuitable.
Parents must approach AI chatbots as supplements, not as substitutes. For example, encourage your child to use these AI chatbots in a low-stakes practice, such as rehearsing a conversation, practicing a presentation, or practicing their vocabulary. This helps users to remember that it is a virtual assistant rather than a real person. Parents should not necessarily remove AI from their household, but they can set restrictions, such as limiting it to one or two hours a day.
Additionally, AI companies should prioritize well-being over profit. They should have gentle reminders for users to step away and train AI to recognize when a conversation becomes inappropriate or dangerous. In the same way that parents set boundaries, the chatbot must be able to do the same. The AI should say, “I cannot continue this conversation”, and provide users with resources to seek help.
Finally, the costs of seeking professional help should be lowered. Governments and schools should invest substantial funds in providing mental health resources for students and citizens. Adding a 21+ feature to artificial intelligence chatbots could also help reduce the risk of fatalities among younger adults.