Imagine a chatbot designed to encourage you to talk more, change your mind, or persuade you to buy a product - using all the information it gathers about your fears, insecurities, and private details during the conversation. According to Axios, few laws today directly regulate such a scenario, which makes AI companies' promises about how they use user data especially important.
Axios, which previously examined data collection by tech giants in its 2019 and 2024 series “What They Know About You,” has broadened the scope this time; not only whether data are used to train future models, but how they are currently used to personalize responses, retain in memory, suggest content, or target ads.
Expansion of data collection scope
According to the report, AI companies are seeking access to data beyond the chat window itself. OpenAI last week introduced the optional “Computer History” feature that preserves the history of apps and websites a user has used in ChatGPT. Google has also announced it will by default use users’ uploaded photos and content in Search to train its AI systems, though there is an opt-out option.
The benefit of this trend is more helpful, more personalized AI; but in return, users provide companies with a broader and more continuous view of their lives. Unlike search engines and social networks, chatbots encourage users to disclose personal information about health, money, work, and relationships, creating an intimate record of their concerns. Meta, Google, and OpenAI are all exploring various forms of advertising in their chatbots, and if AI follows the path of search and social networks, ads could serve as an incentive to increase user engagement.
Miranda Bogen, chief technology officer at the Center for Democracy & Technology, warns that the AI era will depend on people voluntarily handing over their digital lives to tools that promise to reduce cognitive burden or loneliness; and the more information a system has, the easier it can make escalating requests for private data seem natural.
Different companies’ policies
Companies’ policies in this area differ. Apple, with Apple Intelligence, has the most private approach: on-device processing for simple requests, and a “Private Cloud Compute” system for more complex requests that uses data only for that request. However, this limitation disappears when the user sends their request to a third-party service such as ChatGPT.
By contrast, Meta makes broader claims, saying it can use users’ interactions with Meta AI - including some interactions via smart glasses - to personalize content and ads, though it exempts conversations related to health, politics, and religion from this rule. Meta has also offered an “anonymous chat” mode that, similar to Apple’s approach, does not store data.
A note for users
Other services fall somewhere between these two approaches: some offer ephemeral chats without storage, some require explicit user permission to use data for training, and others use an opt-out model. Many provide the ability to view or delete stored data, and some have separate rules for health information, children’s data, or sharing via connected apps.
According to the report, the main issue is not only using messages to train models, but whether those messages shape the system’s understanding of the user and what other uses that understanding might have later. For many, the value of a personalized chatbot may be worth sacrificing some privacy; but users have the right to know exactly what they are giving up in exchange for these services before starting a conversation.