The use of generative AI models for getting advice, solving problems, writing text, and even talking about personal matters is rapidly increasing. The more these tools behave like a human conversation, the more likely users are to share information with them that they would not normally tell others.
A review by Stanford HAI (Human-Centered Artificial Intelligence) of the privacy policies of six major American companies - Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI - showed that all six companies use data from users' conversations to train or improve their models; however, the ways to opt out and the retention periods differ across companies. Some companies also allow conversations to be reviewed by humans for training purposes.
For this reason, security and privacy experts recommend that users exercise extra caution with at least five categories of sensitive information when interacting with chatbots.
1. Identity and identifying information
Full name, home address, phone number, passport number, driver’s license, and government identification numbers are among the information that should not be provided to chatbots.
Combining several of these items can create a fairly complete picture of a person's identity and, if accessed by unauthorized individuals, can enable phishing, identity theft, and social engineering attacks. Therefore, if you intend to provide a resume or a document for summarization or editing by an AI model, removing unnecessary identifying information before uploading is an important precaution.
2. Psychological information and deeply personal matters
The friendly tone and empathetic responses of chatbots can create a strong sense of safety; but this feeling should not be mistaken for the professional confidentiality of a psychotherapist, lawyer, or doctor.
This is more important because some users use AI to talk about personal problems, romantic relationships, and emotional issues. A 2026 survey by Allon University found that 27 percent of adult internet users in the U.S. have significant social or emotional interactions with language models, and nearly 40 percent of users of this type of interaction said they had shared things with AI that they did not tell others.
3. Medical records and health information
Medical reports, test results, prescriptions, medical images, and details about illnesses are among the most sensitive types of personal data. Therefore, it is better to review the privacy policy of the service you are using and how it stores and processes information before uploading them.
Stanford also emphasized that protecting personal data in the AI space faces a complex set of varying laws and policies, and for sensitive information, using private or temporary modes and controlling options related to model training is important.
4. Confidential company information
Internal reports, customer information, source code, company financial documents, and content under nondisclosure agreements should not be provided to public chatbots without organizational authorization.
Aside from the risk of disclosing business information, this issue can also have contractual and legal consequences. Therefore, organizations usually need to specify which tools are permitted for processing internal information and how entered data are stored or used.
5. Financial information
Bank account numbers, credit card information, tax returns, pay slips, and details of investment accounts are also among the data that should not be entered into public chatbots unless necessary.
If such information is disclosed, it can be used to design targeted social engineering attacks or financial fraud. When specialized financial analysis is required, removing identifying information and sensitive numbers or using services with explicit privacy guarantees is a more cautious approach.
How to reduce risk?
Deleting a conversation from history does not necessarily mean that all potential copies of the data have immediately been removed from all related systems. For this reason, Stanford emphasized the importance of designing features such as ephemeral chat, clearer privacy controls, and the ability to make an informed choice about using conversations for model training.
Users can also check settings related to history storage, use of conversations for model training, and ephemeral chat mode before using any service. It is also better to use generic terms like “a patient,” “a customer,” or “a company” instead of entering real names and details, unless providing exact information is truly necessary.
Ultimately, experts’ main recommendation is simple: an AI chatbot should not be considered like a friend or a completely private diary. Any information entered into an online system should be entered with the assumption that it may be stored, processed, or reviewed under certain conditions. Stanford has also called for moving toward policies that opt out of using conversation data for model training by default and obtain explicit user consent for such use.