How to safely use AI for medical information

By Prav Prathapan 

Artificially ‘intelligent’ chatbots have quickly become the first stop for millions of people with a health question. That is not going to reverse, so the useful question is not whether to use them, but how.

Key points: How to use AI for health questions

1. Your health professional comes first.
2. Verify anything that matters.
3. Keep your personal data out.

Where chatbots go wrong

Here’s the thing about a chatbot getting it wrong: it doesn’t sound wrong.

The same calm, well-articulated paragraph ensues whether the answer is solid or completely made up.

Here are three examples where they make mistakes:

1. They make things up.

   Also known as “hallucination”, chatbots can make up a drug interaction, a study or a dose. Did they cite a reference? These could be made up too. All stated with total confidence.

2. They can be out of date.

   Chatbots are trained on data and then cut-off for use by consumers. Therefore, any medical guidance that has been updated or newly released may not have made it in. However, newer models can search the web for current information, which narrows this gap, but it is worth checking whether it is drawing on reputable sources.

3. They follow your lead.

   The importance of a user “prompt” cannot be overstated. This is not to say it’s the user’s fault but that this, for many chatbots, is all they have to go on to know who you are.

In February this year, a landmark Oxford study published in Nature Medicine warned of the risks of AI chatbots in giving medical advice (1). More than a thousand participants took part.

The researchers identified several recurring problems (2). First, people didn’t know what information to give the model. The answer you get depends heavily on what you prompt, and patients are not trained to know what’s clinically relevant.

Related to this is the fact that models were deemed unstable: just small changes in wording produced substantially different recommendations. Most worryingly, responses also tended to mix sound advice with poor advice in a way the participants could not reliably distinguish.

The health-information sector reached the same conclusion first

Years before the Oxford trial appeared, the Patient Information Forum had already published a position statement on the risks and benefits of using generative AI (GAI) in producing health information. Their conclusion was blunt (3):

“Currently, GAI is not suitable for the creation of health information and content in isolation. The risk of inaccurate, biased outputs that lack the necessary nuance and context to provide individualised or specific health information are too high.”

Two very different exercises: a randomised trial of public users, and a professional body reviewing its own field. Yet both arrived at the same warning.

Top tip: Do not enter your personal data

The great thing about chatbots is that they have excellent memory. One study even went as far as saying that ‘chatbots have great potential for history-taking… the health care system can be improved by 24/7 automated data collection’ (4).

However, a consumer chatbot is neither your doctor, nor your hospital. And it is not bound by the confidentiality rules that cover them.

Depending on the service and your settings, conversations may be stored, reviewed by staff for quality or safety, or used to improve future models. None of that is sinister. It’s just the way software is developed, and you should treat it that way.

This applies even more to photos, voice recordings, and videos. Many chatbots can now analyse an image of a rash or listen to a description of your symptoms, which can be genuinely helpful. But these files can identify you far more easily than text and may be stored or used in ways you did not intend. Crop out faces, tattoos, and anything else that identifies you.

The protections you would assume in a consulting room do not apply to a chatbot, so behave accordingly. Make use of the ‘Private’ or ‘Incognito’ options offered by the chatbot to ensure your data is safely erased after use. You can also manage your data in the chatbot’s settings.

How should one use AI?

First, consult your health professional. Your health care professional should always be your number 1 port of call if you have any concerns about your health. What your family doctor/GP says should always take precedence over anything AI tells you.

Give context, not identity. You can describe your age, relevant conditions, medications, and how long symptoms have lasted, but not who you are. Take out your full name, date of birth, address, healthcare or insurance number, and the name of your clinician. “A 54-year-old with high blood pressure has been prescribed X. What does it do?” gets you the same answer as a version with your name on it.

Treat every answer as a starting point, not a conclusion. AI can state wrong things fluently and it can be out of date on guidelines. Check everything that matters against your government healthcare website, a pharmacist, or your GP/family doctor.

Get AI to factcheck and ‘think harder’. If you challenge the response you receive, you may get a more nuanced answer to your initial query.

Don’t skip the warnings. Many chatbots add a note telling you to consult a professional, and these are there for a reason.

Conclusion

None of this is an argument for avoiding AI. On the contrary, it is a useful and arguably the most powerful tool for making sense of a diagnosis. More than that, it gives you a space to describe symptoms freely and without embarrassment. For people around the world who face long waits or have limited access to health services, this support may be poignantly valuable. But safety comes first.

These tools are an aid to understanding. Not a source of clinical judgement. And it should never decide if you should seek a health professional. Only you should.

So use carefully. Do not input your personal data. Verify and double check any answers that would make you change your behaviour or habits. And remember that mere confidence in the AI’s answer says nothing about whether it is correct.

References:

1. Bean, A.M., Payne, R.E., Parsons, G. et al.Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nat Med 32, 609–615 (2026). https://doi.org/10.1038/s41591-025-04074-y
2. https://www.ox.ac.uk/news/2026-02-10-new-study-warns-risks-ai-chatbots-giving-medical-advice
3. https://pifonline.org.uk/download/file/RFdSYkRnT0ZQT3l5UHhpOExxTUk0QT09/pif-position-statement-balancing-the-risks-and-benefits-of-ai-in-the-production-of-health-information/
4. Hindelang M, Sitaru S, Zink A. Transforming Health Care Through Chatbots for Medical History-Taking and Future Directions: Comprehensive Systematic Review. JMIR Med Inform. 2024 Aug 29;12:e56628. doi: 10.2196/56628. PMID: 39207827; PMCID: PMC11393511.