Artificial intelligence tools are listening to many people's medical appointments, but they're not always producing accurate summaries.One patient has described how a letter from a specialist to her GP included a false suggestion that she was taking illegal drugs. A so-called AI scribe had been used by the doctor.New research by a privacy watchdog has raised serious concerns about AI being used in medical settings due to its propensity to "hallucinate" false symptoms or conditions, and warned patients were at risk.CreditsPaige Cockburn, ReporterProgram:More from ABC News Top StoriesTranscriptPaige Cockburn: AI scribes, as they're called, have become prevalent in the healthcare sector, mainly because they free up the doctor from having to write notes during a consultation. When Rebecca Green arrived for an appointment with a urologist, she was asked if it would be okay if artificial intelligence transcribed their conversation. She said yes. But that decision came to haunt her as the AI software made up a false claim that Rebecca takes psychedelic drugs. She only realised the mistake weeks later when she read a letter her specialist sent her GP, which stated that Rebecca micro-doses mushrooms.Rebecca Green, patient: I was in tears. What is going on here? You can't be doing this kind of thing.Paige Cockburn: Rebecca was particularly stressed as she's receiving workers' compensation, and she worried that any mention of illegal drugs on her medical record could impact her case. After she made a complaint, the urologist sent a letter apologising and said the error appeared to be linked to the dictation or transcription process.Rebecca Green, patient: I love AI, but really is there ever going to be something that's better than human ears and eyes?Paige Cockburn: A recent report by privacy advocates Digital Rights Watch warned inaccuracies like the one affecting Rebecca Green can lead to serious patient harm. The research found examples where an AI scribe had recorded the wrong breast in a diagnosis of breast cancer, or said someone had epilepsy when they did not.Elizabeth Deveny, CEO of the Consumers Health Forum of Australia: So there are lots of people reporting errors where the scribe has misunderstood something.Paige Cockburn: That's Elizabeth Deveny, CEO of the Consumers Health Forum of Australia.Elizabeth Deveny, CEO of the Consumers Health Forum of Australia: And they only pick it up sometimes by chance because they see correspondence, something goes into their My Health record, or when they see another doctor they're asked a question and they go, that's not correct.Paige Cockburn: She's worried doctors may become too trusting of AI and she wants the software to have built-in checks which mean a transcript can't be uploaded until a human has reviewed it.Elizabeth Deveny, CEO of the Consumers Health Forum of Australia: Part of it sits on the designers too, the people selling this stuff. Are they making sure it's safe?Paige Cockburn: Perth GP Sean Stevens says AI scribes can get confused, but he feels they are becoming more accurate.Sean Stevens, Perth GP: When I first started using the scribes in 2023, I was having to correct multiple things on each output, whereas these days it's probably every third or fourth consult I'd have to correct something.Paige Cockburn: Dr Stevens is the Royal Australian College of GPs Chair of Digital Health and Innovation and also an advisor to a major AI scribe. He says having a scribe allows him to run on time more often, but he understands some patients don't like them.Sean Stevens, Perth GP: If a patient expresses concern or ambivalence, it's very important that the doctor respects that because the trust in a doctor-patient relationship is really pivotal.Paige Cockburn: There are dozens of medical scribes available in Australia, but none are registered with the Therapeutic Goods Administration. That's because it would only regulate those that suggest treatments or diagnoses. Tools that just transcribe or summarise are exempt from regulation. Dr Stevens says this is a fair way to make sure innovation isn't stifled, but Dr Elizabeth Deveny isn't so sure.Elizabeth Deveny, CEO of the Consumers Health Forum of Australia: When a person talks about things and it is summarising that because that's what it's doing, it is going to be making some decisions about what to keep and not keep in a record. So while that might not be a diagnosis, it is leading in that direction.Paige Cockburn: The Therapeutic Goods Administration is due to hand down a review into AI scribes in coming weeks.