Image: 'Scrunch Face' Collage by Sarah Carpenter

I have been thinking about the added value and complex ethical issues around documenting the patient voice; using acoustic cues in conveying emotion and attitudes in speech, and if this has the potential to become quantifiable data for use within CRIS.

I did a little research into this and came across Nicholas (Nick) Cummins, a lecturer in AI for speech analysis for health at the Department of Biostatistics and Health Informatics at King’s College London.

Nick’s current research interests include speech processing, affective computing and multi-sensory signal analysis. He is fascinated by the application of machine learning techniques to improve our understanding of different health conditions and mental health disorders in particular. Nick is actively involved in RADAR-CNS project which you can find out more about here:

Nick brought a lot of insight, interesting perspective and his work is fascinating. Unfortunately, technology was not on our side, so the recording was unclear. Nick kindly agree to document our conversation in note form from questions that I felt summarised the topics that we had covered. This in itself is interesting to me as it begs the question, how reliable would the technology for recording patient narrative be? where might problems arise and what backup procedures would need to be put in place?

Here is our documentation of the conversation:

Q1. What benefits are there to recording the patients/ clinician interaction?

• reflect upon clinician and patient exchange to improve upon the work that the therapists is doing. Construction of sentences etc
• determining if someone is in a positive or negative mindset from their speech (what language is used as well as how it is used) = Tracking for possible patterns / Spotting possible trajectory of illness and course of treatment needed- personalised treatments
• Monitor the quality of the dyadic exchange, patient health information could reflected in this, it could also be helpful training material for clinician
• Monitoring for positive/negative mindset of patients – help clinician guide the therapy session
• Non-verbal communication cues can be tracked and use to aid diagnosis/treatment planning

Q2. You spoke about the problem of linguistics within your work, please could you tell me more about this?

Might not use the term problem, but an issue I have observed is linguistics analysis is often based on manual-transcripts of therapy sessions. This is not sustainable in the longer-term if patients/ clinicians want more real-time feedback. It also mean linguistic analysis is not a subject to noise, and should be considered an upper-bound on the accuracy of such AI systems. If an automatic speech recognition system was used instead of humans, there would be more errors inputted into an AI system and the accuracy would be reduced. IMHO it is better to be working on fully automated pipelines and learning how to make such an approach work as well as one with humans in the chain.

Q3. Do you think there is a way to obtain information both from the content of what a person is saying and how they are saying something using AI? If so, how useful might it be to cross reference / combine these findings?

It could be very useful and this is some work I would like to do in the future. Not everything we say will contain salient health/mental health information, most of what we say could be consider very neutral. So yes, it could be useful to cross-reference between verbal and non-verbal cues to help each other out

Q4. What are the cons of recording the patients clinical interaction?

• Privacy: Recording speech is one level, but recording face and voice and then everything in the background is another level!
• Cost of long-term secure storage as well

Q5. What might AI capture that humans cannot?

• Body language (trying to attribute body language to mental / physical health). Clinicians can pick up on general movements, gestures and positioning, but subtle clues can be easy for us to miss as humans.
• Microsoft game systems (Kinect) which give an idea of skeleton of movement
• minutia of details could be picked up on by AI (Face as an example, having 88? muscles to monitor)
• Subtle cues in or verbal and non-verbal/ behavioural cues. It is hard for us to pay attention to all of this, but AI can

Q6. Are there universal non verbal cues across cultures? If so, could you give an example?

This is a really interesting questions which is really hard to answer. It has been proposed that certain facial expression are universal, but this is not a commonly held view. It maybe possible for AI to learn mappings between non-verbal cues of different cultures.

Q7. Where might using AI fall short when it comes to capturing this kind of data right now?
​

• AI not good at saying why it thinks something
• difficult to find a baseline for a person when it comes to paralinguistics? Changes when ill or healthy, Different accents, Our own identity in how we speak / what we say.
• Lack of robust explainability in AI. It is currently not possible to understand why complex AI systems reach certain decisions.

Q8. Would it be possible learn to fool a system that analyses interactions? (From our discussion, I started to think about how good I have been at hiding my state of mind / thoughts from people in in the past, might this be a problem when using AI?)

Of course, but many non-verbal cues are not consciously controlled, so hiding might be difficult.

Q9. How else might we document the patient narrative using AI outside of the hospital setting? And what might be the pros and cons of this?

• Using a phone / watch etc in order to gather longitudinal data and build a more fully rounded picture of the individual, their experience and their life.
• Requires little input / less participation from service user
• With so much time between appointments, remembering what has happened and how you have felt is difficult and huge parts of my story might be being missed. When asked to record food, sleep, exercise, thoughts, events etc (basically everything), I spend so long documenting, that I don’t have time for living, plus I spend so much time focusing on the negatives which worsens my illness.
• help with tracking health/mood states, this would mean less recall biases when informing a clinician about recent events. This is a key step toward more personalised health care

Q10.What are you working on at present?

I have a new dataset of speech samples collected through the RADAR-CNS project. This data is the first time I have had longitudinal speech sample from individuals with depression (data collected from multiple time points). This gives me a chance to understand how speech changes with fluctuations in mental health overtime.

Q11. What’s the next steps for you in terms of your research?

Applying for grants to collect more data to learn more about the effects of physical and mental health on speech

Image Credit: Beth Hopkins. 

One persons experience of having her narrative lost in translation : 

What is your experience? 

​​Deconstructing and reconstructing my records.
​Creating a fresh blank page ready to tell my own story in my own way.

The story so far...
​