It is widely appreciated that artists take inspiration from their own life experiences. Having accessed group sessions at the eating disorder clinic at the Maudsley Hospital in 2015, I was compelled to produce a new series reflecting upon my journey.

‘Emerging’ was my first self-portrait series and is special as it allowed me to see my own body objectively as form within a composition in a more self-compassionate light, something that has become increasingly difficult for women today.

I have always felt very lucky to be able to communicate through my work. In my experience, creativity is great at facilitating dialogue. Through this work I wanted to tell my story in my way, with the aspiration that others who share similar experiences may take comfort in sharing and that it may help towards breaking down the stigma surrounding mental illness.

Image: 'Scrunch Face' Collage by Sarah Carpenter

I have been thinking about the added value and complex ethical issues around documenting the patient voice; using acoustic cues in conveying emotion and attitudes in speech, and if this has the potential to become quantifiable data for use within CRIS.

I did a little research into this and came across Nicholas (Nick) Cummins, a lecturer in AI for speech analysis for health at the Department of Biostatistics and Health Informatics at King’s College London.

Nick’s current research interests include speech processing, affective computing and multi-sensory signal analysis. He is fascinated by the application of machine learning techniques to improve our understanding of different health conditions and mental health disorders in particular. Nick is actively involved in RADAR-CNS project which you can find out more about here:

Nick brought a lot of insight, interesting perspective and his work is fascinating. Unfortunately, technology was not on our side, so the recording was unclear. Nick kindly agree to document our conversation in note form from questions that I felt summarised the topics that we had covered. This in itself is interesting to me as it begs the question, how reliable would the technology for recording patient narrative be? where might problems arise and what backup procedures would need to be put in place?

Here is our documentation of the conversation:

Q1. What benefits are there to recording the patients/ clinician interaction?

• reflect upon clinician and patient exchange to improve upon the work that the therapists is doing. Construction of sentences etc
• determining if someone is in a positive or negative mindset from their speech (what language is used as well as how it is used) = Tracking for possible patterns / Spotting possible trajectory of illness and course of treatment needed- personalised treatments
• Monitor the quality of the dyadic exchange, patient health information could reflected in this, it could also be helpful training material for clinician
• Monitoring for positive/negative mindset of patients – help clinician guide the therapy session
• Non-verbal communication cues can be tracked and use to aid diagnosis/treatment planning

Q2. You spoke about the problem of linguistics within your work, please could you tell me more about this?

Might not use the term problem, but an issue I have observed is linguistics analysis is often based on manual-transcripts of therapy sessions. This is not sustainable in the longer-term if patients/ clinicians want more real-time feedback. It also mean linguistic analysis is not a subject to noise, and should be considered an upper-bound on the accuracy of such AI systems. If an automatic speech recognition system was used instead of humans, there would be more errors inputted into an AI system and the accuracy would be reduced. IMHO it is better to be working on fully automated pipelines and learning how to make such an approach work as well as one with humans in the chain.

Q3. Do you think there is a way to obtain information both from the content of what a person is saying and how they are saying something using AI? If so, how useful might it be to cross reference / combine these findings?

It could be very useful and this is some work I would like to do in the future. Not everything we say will contain salient health/mental health information, most of what we say could be consider very neutral. So yes, it could be useful to cross-reference between verbal and non-verbal cues to help each other out

Q4. What are the cons of recording the patients clinical interaction?

• Privacy: Recording speech is one level, but recording face and voice and then everything in the background is another level!
• Cost of long-term secure storage as well

Q5. What might AI capture that humans cannot?

• Body language (trying to attribute body language to mental / physical health). Clinicians can pick up on general movements, gestures and positioning, but subtle clues can be easy for us to miss as humans.
• Microsoft game systems (Kinect) which give an idea of skeleton of movement
• minutia of details could be picked up on by AI (Face as an example, having 88? muscles to monitor)
• Subtle cues in or verbal and non-verbal/ behavioural cues. It is hard for us to pay attention to all of this, but AI can

Q6. Are there universal non verbal cues across cultures? If so, could you give an example?

This is a really interesting questions which is really hard to answer. It has been proposed that certain facial expression are universal, but this is not a commonly held view. It maybe possible for AI to learn mappings between non-verbal cues of different cultures.

Q7. Where might using AI fall short when it comes to capturing this kind of data right now?
​

• AI not good at saying why it thinks something
• difficult to find a baseline for a person when it comes to paralinguistics? Changes when ill or healthy, Different accents, Our own identity in how we speak / what we say.
• Lack of robust explainability in AI. It is currently not possible to understand why complex AI systems reach certain decisions.

Q8. Would it be possible learn to fool a system that analyses interactions? (From our discussion, I started to think about how good I have been at hiding my state of mind / thoughts from people in in the past, might this be a problem when using AI?)

Of course, but many non-verbal cues are not consciously controlled, so hiding might be difficult.

Q9. How else might we document the patient narrative using AI outside of the hospital setting? And what might be the pros and cons of this?

• Using a phone / watch etc in order to gather longitudinal data and build a more fully rounded picture of the individual, their experience and their life.
• Requires little input / less participation from service user
• With so much time between appointments, remembering what has happened and how you have felt is difficult and huge parts of my story might be being missed. When asked to record food, sleep, exercise, thoughts, events etc (basically everything), I spend so long documenting, that I don’t have time for living, plus I spend so much time focusing on the negatives which worsens my illness.
• help with tracking health/mood states, this would mean less recall biases when informing a clinician about recent events. This is a key step toward more personalised health care

Q10.What are you working on at present?

I have a new dataset of speech samples collected through the RADAR-CNS project. This data is the first time I have had longitudinal speech sample from individuals with depression (data collected from multiple time points). This gives me a chance to understand how speech changes with fluctuations in mental health overtime.

Q11. What’s the next steps for you in terms of your research?

Applying for grants to collect more data to learn more about the effects of physical and mental health on speech

Fellow artist Sue Morgan gives us her thoughts on the project, specifically the the idea of recording the patient narrative in a clinical setting.
​

Sue and I had a fruitful discussion about how she might communicate her thoughts. 

Most of our dialogue was through a voice memo exchange which was a first for me.  I found it a really lovely way to communicate - listening to a voice rather than reading it is so friendly and personal.  This method also gave me the opportunity for reflection before responding, helping me to practice thinking before I speak. It was also really practical as you can record whenever you have a moment, and in whatever environment you choose - walking in nature helps to inspire thoughts for sure!

So, Sue decided upon this beautiful diagram with explanatory film to express herself: an artwork in its own right as far as I am concerned! 
​

A huge thank you to Sue for investing so much of her thoughts and time in this - always great to have peer to peer reflection time with those who's input you respect and who's work you hold highly. 

• ​At its core, the alphabet is a series of symbols. 
• Breaking down fonts into their essence - a series of shapes / forms.
• Lifting text off of the page, turning letters into sculpture.
• Is it text, or art or both? Are they one and the same?
• Ambiguity of a statement - Open to interpretation
• Typography: arranging text to be clear and legible - breaking these rules to turn language into something "other".
• Traditional Printmaking: Screen printing - something I love to explore in my creative practice.
• Reframing meaning of language by changing the context

Here are some of my favourite examples:
​

Image Credit: Beth Hopkins. 

What form of communication makes you feel
​best understood?
​

One persons experience of having her narrative lost in translation : 

What is your experience? 

​Today Anna was welcomed into  our weekly drop in.

Some of the interesting themes that arose from the discussion were: 

- Phobia of data (the narrative surrounding this) 
- The title: Electronic Patient Journey system
​(great name indeed!) 
- The parallels with deconstructing and reconstruction in my art practice and with the data (a process of learning for both the machine and I?) 
- Asking questions in multiple ways in order to be more inclusive. 
- The idea of the value model - looking at pros vs cons of documenting narrative in different ways
- Being given the option in the moment to document your own story. 
- Editorial / shorthand/ note taking 
- Offering confirmation that you have understood what someone tells you (a classic teaching technique and also used in metacognition testing) “so what I hear is..... is that correct?” 

Image: Sarah Carpenter darkroom experimentation
​

How can you possibly look at storytelling without thinking about listening? 

One message that stands out to me in this podcast is that we can often be solution driven rather than actually listening.

Is this what is happening in clinical settings? Is that why our narrative is being lost?

Imagine clinicians having the time to listen rather than record! AI could play a part in that.

As part of an interview process, I was asked to chose an object that represented power to me. I chose a blank sheet of paper. 

I was given 2min to discuss my choice: 
​

​​Deconstructing and reconstructing my records.
​Creating a fresh blank page ready to tell my own story in my own way.

The story so far...
​

The most interesting part of this discussion in relation to this project, was listening to Dr Emilia Molimpakis talk about neuroscience, linguistics and her video gaming system called Thymia.

Probably best to just show the 3min video of her explaining it much better than I could in a full blog post (the power of creative/ video storytelling!) ...
​

A really interesting talk thinking about AI vs. Humans and computers replicating and replacing human processes.

In research, the outcome is often measured as  human performance vs. AI, so essentially can it be proven that AI is as good or better than humans?

Dyer demonstrated that the combination of AI plus human performance always gives better results than either AI or humans can produce alone. The question is how can we work with AI to improve existing processes and complement human skills?

Dyer spoke of the narrative of AI in the media: The reoccurring theme of robots being a threat to human beings, they will take our jobs and eventually threaten our very existence by replacing us. Robots are seen as a  threat to human beings - we are portrayed as being weaker. 

​Dyer framed things differently, he spoke of AI cooperating with humans, complimenting our skills, being used as a second opinion / cross reference. 

In terms of diagnosis, AI is easily trained to determine severity of symptoms, and easily and accurately decide if a patient needs either immediate treatment, or no treatment at all. This means that the workload of a clinician can be significantly decreased and more time can be spent on more complex cases that need further analysis. 

​The crucial part of the AI puzzle which is yet to be deciphered, is if the machine sees something unfamiliar. Instead of it saying "I'm not sure", it might automatically place something in its nearest  category. These are the instances where we see how vital the clinicians role is and how irreplaceable they are.  

Imagine though, in a clinical setting, AI taking on the  tasks that involve memorising and recalling information (basically what we would define as the  'knowledge" involved), this would mean that clinicians would have more time to engage other important skills such as compassion, empathy and understanding to make a fuller, more rounded diagnosis. 

This resonated with me and my thoughts on our project. Could this lead to more of the patient narrative being heard and given value in the clinical setting? By allowing better resources for a clinician to utalise their human skills (e.g. compassion), richer and more valuable information might be  drawn form from the patient, helping the clinician to make a better assessment. At the same time AI could capture the patient narrative and analyse it  using its knowledge. Together, this would provide a more rounded picture of the patient. 

My mental wellbeing is severely effected by feeling misunderstood. Simply feeling heard is in itself medicine for me. ​

​Acrylic, Collage and Charcoal on cardboard
​

• Text
• Language vs visual grammar
• Nonverbal communication
• Narrative/ story = identity
• The other side of the story
• Murky/ wishy washy language:

         nothing can quite explain how I feel

• Look and see, don’t ask me to tell you in words
• Academic language / accessibility for everyone

​

​Thought 1: 

Until this talk, I hadn't made a direct correlation between the brain and computers. Artificial neural networks as computational methods to replace the brain so in essence, neurons = nodes. ​

​Thought 2: 

I was fascinated by the work that Pearse is doing that relies on image recognition;  computers are fed lots and lots of images, training them to recognise certain things. I recently came across the app  Google Quick Draw, it prompts you to draw certain objects and then guesses what you've drawn. It stores the information from each new drawing, uses it to learn and gets better at recognising - the more people interact, the better it becomes. 

​In fact, rather than explain, why not just let you see for yourselves :

Thought 4: 

During the talk I was introduced to the term "AI chasm". This is the difficult gap between imagining a prototype and then actually putting it into action in a clinical setting once it has approval. We're talking years (on average around 8!) How can this be navigated? By the time useful technology is developed and in clinical settings, it's already outdated. ​

Thought 5:

When it comes to coding, Pearse had an opposing view to the last talk, believing that experience in this area is not necessary. I've always felt I should learn to code in order to  understand the process, however, I learned today that there are new "drag-and-drop" systems for anyone who wants to have a go at this kind of . This opens up a whole new world of possibilities and makes me feel that I might actually be able to put some of my creative research into action. I must delve deeper into this! TBC...

​​It seems so obvious, but the idea that this data is dynamic; it moves, and shifts and needs to be captured as a continuum, was something that I hadn't considered until now.  

Of course the patients data (and therefore narrative) will move, develop and change when just as any good story does.

​So how do we go about capturing this rich, ever evolving narrative?​