Cost benefit analysis

If 100 trainees use Clinitalk and one 6 month extension prevented there is a saving of £12, 199.

A diverse group of GP trainees.

Is it effective?

An impression of efficacy can be obtained by triangulating 8 areas of evidence.

Area 1
User Engagement

Do users engage with the intervention? ✅


5514
consultations were analysed by trainees in the first 9 months of release

Area 2
Alignment with theory

Does the intervention align with educational theory? ✅

key educational theories and evidence are built into the design.

Area 3
Feedback Accuracy

Does evaluation of feedback show the intervention is accurate? ✅

“The AI feedback was impressive and closely matched real feedback." (Dr SU Trainer).

97.6% accuracy in evaluation.

Area 4
User reported benefit

Do users report benefit? ✅

"It has greatly aided my understanding of the consultation and highlighted my strengths and areas for improvement." Dr JC

Area 5
Educator reported benefit

Do educators report benefit? ✅

“It’s a powerful tool that has improved my trainee's structure and consultation speed.” (Dr SC Trainer)

Area 6
Feedback continuity

Does the intervention offer continuous support? ✅

“The AI feedback was impressive and closely matched real feedback." (Dr SU Trainer).

97.6% accuracy in evaluation.

Area 7
Improvement in outcomes

Have users reported an improvement in outcomes?

“It helped me improve by ability to pick up on  ICE, recognise psychosocial issues, understand impact and align my management to NICE guidelines”, Dr CC

Area 8
Targets areas of need

Is the intervention targeted to address differential attainment?

To support groups at risk of differential attainment, the app gathers data on user characteristics and delivers targeted messaging.

Cost equivalence

140 annual Clinitalk licences cost less than one 6 month extension of training.

An image of a stethoscope and a £50 pound note with an NHS logo.

A deep dive

If you want to take a deeper dive into the evidence this next section will help. It includes links to evidence, quotes from users and links to video testimonies.

Alignment

Clinitalk's feedback is evidence-based, utilising key educational theories, NICE guidance and seminal papers as its sources to ensure it is equipping users with reliable and validated guidance.

Integrated theories include: Interleaving, Constructivism, Meta cognition, Experiential learning, Hofstede’s cultural dimensions. Integrated papers include: Linguistic and cultural factors in the MRCGP, RCGP chief examiner reports and papers on differential attainment.

There is growing evidence supporting the effectiveness of Technology Enhanced Learning (TEL) in GP training. The improved accessibility and flexibility that TEL provides allows trainees to learn at their own pace and revisit materials when necessary, accommodating the schedules of busy and diverse learners.

Engagement

Over the first 9 months of release, users have recorded and analysed 5514 consultations.

Evaluation

Out of 290 feedback statements, 7 errors were found in traffic light colour ratings, with no significant errors in textual feedback. Users and educators praised its high accuracy:

 Dr GP: "Very impressed with its accuracy"
Dr KMC: "Highly impressive program"
Dr RG: "Really good"
Dr HA (trainer): "Impressed with the software and comprehensive feedback"
Dr AM (trainer): "Surprisingly good feedback on interpersonal skills"
Dr HK (trainer): "I'm impressed"
Prof AK (lead examiner): “It’s really very impressive”

User Benefit

User testimonials, user feed back scores (9.3/10), and independently collected feedback via a third party evaluation have all been positive.

Dr JC: "It has greatly aided my understanding of the consultation stages and highlighted my strengths and areas for improvement."
Dr IM: : "It is transformative, although occasionally the AI can be quite stringent."
Dr LC: "My previous SCA score was 66 before using the app and increased to 89 after a few months of use and attending the course."
Dr SU: "The AI feedback was impressive and closely matched real feedback."

Video testimonials from users

Educator benefit

Educators report benefits for their trainees:

Dr AME (trainer): "Excellent, I think the notes and feedback sections are really helpful."
Dr SC (trainer): "It’s a powerful tool that has improved my trainee's structure and consultation speed."
Dr HA (trainer):"It encourages curiosity and helps move away from the medical model."
Dr SS (trainer):"It gives them confidence and helps them take responsibility for their own improvement and consulting techniques."
former RCGp examiner and area director Nicky turner talks about her experience

Continuous support

When compared to one-off interventions like traditional courses, continuous support has been demonstrated to better foster sustained improvements in learners competency, confidence, and critical thinking.

Educational theories emphasise the importance of regular, formative feedback such as that provided by Clinitalk.
Reference:
A Review of Feedback Models and Theories, Front Edu, 2021

Improved outcomes

Whilst improvement in outcomes is multifactorial. Users andeducators have reported an improvement in outcomes they have attributed toengagement with Clinitalk.

Dr CC: “It helped me improve by ability to pick up on  ICE, recognise psychosocial issues, understand impact and align my management to NICE guidelines” IMG
Dr LC: "My previous SCA score was 66 before using the app and increased to 89 after a few months of use and attending the course." IMG
Dr ME: “It helped me train and learn at the same time; it helped be become more patient centred. I will continue to use it after my exam” IMG
Dr OA: “Clinitalk showed me how I was using my time and helped me toimprove, it helped me see how I was consulting, I would recommend it to anyone”
Dr KV: “It helped me improve my patient doctor talking time balance for the whole process which helped give the consultation a more natural feel, had better outcomes and was reflected in my trainer feedback, I honestly believe that Clinitalk helped me pass the SCA on my first attempt’ IMG
Dr KM: “I used Clinitalk to refine and improve my consultations, the traffic light summaries are very useful, and you are learning and analysing and not losing out on time while you are in a busy clinic. It provides you that feedback you normally get with your ES but it just means that you can get so much more feedback, It really did help me when preparing for SCA and just generally’
Dr SS: (Trainer) “It is particularly interesting how different trainees use different training features to meet their training needs
Dr SC: (Trainer) “I think it is a powerful tool and has helped my trainee see several issues in their consulting”

Targeted intervention

: Clinitalk focuses on consulting areas most linked to exam failure, as well as the groups (IMG, neurodiverse) most susceptible to such failures. The types of information fed back to users has been selected to be the most useful for development, based on a survey of RCGP examiners and trainers. Trainers & examiners ranked consulting behaviours to identify those critical for success. To support groups at risk of differential attainment, the app gathers data on user characteristics and delivers targeted messaging towards at risk groups. The app can recognise cases that groups of users have poorer outcomes in and highlight the importance of the learning opportunity to the user to enhance engagement and understanding.  

Chat