Well I
often moan about the state of statistical communications and point out that
people, although not wantonly misleading people, are at the very least implying
a different meaning than might be appropriate. After the last post I thought
I’d delve a little deeper into why I felt it was a bit daft in my opinion to
assess the effectiveness of the IAPA (improving access to talking therapies) in
this manner. As I pointed out it is a little bit involved.(full paper here)
First off
the figures they quoted as using for
their ‘effectiveness value’ were calculated in this instance with this aim
A more robust and transparent measure of the intervention’s
effectiveness may entail use of the larger denominator of all those who entered
therapy rather than only those who completed it.
patients categorised as ‘moving to
recovery’ (KPI 6a) as a proportion of those who
entered psychological therapy (KPI
4).
This will include those who left without completing the
therapy (although there is no clear KPI which is defined as “leaving the
programme during the quarter” within the IAPT dataset). ‘Entered psychological
therapies’ is defined as:
‘attending first therapeutic session,
which may be during the same appointment as initial assessment.’
Please note not
in format in the paper found here and that the notations kpi are different
depending on the dataset collection data you can now online here.
Which is basically a percentage of all people who were seen
according to the questionnaires to not need therapy at the end of their
treatment over all those who came to the service.
Understanding the system.
People are referred either by themselves or from a GP or
other medical group/service to talking therapies. They make their case for why
they need it and are then offered an assessment appointment for ‘therapy.’
There should be more than one type of therapy on offer which is why they need
an assessment to decide which one is most or best suited to them. They will
have been sent/given an assessment questionnaire to fill in and bring with them
to their appointment to assess ‘caseness’ (baseline indicator questionnaire on
wellness).
This assessment period can take more than one appointment
depending on the person and what type of therapy they are being offered. If
they agree to therapy they will then be referred to the person who will conduct
the therapy, which may not be the person who assessed them. And then they will
start their therapy.
At the end of their
therapy they will cease appointments and depending on the provider they will be
sent/given another questionnaire to complete and send back. At any point in
this process a person can choose to not return to the service or therapy be
refused.
Now it seems to me if you are going to look at effectiveness
of a service you need to consider the difficulties of each part of the process
and although ball parks can give a quick overall assessment they really don’t
help understand the problem.
In this instance the assessment period seems quite different
to the overall effectiveness of giving treatment. So why assess the overall
process when splitting it up would make more sense?
For my mind I would split it into how effective treatment is
when adherence is good, how good is adherence and how effective is the
assessment process in getting people to adhere to treatment.
Effectiveness of treatment.
I think there are several good reasons not to do it the way
these authors have. If you look at the data for IAPTA
you will see one of the datasheet contains
Number of service requests with completed treatment where
the service user was not at caseness at initial assessment
i.e. not at the level at which they deemed treatment
necessary.
Total number for England 6416 people treated.
Oh dear it looks like the NHS assessed and treated 6416 when
there was no apparent need.
So why were they treated?
Well they were treated because after a face to face
assessment these people where seen to not only need but would benefit from
having therapy. Now you could argue two things here, either the assessment questionnaire
does not give a good overall picture of ill health and should be changed for
something else or you should exclude
these people from the overall effectiveness figures because they could not
improve under these conditions and therefore would always return a negative to
the statistics ie reduce the % seen as recovering.
Secondly adherence to treatment is important and the more
people attend the sessions the more improvement they see. ie the more effective
the treatment is.
Stands to reason right, so currently completion of treatment
is to have undertaken more than 3 sessions of therapy but most courses of
therapy are at least 15 sessions if not 20-25 and in some cases of group
therapy upwards of 60 sessions. Well I think anyone can see that attending 3
sessions of a course that lasts more than 60 sessions is not going to produce a
good indicator of whether that therapy helped. Even 3 sessions of a 15 is
pretty bad (20% attendance) A more realistic outlook would come from putting
completion of therapy as at least 50% percentage of the overall number of
session usually given for that therapy. If people are only going to attend a
very small percentage of sessions how can they have completed therapy in any
meaningful way and therefore should be excluded from the recovery rates %. Currently
there is no published information on this but maybe there should be.
Adherence to treatment.
This is people who start but don’t finish and in my opinion
should be a separate figure as this is about the services ability to assess and
allocate people to the right kind of therapy. For this calculation I would
exclude all those people who were assessed and sign posted to other services
within psychological services as these people are in the wrong place and this is
not a problem of effectiveness of this service but of the referral process (see
later).
Assessment process
What you really want to assess is how effective are the
staff at getting people to the right therapy. ie that they adhere to their
treatment and improve. And I’ve used the word improve deliberately because it
is not always that easy to see improvement. So you want to know how many people
come into your service and adhere to treatment of those who are suitable for
it. Now I would exclude people who are referred to other services and those who
choose not to take the therapy for other reasons. And I would also change the
way recovery is counted for instance, did the score on the questionnaire go
down by a certain amount. Not, did they not meet caseness but was it less than
the previous one as people coming in will vary in the degree to the problems
they have and some may score a lot higher than others.
Referral process
The biggest problem I have is in working out whether the
right people are being referred to the service. Is there any way to pre-select people
based on a telephone call or whether an assessment is necessary for all. I feel
there should be a way to assess whether reselection is working but the only
thing I can think of is how many people get referred to other services. However
I am not sure that is a good indicator as this is a self referral scheme and
there may be many people coming to the service who are on the cusp of problems.
Hence the figure from earlier of people not making caseness and therefore not
seemingly ill although I fear that is more a limitation of the questionnaire rather
than the people not needing help.
So after all that do I think any of it could be implemented?
Well some of it already is but I’m not sure it’s practical to do the rest
nationally. Maybe on a local PCT level yes but nationally I think they’d
struggle to get all the returns in.
And I certainly don’t think it appropriate to lump all the
different parts together in the way that those authors did and if those figures
were used in the papers it would undermine a service without just cause.
If there’s just cause then feel free but that assessment to
my mind doesn’t really support it to be fair it doesn't really support much at all and neither does what they are doing.
No comments:
Post a Comment