Creating a culture of analysis and evaluation in local government

One of the phrases I’ve taken to heart recently has been “turning ‘I think’ into ‘I know’”. I saw it on some slides in our recent webinar about analytics and A/B testing, where we talk about discovering through unbiased means which messages actually work with which audiences; it struck a chord with me for some reason and it’s taken me a while to figure it out.

You see, for years I’ve heard people, many of whom are widely regarded as experts in their fields, telling others how to do things. Whether in small meeting rooms or in front of huge audiences, they stand up and say “I know how to do this thing, so do it like me because it works”. And people do.

Only, none of them ask for much in the way of proof. The closest to evidence most ask for is a case study, but in reality this simply tells the reader how things worked in one situation in one area with one particular set of circumstances. Other than that, the evidence that an approach works is more often than not scant, with people’s willingness to believe and replicate it based more than it should be on the charisma and/or reputation of the person presenting.

That’s not to say, of course, that a certain approach will not work elsewhere, only that there is no guarantee that it will. With the added risk of feeling like you are following a fad, or will be left behind if everyone else does it and you don’t, it’s no wonder then that decisions to proceed with major projects or strategic directions are based far more on personal opinion than on verifiable facts.

Local government is not scientific in that regard. Yes, there may be a few examples of deep, long-term research which has guided future policy changes, but if there are any they are few and far between. This is of course in no small part down to the fact that they are inherently political institutions, and that they cannot afford to experiment when people’s lives, homes and health are at stake.

However, there is a lack of scientific rigour to back up far too many major decisions which is alarming at the very least.

Perhaps things are swinging the way of data however. With the modern day ability to collate huge datasets and a drive across most organisations to publish data more freely and openly will over time allow armchair researchers to collate and interrogate data more scientifically than ever before. Incidentally, if you happen to run a Drupal-based website, do get in touch to find out about our ace DKAN Open Data platform.

This situation is part of a similar journey to the one taken by medicine over the centuries. What started out as trial and error then morphed into what turned out to be generally accepted practices that worked more often than not. Each could be slightly improved as new techniques and drugs became available, but the underlying thought processes remained the same.

While most of the time this was correct, in too many cases it later turned out that widely held beliefs that doctors swore by and based prescriptions on were actually not always true. Often these beliefs stemmed from very small datasets without much by way of independent data checking or control groups, meaning it was at best misleading. Treatments which worked well with the early sample groups therefore became the default, despite the fact there could be vast differences between different groups and situations. They also rarely controlled for the placebo effect, so drugs were issued as normal procedure when doing nothing may have been just as effective.

These days more demanding trials are required and results are double and triple checked before being released as “fact”. Even then, there remains a healthy dose of scepticism which prevents things being described as wonder drugs based on little more than the opinions of the person who created them.

Perhaps more of this approach is needed in local government. Collecting data and then accounting for differences in deprivation, population, industry and more would be a challenge, but on a small scale would not be impossible. Understanding what works based on empirical data would surely be preferable over pinning hopes on something because it simply feels like the right thing to do to the decision maker.

This does, however, require an acceptance of failure that simply doesn’t exist within the public sector. Acceptance in this case does not mean being happy with it, but simply accepting that sometimes things do go wrong despite the best of plans, intentions and endeavours. Where things do go wrong lessons need to be learned and future plans tweaked accordingly, rather than vilifying those involved and not letting them near any project of import in future. Science doesn’t always get it right, but it knows when it gets it wrong and works to understand why so it can evolve.

It also requires leaders to stand up and proudly say that they don’t, in fact, have all the answers. This is a big step, not to be underestimated; people often feel they are promoted based on them knowing how to get things done, when in fact they are promoted because they know how to go about getting things done.

It’s a subtle difference, but an important one. The former implies that they themselves know what to do and how to make something a success so just need to get on and do it. It won’t fail; they know what they are doing after all. If it does fail, it’s probably down to things beyond their control.

The latter, however, indicates that they know how to go through the right processes to deliver on something and understand whether it did actually work. If it did work, why? If it didn’t, why not? They can then lead their teams to refine and try again with some tweaks, small or large. Every time it will get a little better and give them more evidence on which to make future decisions on policy and projects.

It took Norman Larsen 39 wrong attempts before he invented WD-40. Edison tried over 1,000 different designs and materials before he perfected the lightbulb. Why do we think local government will always get it right first time, with no mistakes?

Let’s collect, collate and analyse. Let’s take a scientific approach to our planning. Let’s turn ‘I think’ into ‘I know’.