Regeneration & Renewal has now published 150 Evaluation Lessons – our weekly reports examining renewal projects. They reveal that the sector keeps on making the same mistakes, says Herpreet Kaur Grewal.
In the 1993 film Groundhog Day, Bill Murray’s character finds himself reliving the same day over and over again. Until he changes his behaviour and becomes a better person, he is condemned to endlessly repeat the same experiences.
Things aren’t so different in the world of regeneration. For more than three years, Regeneration & Renewal has published a weekly column – Evaluation Lessons (see p22) – in which a regeneration practitioner explains the findings of an evaluation report examining a regeneration project. We’ve now reviewed all of these columns – over 150 of them – and have found that the same criticisms are being made again and again. The sector is, it seems, locked into a depressing cycle whereby it constantly falls foul of the same set of classic errors.
Evaluation Lessons has covered endeavours as small as a local community cafe scheme, and as extensive as government housing policy over 25 years. Despite that diversity, a fifth of the reports criticised schemes for failing to engage with and meet service users’ needs, and about the same number attacked projects’ monitoring and evaluation. About ten per cent of all schemes failed to share budgets and information in order to foster effective partnerships, and ten per cent suffered because short-term funding distracted from delivery.
Perhaps the frequency of certain criticisms isn’t surprising: there are structural problems with policy-making and delivery within the sector, and it isn’t easy to learn from your mistakes and change your ways. But until regeneration practitioners stop repeating classic errors, money and time will continue to be wasted and opportunities missed. For Bill Murray, the prize of changing his ways was a happy union with Andie MacDowell; for deprived communities, the rewards could be greater still.
Engagement & delivery
Some 39 evaluation reports criticised projects for failing to engage with, market services to, or meet the needs of clients. About 10 per cent of all projects were found to have failed to communicate and engage with service users, and the same proportion to have failed to meet their needs. Here, the latter problem may be caused by the former: it’s hard to meet a group’s needs if you’re not talking to them – a problem worsened by the fact that, according to these evaluations, a further five per cent of projects failed to market services adequately.
Jonathan France, senior consultant at consultancy Ecotec, says consulting and engaging with people is an expensive process, and one that must be tailored to local communities. It is “easy to criticise” projects for falling down on consultation, he says, “but it’s not an easy exercise, particularly if there’s no history of resident involvement.” Nonetheless, he suggests “regeneration professionals could be more creative when trying to reach out, and do more to tap into information that local development trusts and housing associations may hold about residents.”
Such information can be used to bolster more traditional consultation methods, which Tim Horton, research director at left-wing think-tank the Fabian Society, says often involve “meaningless or thin ways of engaging” with people. “Sometimes tokenistic surveys are carried out. I doubt the tools used are always sophisticated enough to assess what users’ needs are,” he says.
Neil McInroy, chief executive of think-tank the Centre for Local Economic Strategies, says effective consultation is an incremental process that must be consistent “to build up trust” with potential service users. One way to form longer-term relationships with local people is to give them feedback after a consultation, adds France; while reporting back with these results, practitioners can encourage people to get more involved in projects’ design and delivery.
The evaluation reports also contain some suggestions for tackling the problems they identify, including better support for regeneration agencies and service providers trying to consult with people, and the need to ensure that projects have a clear strategy for consulting all sides of the local community.
Monitoring & evaluation
One fifth of all our reports identified failures in monitoring and evaluation, half of those finding that no effective impact measurement system had been set up at the project’s outset. Ten per cent of reports criticised inadequate collection of baseline data and statistics during the project, while a few found that “softer”, harder-to-measure results – such as increases in individuals’ confidence or life prospects – were not identified. A couple of projects were criticised for concentrating on monitoring at the cost of effective delivery.
Such widespread failure to establish and use impact monitoring systems will cause real harm, both to projects unable to demonstrate their achievements and to the work of policymakers deprived of evidence about what works. Katharine Knox, research manager at the Joseph Rowntree Foundation, says: “It is important to monitor and evaluate properly, so that we know how to invest.” Without data on the costs and benefits of projects, she adds, it’s impossible to identify what provides “value for money in terms of public investment”.
When it comes to physical regeneration, says Stephen Hill, director of consultants C20 Futureplanners, time pressures often push agencies into devising masterplans without carrying out thorough research into an area’s problems. The need to establish social, environmental and economic baselines and build in monitoring and evaluation systems, he says, “seem less immediately important and there is always the false expectation that they can be bolted on later”.
McInroy says the problem is partly due to a lack of training, and a perception that monitoring is a complex task that can distract from tackling local people’s problems. But only two reports identified monitoring as a distraction from delivery; in the overwhelming majority of cases, criticisms revolved around the failure to establish effective evaluation systems.
Partnership working
Mistakes and failures in partnership working comprised the third category of criticisms, cited in nearly a fifth of reports. More than half of these projects were attacked for failing to share information or budgets with partner organisations, with the remainder exhibiting a range of errors. The most common of these was poor relations with local or national government, which evaluators said led to inadequate support or skewed objectives: clearly, this is a criticism aimed as much at councils and government departments as at the projects themselves. Indeed, some evaluators warned that the imposition of political objectives on regeneration schemes can damage their impact: for example, a 2007 report on urban regeneration company Sheffield One found that political pressures worked against its ability to create positive change in the long term.
Three schemes came under fire for failing to involve the private sector properly, one for an absence of decisive leadership, and two for failing to align their work with that of other regeneration projects. If practitioners were awarded points for talking about partnership working rather than actually undertaking it, they’d all receive glowing reports – but as our evaluation reports show, professionals are often happier to use the language of partnership than to loosen their control over their staff, budgets and information.
McInroy says partnership working is not an unalloyed good – it often “elongates conversations and delays things” – and suggests that the “standardised and generic” systems used by evaluators to measure the quality of partnership working don’t always take account of local circumstances. For example, a project that operated outside the direct control of its local authority might be marked down on this issue – even if its autonomy actually improved its work.
Funding & lifespan
One sixth (24) of our evaluation reports found that funding issues damaged the projects’ efficacy: in 14 studies evaluators concluded that short-term funding allocations and project workers’ need to seek additional funds distracted from project delivery. Two studies criticised projects for failing to plan for financial sustainability after initial funding ran out: evaluators recommended that schemes put in place follow-on funding arrangements early in the project cycle. They also noted that regeneration practitioners might in some cases have been able to secure more funding from the private sector.
So short-term funding is seen as a significant limiting factor on regeneration schemes’ ability to achieve their objectives, constraining managers’ freedom to make long-term plans or build the kind of well-established projects that maximise the involvement of and impact on local people. Several other reports concluded that resources were simply inadequate: four found that schemes hadn’t been given enough cash to achieve their objectives, and a further four said that they hadn’t been given enough time.
France is sympathetic to short-term funding complaints. He says it takes time and experience to make a sustainable impact: “that’s why local regeneration needs dedicated agencies – you need one body to plan long-term regeneration in an area.” The problems of poverty and dereliction often arise over a long period, and tackling them demands a similar sustained effort, says France.
Objectives & planning
Our last major category concerns strategic and daily management, on which 20 of our 150 reports found significant failures. Management issues are vital in deciding a project’s success – and clearly also contributed to almost all the other problems these reports identify.
Eight of the 20 studies criticised project managers for not establishing a clear strategic direction, defining organisations’ roles and/or developing business plans. A further four projects were attacked for failing to set clear targets, and three for failing to identify a defined target group – so around ten per cent of all those evaluated were found to exhibit serious flaws in their conceptualisation, planning and development.
In five cases, evaluators identified failures in the ongoing management of projects: three cited inadequate flexibility in the use of target systems, which had skewed project outcomes, and two more found that managers had not addressed policies that were producing unintended and undesirable outcomes.
Dr Julia Griggs, research officer at Oxford University’s social policy department, who co-authored a Joseph Rowntree Foundation study of the evaluation of regeneration schemes (R&R, 7 March 2008, p2), says that such unforeseen negative outcomes usually arise when project managers haven’t properly thought through the potential consequences of their work.
Conclusion
Errors will always occur when working on regeneration – and perhaps the greatest one is failing to learn from your mistakes. Our evaluation reports show that too many schemes fail to communicate with, identify and target a particular group of service users or clients; don’t gather baseline data and monitor their impact from the outset; are reluctant to share power with partner agencies; or find their work constrained or twisted by short-term funding. And the solutions to all these problems lie mainly in one over-arching asset: good planning.
Good planning and management can produce better engagement with potential service users, improved partnership working, more fruitful short-term projects and more effective monitoring and evaluation. Establishing projects with such care isn’t easy, particularly in a political environment rife with new initiatives – “intellectual vogues” – which eclipse tried and trusted projects and strategies. Yet if practitioners develop a strategy carefully and pursue it effectively with a wide range of partners, their success will speak for itself.
In Groundhog Day, Murray’s character reacts to his situation by going slightly crazy: he joyrides, commits suicide, carries out petty crimes, eats food that’s bad for him, and gets arrested. It is only when he starts to slow down and works with those around him that he wakes up to a new morning.
Government and funding agencies should take note that if we keep on making the same old mistakes, we’ll never create the change that communities need. Certainly, regeneration agencies must be free to explore new techniques: as Einstein said: “A person who never made a mistake never tried anything new.” But many regeneration agencies seem trapped by a system and a set of skills shortages that condemn them to making the same old mistakes again and again. So let’s sum up all the recommendations made in our 151 reports with three simple words: more thinking required.
ENGAGEMENT AND DELIVERY – HIGHLY-PRAISED SCHEMES
Our evaluation of Intensive Family Support Projects (R&R, 29 February, p22) showed that providing dedicated accommodation and on-site support was effective in dealing with problems faced by homeless families.
Community Access to Lifelong Learning (R&R, 13 May 2005, p22) achieved good results because it targeted a range of groups and offered appropriate services
PARTNERSHIP WORKING – HIGHLY-PRAISED SCHEMES
Childrens’ Trust Pathfinders (R&R, 25 January, p22) managed to pool budgets and develop strong partnerships, leading to coordinated, locally-distinctive programmes.
The second evaluation of the Sure Start early years initiative (R&R, 11 April, p22) revealed that improved targeting of services was producing real benefits for poor families.