Questions for the Integrated Review #3: How Should the UK Measure Success?
Authors' note: Thanks to Georgina Sinclair, Feargal Cochrane, and Tim French for their help with this briefing (all mistakes are the authors’ own).
In the first half of 2020, the Oxford Research Group (ORG) held a series of four face-to-face and online roundtables to understand the risks and challenges remote warfare could present over the next five years and how the Integrated Review could address them.
Three key questions emerged:
- How should the UK define its national security?
- How should the UK respond to threats?
- How should the UK measure the success of military interventions?
This briefing seeks to address the third of these questions. You can read the first of the three briefings here and the second one here.
Introduction
When the Integrated Security, Defence and Foreign Policy Review (or Integrated Review) is finally published, the UK Government will have to begin the much harder, and more important, task of implementing it. In the past, this is where many reviews have failed; while they have been bold in rhetoric, “significant changes to a policy” have only occurred, as one expert we spoke to noted, “in very limited circumstances.” If the UK Government is sincere in its stated ambition to make this review “the most radical reassessment of [the UK’s] place in the world since the end of the Cold War”, it will have to be mindful of the past mistakes which prevented reviews from having a tangible impact.[1]
In overcoming this challenge, the Government must continue efforts to improve monitoring and evaluation systems.[2] This would ensure that its bold rhetoric and ambitions actually matches the reality on the ground. As we have written elsewhere, the past few years have seen a number of important changes which have greatly improved the ability of UK delivered programmes to understand their impact.[3] This success is positive; however, translating these programme-level successes into systems that effectively measure the impact of UK activities in the long term has proven much more difficult.
This is particularly acute when it comes to remote warfare. Through this approach, UK forces are no longer tasked with the military defeat of an enemy but are deployed (often in relatively small numbers) to train local, national and regional forces to undertake much of the frontline fighting.[4] In such cases, the UK’s own objectives may be less tangible than the military has traditionally been used to, instead focussed on building international reputation or improving regional influence. This is even truer in a time of state-based threats, where international engagements are often focused on building influence so potential adversaries cannot.[5] Recognising this, senior military and political decision-makers have publicly acknowledged the need to improve the UK’s approach to measuring the long-term impact of these activities. Yet this is easier said than done.
To understand the role the Integrated Review can play in improving the UK’s approach to measuring success, ORG convened over 20 experts from a variety of backgrounds (including the military, civil society, government and academia) for a series of online roundtables. In general, the experts agreed that the UK had improved its ability to measure success but indicated that four problems remain:
- Current evaluation efforts are inconsistent between UK departments
- International engagement tends to be too fragmented to effectively measure success
- The focus on building influence, hinders efforts to measure impact on the drivers of instability
- Efforts to measure impact often fail to engage with the right people
This briefing will take each of these in turn, and then conclude by examining how the Integrated Review can better account for them.
Inconsistent Evaluation
In improving how the UK measures its impact, the Integrated Review should start with increasing coherence between UK departments. This has long been a challenge; in 2009, Sir Bill Jeffrey, the former Permanent Under Secretary of the Foreign and Commonwealth Office (FCO) said at a Defence Committee Inquiry: “I admire my military colleagues greatly, but they have a very special way of doing things.”[6] The Government has championed systems like the Conflict Stability and Security Fund (CSSF) and the roll out of Fusion Doctrine as ways in which these issues are being improved;[7] however, over a decade after Sir Bill’s comments, many of the same problems remain.
In our recent roundtable, we heard that there are stark differences in how the Department for International Development (DFID), the FCO, and the Ministry of Defence (MoD) measure their impact.[8] It seems that differences in language and planning processes continue to plague efforts to improve shared assessments.[9] In fact, we heard that the MoD was often “petrified” by the language in cross-departmental conflict and stability assessments and – feeling they couldn’t engage with them – did their own analysis instead. This is unworkable when the MoD is meant to be working with other departments to deliver stability on the ground.
In June, Prime Minister Boris Johnson announced a merger between DFID and the FCO, as a means of addressing some of these problems. He stated that bringing these departments together in the new FCO-led Foreign, Commonwealth and Development Office (FCDO) would “unite our efforts [and] take a comprehensive overview.”[10] However, this assumption needs carefully unpicking, especially when it comes to measuring UK impact.
It may well be that, in its pursuit of shared working, the Government moves backwards on monitoring and evaluation. As one participant said, DFID’s decentralised structure, along with “the media attention [on their] efficiency” in recent years, has meant they have “a good system” for measuring the impact of their activities. In contrast, the FCO and the MoD have much less experience in this area.[11] For instance, participants argued that the MoD “are good as measuring success on the battlefield” but have not “historically” measured longer term impact as effectively as other departments have done.
More generally, in addressing inconsistency in departmental efforts, a better approach would be to bridge differences between departments rather than merging them into one. The focus should be on maximising the ability of departments to pool resources and capabilities, while still recognising the unique and essential contributions each department offers. If the FCDO is to go ahead, then, finding ways of maintaining and safeguarding the unique skills and experiences of DFID is essential – including when it comes to monitoring and evaluation.
Fragmented nature of international engagement
In many areas of the world, the UK’s efforts are one small component in a complex network of overlapping unilateral, bilateral and multilateral efforts. A multitude of actors – including states, multilateral international organisations and private companies – all operate on the ground.[12] This is particularly true given the rise of remote warfare, where no one country can expect relatively minor contributions to have a significant impact on regional security sectors without the assistance of coalition partners.[13]
Unfortunately, the impact of the congested internationalised space is that, even when some states have good systems for measuring success, they tend to be lost in the broader fragmentation of international efforts. [14] As Matthias Deneckere, Ashley Neat and Volker Hauck noted in May 2020, “there is a whole range of actors … [and e]ach …[has] the ambition to play a role …, yet also have their own mandates, institutional interests and capacities.”[15]
This is clear in Somalia. The African Union Mission in Somalia (AMISOM) relies on a network of external partners for “logistical, financial and security force assistance”, with the UN, EU, and bilateral partners such as the UK and the U.S. all offering support.[16] Yet, while the volume and variety of these activities requires careful coordination, they “have been characterised by fragmentation rather than unity of effort.”[17]
This is not an easy fix and is hard even when countries have had a desire to learn and share lessons with others, as the UK has. For instance, one British military trainer we spoke to emphasised that differences in approach (including the fact that progress is measured at different stages of training and using different metrics) makes it difficult to compare results. However, it is essential that the UK tries to address this. As we have previously argued, it would be counter-productive to succeed in getting all of Whitehall to work as one, only to end up duplicating other countries’ existing efforts once engaged.[18]
Considering how the UK can play an active role in bridging international efforts should therefore be a key focus of the Integrated Review. It is promising that the Cabinet Office is seeking advice on “how to build alliances” ahead of the review.[19] However, as the next section explores, to truly have a positive impact on such activities (and measure this impact effectively), the UK must look beyond its immediate short-term national interests to ensure its activities support and encourage international coordination and collaboration.
Short-term v long-term aims
Over the last few years, the UK has put a lot of effort into improving its ability to measure increases in British influence and reputation in the places it has engaged. Last year, the Independent Commission on Aid Impact noted that CSSF “is incorporating influencing objectives into its theories of change, and further developing its Political Access and Influencing tool to support both monitoring and flexible decision making.”[20]
However, there has been much less discussion on how improving the UK’s ability to measure short-term national objectives, like influence, could distract from its efforts to measure the long term impact of its activities on peace and stability in the places it intervenes. For instance, many UK officials and soldiers acknowledge that the UK’s upcoming deployment to the UN mission in Mali is largely about building international relations with European allies (especially France) in light of Brexit.[21] The FCO said in evidence to the Foreign Affairs Committee that its pivot to the region will “support our alliances with international partners such as France, Germany and the [African Union] as we exit the European Union.”[22] A British soldier that ORG spoke to, put it more bluntly: “post-Brexit we need trade deals with France.”
The UK is not alone in pursuing influence in this manner. In previous roundtables, several experts noted that many Western nations have engaged abroad to build their own regional influence or international reputation. For instance, one said some Scandinavian countries had often engaged in the hope of getting a non-permanent UN Security Council seat. While Nina Wilén, of the Egmont Institute, said Belgium engaged in Niger because it “found itself in a situation where it did not have a presence in a region which has been considered for decades a strategic priority.”[23] Peter Albrecht and Signe Cold-Ravnkilde, of DIIS, argued that regional powers in the Sahel (such as Chad) and the Horn of Africa (such as Ethiopia and Kenya) have engaged in peacekeeping missions “to gain international recognition, fund their militaries, promote norms and values, or build an identity for their armies.”[24] These nations, then, are likely to measure success by these national objectives rather than long-term peace and stability.
Of course, nations rarely engage abroad for purely altruistic reasons; however, it is problematic when each nation measures success according to their own national goals, at the expense of assessing how their actions impact regional stability. Albrecht and Cold-Ravnkilde argue that, in Mali and Somalia, friction between national and international objectives “produces unpredictable and seemingly counter-intuitive practices on the ground that contradict the formal aim of the two [peacekeeping] missions: to stabilise the countries in which they deploy.”[25]
To improve this, the UK must ensure that long-term assessment is central to the UK’s approach to measuring success. As one participant argued, the UK should be asking of every intervention, military and otherwise: “Are we doing the right things to address the drivers of conflict?” They added that the UK should ask “whether [its actions are] a logical first step to get to peace, if the answer is no, we know you help spread terrorism and instability.” There is a risk that if the UK focusses on whether its actions are increasing its own influence, it will fail to ask these more important questions.
The UK may at times find that deploying troops is not the right approach, and may instead engage through other means. This may be no bad thing, recent campaigns in Iraq and Afghanistan have, as one participant noted, already shown the dangers of engaging “without significant acknowledgement of what can be done or the possibility of success.”[26]
Engaging with the wrong people
The importance of local buy-in has long been acknowledged by the UK government and experts alike. In April 2018, the Stabilisation Unit’s Elite Bargaining and Political Deals report noted that “interventions can be ineffectual, or counter-productive, when interveners fail to analyse and engage effectively with underlying configurations of power.”[27]
The Integrated Review should, then, address how to improve communication with both the host government and with civil society.[28] It remains the case, for instance, that the UK still does not engage key stakeholders in-country throughout the design of a programme, including in the development of success indicators.[29] Instead, the UK’s approach reflects a wider trend in which “many states and donors often do not ask partners to shape the analysis or design … [r]ather they are asked to ‘comment’ on plans already made.”[30] As a result, the UK’s understanding of success does not necessarily align with the host country’s understanding.
At the same time, the UK must be mindful that partners may contribute to the drivers of instability, rather than address them. The Armed Conflict Location & Event Data Project (ACLED) reported that last year: “Governments continue to pose the greatest threat to civilians around the world, with state forces responsible for more than a quarter of all violence.”[31] In such contexts, building the capacity of predatory armed forces will feed a self-perpetuating cycle of violence and conflict.[32]
To address this, the Integrated Review should necessitate open and meaningful conversations with civil society groups in countries where the UK engages. These groups often have a much better knowledge of the community they represent or engage with than the state elites that the UK and its allies tend to speak to. Sufficient engagement with these groups will also highlight the ways in which men, women, boys, and girls are impacted differently in conflict and post-conflict situations. In this sense, they can cast an essential light on who is secure, and how meaningful the “security” that a community or a country is moving towards actually is – and point out when UK programmes are having unintended consequences for certain groups.
One roundtable participant noted that there are two important challenges when it comes to including local civil society:
- There are insufficient “resources” to undertake such engagement
- British policymakers “don’t see civil society as useful for analysis”
We have written elsewhere about the second of these challenges, however it is clear how it interacts with the first. The fact that the UK does not see civil society involvement as useful is the reason it is not given enough resources.[33] This has dire consequences because, as the same participant noted, “decisions are made in London which are not grounded in the reality of people’s lives.”
It also raises an important lesson for the Integrated Review: the importance of properly financing monitoring and evaluation efforts. As another roundtable participant said, it is clear that “the prioritisation of resources” dictates success. Thus, the UK must ensure there are systems (and more importantly) resources in place so that those who best understand the UK’s impact in country are meaningfully engaged in UK policy making.
Conclusion
The Government has repeatedly pointed to the ongoing Integrated Review as an exciting opportunity for the UK to become a “force for good” in the world.[34] For this to become a reality, the review must go beyond boisterous rhetoric and set up strong systems for measuring success. This is as important in remote warfare as any other form of engagement – particularly because, relatively small or not, poorly planned or poorly coordinated activities can still have a lasting and detrimental impact on peace and stability.
Despite a growing recognition among governments, international organisations, academics and civil society that there must be better efforts to monitor and evaluate partnered operations abroad, serious challenges remain.[35] This is true both for the UK’s own approach, and the international community as a whole. These challenges are unlikely to be addressed soon. In fact, as we discuss elsewhere, these problems may get even worse over the next few years, as international attention shifts to state-based threats.[36]
The Integrated Review offers an opportunity to mitigate against future risks by putting in place strong systems for measuring success and, perhaps more importantly, to reassess how it measures success and the values that define such systems. To do this, it should:
- Learn from DFID’s progress in developing better cross-government systems and ensure that unique departmental skills are not lost in the pursuit of shared working.
- Assess the UK’s own role in improving international cooperation and, in doing so, prioritise long-term peace and stability above national influence. In doing so, prioritise long-term peace and stability above national influence.
- Ensure that the UK is reticent of what success looks like for local partners by ensuring its systems (and resources) prioritise local engagement in all of its activities.
The UK’s bold plans to be a “force for good” cannot end with a shiny new report.[37] Success comes from having a clear idea of the UK’s end goal and, as importantly, developing systems that measure whether it is moving closer or further away from that goal. The Integrated Review must, then, acknowledge it is the start, not the end.
References
[1] Prime Minister’s Office, 10 Downing Street, and The Rt Hon Boris Johnson MP, ‘PM Outlines New Review to Define Britain’s Place in the World’, GOV.UK, 26 February 2020, https://www.gov.uk/government/news/pm-outlines-new-review-to-define-britains-place-in-the-world.
[2] Abigail Watson and Megan Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps: Lessons Learned from Remote Warfare in Africa’ (Oxford Research Group, 6 November 2019), https://www.oxfordresearchgroup.org.uk/fusion-doctrine-in-five-steps-lessons-learned-from-remote-warfare-in-africa.
[3] Watson and Karlshoej-Pedersen.
[4] Emily Knowles and Abigail Watson, ‘Remote Warfare: Lessons Learned from Contemporary Theatres’, Oxford Research Group, 27 June 2018, https://www.oxfordresearchgroup.org.uk/remote-warfare-lessons-learned-from-contemporary-theatres.
[5] Abigail Watson, ‘Questions for the Integrated Review #2: How to Engage: Deep and Narrow or Wide and Shallow?’, Oxford Research Group, 14 July 2020, https://www.oxfordresearchgroup.org.uk/questions-for-the-integrated-review-2-how-to-engage-deep-and-narrow-or-wide-and-shallow.
[6] House of Commons Defence Committee, ‘The Comprehensive Approach: The Point of War Is Not Just to Win but to Make a Better Peace’, Seventh Report of Session 2009–10 (House of Commons, 9 March 2010), https://publications.parliament.uk/pa/cm200910/cmselect/cmdfence/224/224.pdf.
[7] Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[8] Watson and Karlshoej-Pedersen.
[9] Watson and Karlshoej-Pedersen.
[10] ‘Prime Minister Announces Merger of Department for International Development and Foreign Office’, UK Parliament, 16 June 2020, https://www.parliament.uk/business/news/2020/june/prime-minister-announces-merger-of-department-for-international-development-and-the-foreign-office/.
[11] ‘UK Aid Department to Be Merged with Foreign Office’, BBC News, 16 June 2020, sec. UK Politics, https://www.bbc.com/news/uk-politics-53062858.
[12] Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[13] Watson and Karlshoej-Pedersen.
[14] Signe Marie Cold-Ravnkilde and Katja Lindskov Jacobsen, ‘Disentangling the Security Traffic Jam in the Sahel’ 96, no. 4 (6 July 2020), https://www.chathamhouse.org/publication/ia/disentangling-security-traffic-jam-sahel.
[15] Mattias Deneckere, Ashley Neat, and Volker Hauck, ‘The Future of EU Security Sector Assistance: Learning from Experience’, ECDPM Paper (ECDPM, May 2020), https://ecdpm.org/publications/future-eu-security-sector-assistance-learning-experience/.
[16] ‘Assessing the Effectiveness of the African Union Mission in Somalia (AMISOM)’, EPON (blog), accessed 20 June 2020, https://effectivepeaceops.net/publication/amisom/.
[17] Paul D. Williams, “Assessing the Effectiveness of the African Union Mission in Somalia (AMISOM),” (EPON, 2019) https://effectivepeaceops.net/wp-content/uploads/2019/04/EPON-AMISOM-Report-LOWRES.pdf.
[18] Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[19] Cabinet Office, ‘Integrated Review: Call for Evidence’, GOV.UK, 13 August 2020, https://www.gov.uk/government/publications/integrated-review-call-for-evidence.
[20] ‘The Current State of UK Aid: A Synthesis of ICAI Findings from 2015 to 2019’, The Independent Commission for Aid Impact, June 2019, https://icai.independent.gov.uk/wp-content/uploads/The-current-state-of-UK-aid_Synthesis-of-ICAI-findings.pdf.
[21] Abigail Watson, Delina Goxho, and Ewan Lawson, ‘Influence or Strategy: Lessons from Iraq Ahead of Mali’, Wavell Room (blog), 9 July 2020, https://wavellroom.com/2020/07/09/influence-or-strategy-lessons-from-iraq-ahead-of-mali/.
[22] Foreign and Commonwealth Office, ‘Written Evidence from the Foreign and Commonwealth Office on Behalf of Her Majesty’s Government’, House of Commons Foreign Affairs Committee (FAC) Inquiry, September 2019, http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/foreign-affairs-committee/beyond-aid-the-uks-strategic-engagement-in-africa/written/105575.pdf.
[23] Nina Wilén, ‘Belgian Special Forces in the Sahel: A Minimal Footprint with Maximal Output?’, Africa Policy Brief (Egmont Institute for International Relations, 14 May 2019), http://www.egmontinstitute.be/belgian-special-forces-in-the-sahel-a-minimal-footprint-with-maximal-output/.
[24] Peter Albrecht and Signe Cold-Ravnkilde, ‘National Interests as Friction: Peacekeeping in Somalia and Mali’, Journal of Intervention and Statebuilding 14, no. 2 (21 February 2020): 204–20, https://doi.org/10.1080/17502977.2020.1719789.
[25] Albrecht and Cold-Ravnkilde.
[26] Abigail Watson, ‘Questions for the Integrated Review: #1 How Do We Define Security?’, Oxford Research Group, 25 June 2020, https://www.oxfordresearchgroup.org.uk/questions-for-the-integrated-review-1-how-do-we-define-security.
[27] Cheng, Goodhand, and Meehan, ‘Elite Bargains and Political Deals Project’.
[28] Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[29] Watson and Karlshoej-Pedersen.
[30] Timonthy Donais, Local Ownership and Security Sector Reform (Geneva Centre for the Democratic Control of Armed Forces (DCAF), 2008).
[31] ‘ACLED 2019: The Year in Review’, ACLED (blog), 2 March 2020, https://acleddata.com/2020/03/02/acled-2019-the-year-in-review/.
[32] Emily Knowles and Jahara Matisek, ‘Western Security Force Assistance in Weak States’, The RUSI Journal 164, no. 3 (16 April 2019): 10–21, https://doi.org/10.1080/03071847.2019.1643258.
[33] Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[34] Megan Karlshøj-Pedersen and Liam Walpole, ‘Forging a New Path: Prioritising the Protection of Civilians in the UK’s Response to Conflict’, Oxford Research Group, 29 July 2020, https://www.oxfordresearchgroup.org.uk/forging-a-new-path-prioritising-the-protection-of-civilians-in-the-uks-response-to-conflict.
[35] ‘The Current State of UK Aid: A Synthesis of ICAI Findings from 2015 to 2019’; Watson and Karlshoej-Pedersen, ‘Fusion Doctrine in Five Steps’.
[36] Watson, ‘Questions for the Integrated Review #2’.
[37] Dominic Raab, ‘Global Britain Is Leading the World as a Force for Good: Article by Dominic Raab’, GOV.UK, 23 September 2019, https://www.gov.uk/government/speeches/global-britain-is-leading-the-world-as-a-force-for-good-article-by-dominic-raab.
