Stephen Hale

More on London Summit evaluation

I've blogged before about evaluation and our London Summit digital campaign. We've now published the detail: Evaluation of key performance indicators.

It's a full evaluation of the performance indicators we set for the London Summit website back in January. It uses survey responses and metrics from our own web platform and Google Trends to test how we performed against our own objectives.

The highlights:

• 4 objectives
• 13 KPIs
• 3 were fully met
• 4 were partially met
• 4 were not met
• 2 could not be assessed with the data we have
  

Details in the report.

Posted at 16:50 20 May 2009 by Stephen Hale  |  Comments[2]

London Summit digital campaign evaluation

So our website is now a historical record - the debate has moved on. There's still plenty to do of course, and for many the summit marks the start of a new debate. But our digital campaign is over, and we can take a step back and assess how we did. Did it deliver what we intended? And what lessons can we learn and share?

We'll publish all our evaluation on the London Summit website itself, including all the numbers, measurement against all our KPIs, and the independent evaluation of others. But it's been a couple of weeks now, and I think I have enough distance and data to reflect on what we did.

We started this work with a degree of confidence. We've been thinking about - and practicing - how to do digital diplomacy in the Foreign Office for a while now, and as a result we were pretty confident about our methods in January. The lead up to the summit seemed like the ideal opportunity for digital engagement. I published here the performance indicators we set for ourselves at the start of the process, but if I'm honest, I really didn't know whether we'd deliver them all, or how much of an impact our digital campaign would have.

So did we succeed? Well, I'm not sure yet.

There are bits that I think we did well. I think we made good judgments about the ambition for the London Summit website, and the tone of the content that appeared there. We did well to ensure that the site aggregated content from around the world, and provided space for a range of voices. Some of our partnerships with other web platforms and forums worked really well. We produced a lot of video and photo content, which helped bring the debate to life. We provided good authoritative content on the purpose of the summit, an efficient service for journalists and accreditation, live streamed video throughout the summit, and lots of structured ways for people to contribute to the debate.

We've learned a lot about how to deliver intensive digital campaigns. We had a great team working on the campaign, and we relied on the expertise of others across government, the FCO network and our partners. We published new content many times a day which enabled us focus our work on the narrative of the debate as it emerged.

We also learned a lot about the FCO web platform, about delivering content to a global audience with a huge concentrated peak of traffic. And we learned a lot about citizen engagement, how and when to encourage debate, and when to just take a step back and reflect what's going on elsewhere.

There were plenty of things that didn't work. As you'll see from the performance indicators below, there are some things that we set out to deliver, but just didn't achieve. Some of our ideas were never realised and some of our partnerships just didn't come off. We hosted and commissioned a lot of debate, and we listened to what others were saying online, but we didn't often actively participate ourselves other than in delivering messages, aggregating, or summing up.

Our website was a destination for authoritative content and specialist debate, but of course other websites provided alternative places for the popular global debate to take place online. And our success in encouraging debates in some countries, wasn't matched with all our target audiences.

Of course, this is all just subjective, and I'm possibly not in the best place to judge. So we'll publish all the bits of evaluation that we and others produce on the now frozen summit site, and I'll highlight the best bits here.

But I promised to report on our performance against the original KPIs. So here you are (we'll publish the full version of this on the summit site):

1. The focal point for engaging and shaping global opinions

- http://www.londonsummit.gov.uk most authoritative site for "London Summit" according to major search engines.

Met. Our content was well optimised for search so we were quickly at the top of natural search rankings for our key terms. Having said that, we found that other terms were far more widely used that some of the initial key terms that we identified.

- influential sites in every target country link back and quote from http://www.londonsummit.gov.uk

Partially met. Of the 23 countries targeted, influential sites (those with a Google page rank of 6 or above) from 12 countries linked to the London Summit site.

- all visitors find it quick and easy to find the info they are looking for

Our user survey will provide more data here.

2. Authoritative provision of in-depth briefings on Summit

- all unclassified policy papers accessible from londonsummit.go.uk in web friendly form

Met. We published everything we had, including communiques, transcripts, summaries, video. And we didn't resort to PDFs.

- only the highest-quality content goes on the site based on the content guidelines. If it doesn't help to achieve an objective is doesn't go on

Partially met. We wrote good copy, kept to our content guidelines, and had a clear process for clearing policy-sensitive content. But I know we sometimes published in a hurry, and corrected later.

- at least four expert bloggers providing authoritative real time content for London Summit

Not met. We ran an editors' blog, Tom Watson blogged from the summit, Stephen Timms posted tweets (#timms), the Foreign Secretary posted blogs, and government economists posts articles on the Vox EU debate. But we didn't have specific attributed expert bloggers contributing to the online debate throughout the campaign.

3. Effective operational functions for 2,000 journalists

- Media centre regarded by journalists as most respected government media site ever

I think we did ok, but our survey of users will tell us more.

- live streaming of all press conferences/keynote speeches

Partially met. We streamed the whole summit. We didn't live stream any other events, but we published a lot of same-day video thanks largely to our partnership with British Satellite News.

- the site is reliable (minimal down time) and meets AA accessibility at all times

Partially met. The site had 99.82% availability, the page templates all met AA standards and have been tested with real users for accessibility, and our key content was always provided in accessible formats. But our content did not always validate as AA accessible (for example, we did not caption, or provide text alternatives for all our content, and when were faced with publishing a video without a transcript, or not publishing it at all, we published the video).

4. Respected Platform for discussion and debate

- seamless integration with all partner engagement sites

Not met. Our referral stats show that user journeys between the parts of our web presence were not common. So although we did some cross promotion through RSS feeds and promos we didn't always deliver reciprocal links, so our users could not move seamlessly between our partner sites.

- clear evidence of link between pre-summit web debate and post-summit outcomes

Met. Our debate issues reflected many of the outcomes. It is less easy to measure whether the online debate influenced - or just reflected - the outcomes, but I hope our detailed evaluation (and the evaluation of others) will provide some data here.

- visitors return to the site, go to other areas of our London Summit web presence or subscribe to feeds/emails

Partially met. 27% of visitors to the site were repeat visitors, and 2,273 people subscribed to receive our email newsletters, but there wasn't significant traffic between areas of the summit web presence.

- the site (and related wider web-presence) becomes a best-in-class example of digital engagement

For others to decide

And some slected numbers (because however much I protest that evaluation isn't just about numbers, I know I can't get away with not mentioning them at all):

London Summit website (29 January - 6 April): 466,159 visits, 1,572,643 page views

London Summit YouTube: 149,580 video views

Live video streaming: 165,000 views on 2 April

London Summit Flickr: 1,065,825 photo views

VoxEU Global Crisis debate (ongoing): 160 articles, 300,000 page views

Yoosk London Summit: 327 questions, 36 answers from public figures

Chosun debate (Korean London Summit forum): 231,832 unique visitors (by 2 April)

I'll post more on the digital engagement, and the other evaluation as it's published.

Posted at 17:22 17 April 2009 by Stephen Hale  |  Comments[1]

Summit bloggers make history

The London Summit broke new ground for digital media. I feel sure that the debate that took place in blogs and forums influenced opinion around the world in the lead up to and during the summit, and that digital debate has contributed to the way in which the outcome of the summit has been received.

We've played our part, running the UK government web presence, but a lot of the debate has taken place spontaneously elsewhere. I've been really excited about the work of G20 Voice, bringing 50 bloggers from 22 countries to the summit venue. I played a very small role in helping to make some connections, so I know that the 50 bloggers owe their place at the summit to the vision and perseverance of Shane, Karina and the rest of the G20 Voice coalition. Their legacy will be that it will probably seem perfectly natural for bloggers to have similar or greater access to the next summit.

And it almost passed me by, but I think there was a historic moment for blogging in the Excel Centre when Richard Murphy - one of the G20 accredited bloggers - was called to ask the Prime Minister a question.

Here he is asking it:

And here he is reflecting on the moment afterwards:

Posted at 01:04 04 April 2009 by Stephen Hale  |  Comments[5]

London Summit: 1 day to go

We're nearing the end of our digital campaign in the lead up to the London Summit.

The foreign leaders are arriving, our debate phase is almost over, and we're now focussing on covering the summit as it happens and reflecting the global reaction.

The summit is everywhere now (in old and new media), traffic to our web presence has soared, and people have taken to the streets of London. It seems silly to think that a few weeks ago, we were thinking about ways in which we could generate interest.

Over the next couple of days you can watch the live stream from the summit, and we'll be featuring other content as it happens on the London Summit site.

I'll post more on what we did, what worked and what didn't, and what we've learned.

Posted at 13:10 01 April 2009 by Stephen Hale  |  Comments[0]

Proving the value of digital campaigns (and why Twitter won't take off in Ukraine)

I met a delegation of Ukrainian government officials the other day to talk about digital diplomacy. It's one of the perks of my job that people outside the UK are interested in what we do. They provide a challenge that I don't necessarily get from my peers in the UK digital community. (Who else is going to tell me that Twitter won't work as a tool in Ukraine - because you only get about 3 Ukrainian words for 140 characters?)

We talked about the online campaigns that we've run recently in the Foreign Office, the way we manage and present web content, and some of the tools we've been using for digital engagement. I think I surprised them (and myself) by how excited I got when they asked how we evaluate our work in Digital Diplomacy Group. But the fact is I am very excited about proving that digital engagement works. And more than that: I think we have a responsibility to measure the actual impact of digital campaigns, rather than get carried away with the ease with which we can develop new tools.

Of course, web practitioners are notoriously lazy about evaluation because everything we do on the web produces numbers. Stats are almost always interesting, and it's easy to present them as evaluation. But they're not enough. The Foreign Office web platform had 2.5 million unique visitors  in January. But so what? I know that I could significantly drive up traffic to the Foreign Office YouTube channel by posting a film of 150 ambassadors line dancing (I'm sure they'd be up for it). But traffic doesn't deliver foreign policy objectives. It just delivers traffic.

Our approach to evaluation was developed by Liam King, who is even more excited than I am about evaluation. It's not complicated - this is what we aim to do:

1. Insist on setting objectives and identifying target audiences for everything we do on the web.

2. Pick something that we can measure that will give us an indication of how well we met our objectives and and reached our target audience.

3. Measure it.

We do use stats, and we welcome independent evaluation (the Hansard Society are evaluating our blogs and our London Summit campaign at the moment), but we concentrate on providing evidence that tells us something about what we set out to achieve. This approach means that all the evaluation we do is useful for the people we're working with (because we are very clear about expectations right at the start), and it's useful for us (because we can use it to improve what we do).

I've pasted below the objectives and performance indicators that Liam and Paul set Digital Diplomacy Group in January for our work on the London Summit website.  Our approach will develop, and we'll measure KPIs for each of our engagement exercises over the next 6 weeks. But the original performance indicators won't change - once the summit is over we'll be able to say with authority whether we delivered what we set out to.

London Summit website objectives and performance indicators:

1. The focal point for engaging and shaping global opinions

• http://www.londonsummit.gov.uk most authoritative site for "London Summit" according to major search engines.
• influential sites in every target country link back and quote from http://www.londonsummit.gov.uk
• all visitors find it quick and easy to find the info they are looking for

2. Authoritative provision of in-depth briefings on Summit

• all unclassified policy papers accessible from londonsummit.go.uk in web friendly form
• only the highest-quality content goes on the site based on the content guidelines. If it doesn't help to achieve an objective is doesn't go on
• at least four expert bloggers providing authoritative real time content for London Summit

3. Effective operational functions for 2,000 journalists

• Media centre regarded by journalists as most respected government media site ever
• live streaming of all press conferences/keynote speeches
• the site is reliable (minimal down time) and meets AA accessibility at all times

4. Respected Platform for discussion and debate

• seamless integration with all partner engagement sites
• clear evidence of link between pre-summit web debate and post-summit outcomes
• visitors return to the site, go to other areas of our London Summit web presence or subscribe to feeds/emails
• the site (and related wider web-presence) becomes a best-in-class example of digital engagement
  
  

I'll report back on how we did against these in April.

Posted at 13:29 19 February 2009 by Stephen Hale  |  Comments[5]

Making the London Summit a 2 month online event

I mentioned last week that we were considering our options for digital engagement in the lead up to the London Summit. I can tell you now where we've got to.

The Prime Minister launched the official London Summit website earlier today (video below). Here it is: www.londonsummit.gov.uk.

There's plenty more to do but we've come a long way - in about 4 weeks we have turned a vague idea into a comprehensive strategy, a functioning website, and a set of tools and partnerships for engagement.

We want the website to be a source of authoritative content, to aggregate content from the debate around the world, and to be a stimulus for - and a home for - debate around the summit issues. So what's on it?

A dynamic multimedia home page that we will refresh every day including, if necessary, at weekends. We haven’t finished with this yet.

The Summit aims section of the site sets the context, describing the background to the crisis and UK positions. This is the most comprehensive part of the website now, and it's unlikely to change that much in the next 2 months.

The Join the debate section is a really a skeleton at the moment, but it will grow organically over the next couple of months as we help direct and respond to the global debate. This will be where we encourage and present the debate, on and off the website. At the moment, we are promoting content from our partnerships with at VoxEU, Yoosk, and the World Economic Forum as well as surfacing the debate from around the world.

We'll use the Global update channel to highlight the debate from around the world. We're well placed in the Foreign Office to use our network of embassies to report on the debate as it happens locally. We need to develop this quickly.

The Media centre will provide journalists with all they need in the lead up to the summit.

We've built all of this on the existing FCO web platform, so at no extra cost to the taxpayer.

Launching a website often feels like an end in itself, but the London Summit website is clearly only the start something - our efforts to stimulate online conversations will undoubtedly increase over the next 2 months.

And of course, the website itself is only part of our digital engagement strategy. I'll post more about the specific engagement exercises as they happen, and I'd welcome your thoughts on what we've done so far, and what else we should be doing.

Posted at 17:09 30 January 2009 by Stephen Hale  |  Comments[5]

How can the web help solve global financial problems?

When the Prime Minister announced that London will host a summit of world leaders on the "major questions of economic action" he set in motion a huge exercise in coordination and communication in Whitehall.

Officials across Whitehall are well practiced at this kind of thing, so I'm confident that by the time the world leaders gather in London on 2 April everything possible will have been done to ensure that the summit delivers.

But how can the web contribute to the success of the summit?

The build up to the summit should be an ideal opportunity to harness the power of the social web. We have: a set of problems that need solutions, existing active communities who are already talking about the issues, an offline process that has real authority and decision making power, and a defined timeframe in which to deliver.

So this could be a huge opportunity for digital engagement, but also a massive challenge. Should we try to lead, or just contribute to the conversation? Should we run big, high-profile headline-grabbing engagement exercises or smaller targeted outreach with particular groups? And how can we ensure that our digital engagement actually feeds into the policy making and decision making process?

At the moment we are planning to:

1. Run set-piece forums for discussion aimed at different audiences. (Because the UK government wants to know how different communities react to our emerging ideas.)

2. Reach out into spaces where people are already discussing the "major questions of economic action". (Because however much governments would like to lead the debate, it already has a life of its own and much of the conversation will inevitably take place elsewhere.)

3. Create a web presence for the summit that sets out the issues as we see them, but also aggregates comment and opinion from elsewhere. (Because anything we produce on this has to reflect the problem-solving nature of the conversation that is taking place around the world, and a genuine openness to new ideas.)

We have some ideas on how to do all this but I'd really like to hear what others think. Can the web can help answer the major questions of economic action? How would you like to contribute to the conversation? Would you prefer to engage with government on official websites, or elsewhere? What web tools would you use to stimulate debate? Where on the web are the lively debates already taking place? Who should we collaborate with? What lessons can we learn from similar exercises?

I'll post more on the detail as we start doing things. But I'd really like to hear what you think.

Posted at 12:34 22 January 2009 by Stephen Hale  |  Comments[4]