Assessing the impact of outreach strategies in cities coping with climate risks

3.1 Resilience Alliance

The “Resilience Alliance” (RA) (Resilience Alliance: https://www.resalliance.org/, last access: 14 October, 2018; Resilience Alliance, 2010; Haider et al., 2012) is an international, multidisciplinary research organisation that has developed guidelines to assess resilience of social–ecological systems and to implement sustainable development strategies. RA outlines an assessment framework that is consistent with the social–ecological approach. According to this method, multiple spatial and temporal interacting scales must be considered to comprehend, for instance, how resilience at the level of a neighbourhood can affect the resilience of a city. Furthermore, for each quantitative variable, a threshold should be defined. If the threshold is crossed, the effects on other social and ecological variables must be identified. Communication is evaluated in terms of “information sharing”, a factor that characterises social relations among stakeholders. According to the RA method, social relations and information flows need to be monitored since they affect the governance system. This specific factor is analysed by comparing different social network structures: e.g. a highly centralised network or a network composed of two isolated subgroups. The RA approach focuses on institutional relations and does not offer any tool to investigate the intensity or quality of information exchange.

3.2 The Disaster Resilience Of Place model and the Baseline Indicators for Communities

The “DROP” (Disaster Resilience Of Place) model and the related “BRIC” (Baseline Resilience Indicators for Communities) (Cutter et al., 2008, 2010, 2014) focus on resilience to natural hazards at community level and on the relationship between resilience and vulnerability. The model is a conceptual basis to identify resilience indicators that can be used at different spatial scales. It outlines a composite resilience index, with sub-indexes corresponding to different dimensions of the urban system. These indexes are exclusively quantitative variables, and a method is proposed to normalise different ranges of values to a unique scale. Social factors that are related to communication and are examined by Cutter et al. (2014) concern public engagement, education, training and communication infrastructure. Examples of indicators are “English language competency (% Population proficient English speakers)”, “Local disaster training (% Population in communities with Citizen Corps programme)”, “Communication capacity (% Households with telephone service available)', “Social capital-disaster volunteerism (Red Cross training workshop participants per 10 000 people)”. Other important communication infrastructure – such as mobile phones, the Internet, TV and radio – as well as public outreach are not considered.

3.3 Disaster Resilience Scorecard for Cities

In 2010, the United Nations secretariat of the International Strategy for Disaster Reduction launched the campaign “Making Cities Resilient” (Making Cities Resilient: https://www.unisdr.org/campaign/resilientcities/, last access: 28 September 2018). Following the adoption of the “Sendai Framework for Disaster Reduction” (UN/ISDR, 2015)³, the goals and priorities of the campaign were updated, as well as the resilience assessment tool dedicated to cities: the “Disaster Resilience Scorecard for Cities” (DRSC) (UN/ISDR, 2017). This methodology is aimed at monitoring the implementation of the Sendai Framework at local government level through “Ten Essentials for Making Cities Resilient”. Two kinds of evaluations are foreseen to monitor the progress towards these essentials: a preliminary evaluation that includes 47 qualitative and quantitative indicators with a graduated 3-point scale, and a detailed evaluation involving the participation of stakeholders, which includes 117 indicators with a 0–5 score.

Among the three selected assessment frameworks, the DRSC is the method that proposes the widest number and variety of indicators that can be related to communication processes. Examples of indicators are “Use of mobile and e-mail `systems of engagement' to enable citizens to receive and give updates before and after a disaster”, “Extent to which data on the city's resilience position is shared with the community organisations and public”, “Consultation in plan making”, “Accessibility of education and training to all linguistic groups in the city”, and “Communication service days at risk of loss”. These indicators are monitored in terms of presence/absence of efforts, frequency of activities, audience diversity and size (e.g. how many of the listed groups of stakeholders are involved in plan consultation?), and exposure (e.g. how many media channels are used for public relations and education campaigns?).

3.4 General remarks on communication evaluation and next steps

The proposed approach draws on the three assessment methods we have presented in this section. Indeed, as proposed by the RA method (Sect. 3.1), we aim to detect possible correlations between social processes (e.g. communication) and environmental processes. In accordance with the first two approaches (Sect. 3.1 and 3.2), we also consider that quantitative variables are adequate to establish RCIs. In fact, quantitative variables facilitate the observation of space–time variability, as well as the investigation of correlations between communication processes and other resilience drivers. We also consider the frequency of communication activities, the exposure, the audience size and composition as relevant variables, similarly to what is outlined in the DRSC approach (Sect. 3.3).

At the same time, the aim of our study goes beyond a description of the state of communication processes and infrastructure in an urban area exposed to risks. We propose a dynamic perspective, with the RCIs outlined in the next section and tested in Sect. 5. The RCIs are tailored to appraise how specific communication activities contribute to achieving local resilience goals. These indicators require a rich and detailed selection of communication variables that are activity driven; i.e. they allow the detection of resilience changes that occur even in a brief time, and the causes related to a past or ongoing communication activity.

The audience knowledge and its behavioural change, due to communication activities, have been investigated by researchers in the fields of risk communication and of public understanding of science. The first series of works explore the impact of risk communication on the audience perceptions and behaviour. Rohrmann (1992) and Lundgren and McMakin (2004) discuss the importance of evaluating the success of risk communication strategies. Other authors (e.g. Terpstra et al., 2009; Maidl and Buchecker, 2015) evaluate the impact of specific risk communication activities such as interactive workshops, focus groups and one-way communication.

The second set of studies describe different approaches and techniques to evaluate science outreach. Godin and Gingras (2000), Bauer (2008), and Neresini and Pellegrini (2008) discuss the methodological principles of science outreach assessments. Examples of evaluation experiments are described by Scheufele et al. (2005), who designed a national telephone survey on nanotechnology, and by Wagoner and Jensen (2015), who assessed adolescents' learning at the zoo by combining questionnaire data with responses with detailed verbally administered questions.

Our research specifically investigates resilience communication and tests what we define as resilience communication indicators: quantitative indicators to comprehend how a communication activity influences urban resilience to climate risks. Nevertheless, the above-mentioned literature provides helpful hints for this study. As highlighted by Neresini and Pellegrini (2008), the evaluation of a communication activity presupposes that the communication goal is taken into account. For instance, assessing a communication campaign aimed at disseminating knowledge on flood resilience is different from assessing a campaign that promotes public engagement in flood resilience strategies. In the specific case of the communication activities examined in Sect. 5, the goal was to raise awareness of flood resilience issues and solutions in the Paris region, as well as to gain the support of the local population for the project implementation. Since this research focuses on communication indicators, we assume that communication activities – planned and implemented in the framework of a resilience strategy or project – enhance resilience if they contribute to achieving the strategy or project goals.

As stated by Neresini and Pellegrini (2008), we consider that the methodological principles developed in the sphere of social research can be applied to communication evaluation. In this paper we refer to the research process and techniques adopted in the field of “social impact assessment” (SIA) (Inter-organisational Committee on Guidelines and Principles for Social Assessment, 1994; Social Analysis at the World Bank: http://www.worldbank.org/socialanalysis, last access: 25 September 2018), an approach aimed to monitor and better take into account the effects of social variables in territorial development. In particular, the experiments presented in this paper (Sect. 5) are based on three investigation techniques that are used in SIA: research based on secondary sources, a questionnaire and interviews. The first two techniques aim to monitor a selection of RCIs, and the last one is employed for exploratory research.

As we mention above, it is necessary to keep in mind the objective of a communication activity when evaluating it. In the next section, we recall that a communication objective also depends on the territorial context where an activity is implemented. The study that is summarised in the next section, and the experiments that are presented in Sect. 5 were all carried out in the context of Paris region and its flood resilience challenges.

5.1 Media coverage monitoring

Among the communication values that were monitored during the RainGain project, the media coverage has reached remarkable results that have far surpassed the target values. The data presented in this section were collected from different sources:

• feedback from the Communication Department of École des Ponts that constantly monitors, through Argus de la Presse (Argus de la Presse: http://www.argus-presse.fr/, last access: 20 November 2018), whether the media mention “École des Ponts”;

• a periodic search on Google News of press news items that include the key word “RainGain”;

• feedback from the researchers that were interviewed by the press on the RainGain project;

• data on the audience size of printed press were collected on each newspaper website.

From July 2011 to December 2015, we have counted a total number of 65 news items on the RainGain project, published by the French, Dutch and Belgian press. These news items include 29 articles on printed press, 6 TV reports, 5 radio reports, 25 Web articles and online video reports.

Figure 1 shows that during specific months the number of news items has rapidly increased. Two kinds of events have occurred when the increase rate was high:

• The RainGain communication activities (a press release in March 2013 and two conferences in October 2013 and May 2015). These are social and endogenous causes of news rate increase, since they are the outcome of the work of the project team;

• Flood events in The Netherlands (October 2013) and in France (October 2015) that are environmental and exogenous causes. The impact of a flood event on media coverage is an example of correlation between an environmental factor and a social factor.

Data on the number of press articles were compared to the data on the newspaper audience size. We refer to audience size as the readership that has been estimated by the newspaper editor or by companies specialised in media measurement. These estimates are not precise as measurements of immediate audience (i.e. a public who is face-to-face with the speaker), but provide an approximate metric of the potential dissemination of a news item. Figure 2 displays a comparison between the temporal evolution of the cumulative number of articles and the cumulative audience size of printed press. The difference between the two cumulative curves is variable. Indeed, different newspapers have different impacts in terms of audience size. Hence, the impact of an article is variable according to the newspaper that publishes it. This is particularly true when we compare the impact of a local newspaper to the impact of a national newspaper. The audience size is also variable in the case of TV, radio and digital press.

Figure 2A comparison between the temporal evolution of the cumulative number of articles and the cumulative audience size of printed press. The differences between the two cumulative curves are due to the fact that different newspapers have different impacts in terms of audience size; hence, the impact of an article is variable according to the newspaper that publishes it. In order to highlight this variability we have overlapped the first and the last values of the two curves.

Download

The frequency of press news items and the audience size are two RCIs that allow identification of the population that has received a specific message. This is a necessary step to evaluate the communication effects on citizens' perceptions and urban community resilience. The RCIs employed in this experiment also allow observation of how the resonance of a message evolves over time (Figs. 1 and 2) and identification of possible correlations with other resilience drivers (e.g. a meteorological event, as is shown in Fig. 1). With respect to the resilience goals of the project, the experiment has shown that press relations allowed increased media attention on urban flood-prone areas and on emerging scientific and technological solutions. We finally should observe that an aggregated analysis of press news items does not give any insight into the quality of the communication contents.

5.2 Questionnaire administered to the visitors of an exhibition

The experiment presented in this section illustrates how RCIs based on a questionnaire can capture the quality of communication and how the audience has interpreted and perceived a message. Indeed, even if a communication activity reaches a wide public, the communication impact on urban resilience will vary according to the way the message is perceived. Questionnaire questions, such as those presented in this experiment, provide variables (e.g. frequency of correct questions, frequency of high risk perception) that can be used as RCIs to assess the respondent's interpretation and perception.

An exhibition dedicated to the RainGain project was held at École des Ponts in April–May 2014. It included five panels (84 cm × 120 cm) with texts and images on the project and the weather radar that was going to be installed beside the school, at the end of 2014. The subject of the exhibition was very technical: only professionals in the field of weather forecasts and flood risk management are knowledgeable about it and have access to other sources of information on the topic. The questionnaire was distributed to the workers and students of the school 1 month after the end of the exhibition. The questionnaire aimed at exploring how the exhibition contents were interpreted, memorised after several weeks, and whether it changed the visitors' perception of RainGain (Persoz, 2014). A total of 37 respondents were recruited on a voluntary basis among the 513 workers and 827 students of École des Ponts (all domiciled in the Paris region). They were invited through internal mailing to complete an online questionnaire (see Supplement S1, for a detailed list of the questions). The sample was expected to be small, since no monetary incentive was provided for questionnaire participation and there was no examiner who could individually reach each potential respondent to solicit his answers. We were also aware that questionnaires give limited insights into the cognitive processes that shape individual and social perceptions. However, rather than obtaining results that can be generalised to a wider population, our main objective was to test if quantitative research can be employed to evaluate the quality of communication. Indeed, this method and the research technique presented in the previous experiment have a common characteristic: they provide numerical data that are adequate to integrate communication assessment in a wider urban resilience assessment.

The questionnaire included questions on the professional background of the respondents. These questions allowed six experts from the sample to be excluded, in order to obtain a relative homogeneity in terms of background knowledge. As a result, the final sample consisted of 31 respondents (see Supplement S1 for demographic data). Other questions were aimed at identifying the source of information through which the respondents learnt about the project. On the basis of these questions the sample was divided into four subsets: (1) 13 visitors to the exhibition; (2) five visitors who also read the brochure distributed at the exhibition; (3) six respondents who received only informal information (by word of mouth⁴); (4) 12 participants who never heard about the project. In order to perform a comparative experiment, subset no. 1 was considered the experimental group (with 13 respondents); subsets nos. 3 and 4 were considered the control group (with a total number of 18 respondents). We have used the Fisher's exact test to compare the answers by the experimental group with those of the control group.

Figure 3The answers to three of the questionnaire questions on the RainGain exhibition held in April–May 2014. 100 % corresponds to the total number of respondents included in each subset: 31 respondents in the first row, 13 respondents in the second row, 5 respondents in the third row, and so on.

Download

Figure 3a shows that the number of respondents who visited the exhibition and have ticked the correct option for the question “what is the spatial scale of the weather data provided by the radar?” is 23 % higher than in the control group. As it appears in Fig. 3b, the wrong responses to the question “what are the advantages of X band weather radars compared to C band and S band radars?” are 20 % less frequent among the exhibition visitors. According to the results presented in Fig. 3c, the number of respondents who visited the exhibition and have provided a wrong response to the question “why is it important to measure precipitations at small scale?” is 15 % lower than in the control group. The discrepancy between the visitors' results and the control group results is between 15 % and 23 % and it provides an approximate indication of the impact of the exhibition in terms of knowledge dissemination.

An unexpected result concerns the responses of the respondents who read the brochure at the exhibition in Fig. 3a and c. In Fig. 3a the rate of correct responses of the respondents who read the brochure is lower (60 %) than in the experimental group (73 %). Figure 3c shows that the rate of wrong answers by the respondents who read the brochures is surprisingly high (40 %): it is close to the rate of wrong answers by the respondents who never heard about the project (42 %). We could assume that the respondents who picked the brochure spent little time reading the exhibition panels and that part of the brochure information was not didactic or suitable enough for the general public.

Figure 3c highlights another interesting result: the lowest rate of wrong answers corresponds to the group of respondents who did not attend the exhibition but heard about the project by word of mouth. We can suppose that face-to-face communication can strongly reinforce transmission of highly technical information.

To sum up, the answer rates, displayed in Fig. 3, show that the exhibition had a modest positive effect on the respondents' awareness about a flood resilience project, the background environmental issues and the solutions being developed. We suppose that this effect was reinforced by word of mouth communication, but was also weakened by the brochure.

Figure 4 presents the answers to a questionnaire question aimed at evaluating the risk perception and project acceptance of the respondents who visited the exhibition. The results show that the exhibition and the brochure, i.e. formal and official information, helped to reassure the respondents on security issues and encouraged them to support the implementation of the flood resilience project. Word of mouth communication did not have such a positive effect as formal information, but neither did it compromise the achievement of the project goals.

The Fisher Exact test⁵ was applied to the results of the four questions: p values are not significant, as these are always greater than 0.05 (the conventionally accepted significance level). Hence, the test confirms that, because of the small size of the sample, the differences between the answers by the experimental group and of the control group are not statistically significant.

Figure 4Answers to a questionnaire question evaluating the risk perception of the visitors after the RainGain exhibition. 100 % corresponds to the total number of respondents included in each subset: 31 respondents in the first row, 13 respondents in the second row, 5 respondents in the third row, and so on.

Download

5.3 Interviews

While questionnaires with closed-ended questions allow the results to be quantified, interviews can reveal more insights into the reactions and reasoning of the respondents. These research techniques do not provide quantitative variables that can be used as RCIs. Nevertheless, this is a helpful evaluation method to be adopted for exploratory studies. Two assessments based on open-ended questions were conducted during the RainGain project in relation to outreach activities.

One of the achievements of RainGain has been the inauguration of a new high-resolution weather radar at École des Ponts during the international conference “Researchers & water managers preparing cities for a changing climate” (8 and 9 June 2015). The promotion of this event has involved a wide range of outreach activities and means. One of the pieces of promotional content, which was produced on this occasion, is a short video (Mulard et al., 2015). It shows the installation of the radar, highlights the importance of this device in terms of research and innovation, and invites the audience to attend the conference. The video is mainly addressed to the students and workers of École des Ponts, since the school is located in front of the radar site. Jeannine Courtot, the manager of the school café and a charismatic and well-known figure on the campus, was involved as the speaker of the video. While the video was broadcast on YouTube and on the school screens, two women and two men were interviewed (they were all university students, aged between 20 and 23 years and domiciled in the Paris region). The questions, displayed in Table 1, aimed to appraise what kind of information the audience expected, how they interpreted the video contents, and how they reacted to the interview questions. The respondents were selected from the list of students invited to the conference and they participated in the interview on a voluntary basis.

Table 1Questions to the audience of the video “Jeannine presents the radar” (Mulard, 2015).

Download Print Version | Download XLSX

The video was appreciated by the respondents who found it “catchy” thanks to its dynamic pacing and the charisma of the speaker. They have also found the images of the radar installation interesting. However, the respondents, who were all engineering school students, expected more information about the radar functioning and its concrete applications. They were curious about the extent of the implementation of the project (“is the radar already operational?”, “how many new radars will be installed in Europe?”, “you should include a map with the pilot sites in the video.”) and the radar functioning (“does the radar allow predicting the rainfall volume?”, “it would have been nice to see some radar images.”). They also wondered about the researchers and engineers that operate the radar, the services that can be developed with these new weather data (“is it used only for weather forecasts?”, “is it possible to use it for Roland-Garros?”). These results will be used to design new questionnaires addressed to students from an engineering school. In particular, it appears relevant to include questions that make the link with their professional interests and that are accurately tailored to their background knowledge.

A similar assessment, based on three open-ended questions (Table 2; see Supplement S2 for the respondents' answers), was undertaken in November 2015 to evaluate the impact of a workshop on RainGain (held during the Provin Climate Forum). The respondents were all the participants of the workshop: 18 pupils (10 girls and 8 boys), all aged between eight and nine years and resident in the Paris region, who had been invited by the forum organisers. We chose the snapshot interviews (Fogg Rogers et al., 2015) as an investigation method since it is an alternative technique that is appropriate for a young audience and the context of a forum. The assessment highlighted that the audience enjoyed and well remembered a manual activity on rainfall observation in which they were active participants. It also revealed that the third question was misunderstood. Its purpose was to assess the clarity and exhaustiveness of the communication contents, but the respondents interpreted it as a question testing their learning capacities. This result suggests that the questions addressed to a young audience should be formulated in such a way that the respondents do not feel like they are being examined.

Table 2Questions to the participants of the RainGain workshop, conceived and held by Auguste Gires in the framework of the Provin Forum (November 2015).

Download Print Version | Download XLSX