Volume index - Journal index - Article index - Map ---- Back
Citizen science, cocreation, codesign, knowledge, toolkit, interdisciplinarity, participation, open science
Citizen science represents a participatory research model that involves the public in scientific projects (Irwin, 1995; Hand, 2010; Gura, 2013), usually in data collection (Cohn, 2008) and, in some cases, in the collective interpretation of results (Delfanti, 2016). In the last decade, citizen science has received greater attention and acknowledgement in the academic literature (Follet & Strezov, 2015), in its development mainly in the natural and experimental sciences (Ferran-Ferrer, 2015), and it has transformed investigative methods applied in these fields (Wylie & al., 2014). The normal citizen science model considers collaboration between scientists and “amateur” participants as mere “contributory systems” (Wiggins & Crowston, 2015). Nevertheless, there is a growing number of cases involving greater collaboration on the part of the population at various stages of an investigation (Shirk & al., 2012; Delfanti, 2016), as also occurs in other collective knowledge-generation processes that adopt an open and innovative perspective (Yáñez-Figueroa & al., 2016). Follet and Strezov (2015) define citizen science projects according to the type of voluntary participation:
• Contributory projects: participants take part in data gathering, analyze the data at certain points in the project and help disseminate the results.
• Collaborative projects: as well as the above, the participants analyze samples and, on occasions, help designs the study, interpret data, draw conclusions or disseminate the results.
• Co-created projects: the participants collaborate in all stages of the project, including the definition of the questions, development of hypotheses, discussion of results and response to further questions that might arise.
Authors such as Bonney and others (2009a) point to the need to go beyond the contributory model of citizen science and involve the volunteer in the design process of the research in ways that are more deliberative and accessible. However, compared to the academic literature and resources generated around the contributory and collaborative modalities of citizen science, in the form of guides (Tweddle & al., 2012) or material for facilitation of this process (Bonney & al., 2009b), there is currently very little detailed information on the mechanisms used for the deliberate design of a co-created model of citizen science.
Apart from some pioneering experiences in techno-scientific participation, such as Public Lab (Wylie & al., 2014), conceptual frameworks for public involvement in scientific research (Shirk & al., 2012) and methodologies based on logical models for citizen participation (W.K. Kellogg Foundation, 2004), as opposed to other co-created knowledge-generation settings (Manzini & Coad, 2015) there are few practical resources available to facilitate the co-designing of research process; the exception is urban cartography experiences (Mindell & al., 2017).
This study analyzes how co-design can contribute to the idea that science can be made in collaboration with society. In our study, co-design is defined from an understanding of the co-created modality of citizen science as “participatory science” or “civic science” (Wylie & al., 2014), which encourages the appropriation of both the means that make it possible, and the knowledge generated as a result of a collective investigation. This approach connects with methodological and pragmatic challenges to develop a “co-production framework” or “language of co-production” in research, following the formulations of Jasanoff (2004), and what she terms the “participatory turn” in scientific studies (2003).
With this in mind, this study analyzes the co-design process in three collective experiments of citizen science directed by a team of scientists with experience in co-facilitating and analysing similar experiments in the public space (Sagarra & al., 2016) using collaborative and contributory modalities (Perelló & al., 2017). The case study, whose sequence is described in detail in the third section of this article, is based on an important conceptual difference in design thinking between “co-creation” (the generic process of collective creativity) and “co-design” (a set of specific participatory design techniques), the latter being a specific feature within the broader co-creation setting (Sanders & Stappers, 2008).
Our analysis addresses the following research questions:
• Can co-creation contribute to a more collaborative form of citizen science?
• How can science integrate social needs and concerns in its design and communication dynamics?
• How can interdisciplinary work be coordinated to construct knowledge collaboratively?
• How has knowledge been developed in this citizen science co-creation experience?
The case under analysis is part of the STEMForYouth (stem4youth.eu) initiative, an European project of the Horizon 2020 programme that aims to encourage young people to study science and technology at the university. Co-creation experiences were organized to design citizen science experiments with three groups of teenagers (95 in total) attending secondary schools in the Barcelona area (Spain) that covered a range of socio-demographic contexts.
To ensure that the research project was truly participatory and co-creative, the participants were involved at the start, from the design phase of the investigation. The co-design process of the experiments, based on a set of materials from a toolkit developed for the task, included collective agreement on the definition of the subject matter, the aims of the research and the research questions, and even the methods and logistics required to carry out the field work1.
This article evaluates this co-created design phase of the investigation, for which a survey and interviews were used to address the research questions posed in the study. These two methods were chosen for the exploratory nature of our study in this relatively novel framework of citizen science, following the example of other advances in this field (Bela & al., 2016). The key aspects covered by the questionnaire and interviews derive from a review of the literature on citizen science (Shirk & al., 2012) and on co-design processes (Sanders & Stappers, 2008), as shown in Table 1.
Following the co-design sessions for each of the three citizen science projects, an anonymous online questionnaire was sent to all the students who took part (a universe of 95 individuals aged 13 to 17 with an equal gender mix, of whom 79 responded to all the questions (81.4%).
Five semi-structured interviews were carried out with all the members of the research team, to support the survey data with an analysis of their perceptions of the interaction that took place during the co-design process. A content analysis of the interviews was made based on the categories presented in Table 1.
The researchers interviewed were: MC, the main researcher, male, aged 42; RS, researcher and project manager, female, 41; AC, researcher in training, female, 27; AF, tesearcher in training, male, 24; CP, researcher and designer, female, 32
Codification was done by two other researchers: one who had conducted the interviews (in this case, also acting as a facilitator of the co-design sessions), and another one who had not participated in the interviews or in the co-design process. Later, each category was tested for reliability to check the level of agreement between the two codifiers. In this study, the overall reliability (0.86) was higher than the indices recommended by Krippendorff (1990) and greater than the 0.80 (alpha) that enables solid and fundamental conclusions to be drawn beyond mere speculation.
A “design thinking” dynamic was used to achieve a co-created research design, in which interaction sequences between the different groups of participants were developed. The only premise for initiating the sessions was to describe a previous example of a citizen science experiment in a public space, as well as to focus the new experiment on an aspect of human behaviour.
A series of sessions took place in the three secondary school settings, with some slight variations and adaptions between each, which dealt with the co-designing for each experiment in four stages: (a) the problem to be addressed, (b) research questions, (c) conceptual diagram and (d) planning the tasks for executing the experiment (Table 2). There were 12 sessions in total, each lasting between one and two hours.
A toolkit was developed for use in the majority of the knowledge-generation dynamics. This key material2 was tested in preliminary versions and discussions during its use by the research team, to get a balance between usability and rigor, with the aim of producing a useful co-design toolkit for the collective generation of knowledge within a citizen science framework.
During the four stages, the use of the toolkit was guided by the research team acting as co-facilitators, to connect concepts and clarify doubts, while the main facilitator provided a framework for the work in order to achieve some informal yet specific ways to generate and present visual information, in accordance with participatory design practices (Kensing & Blomberg, 1998).
The aim of each session was to perform a divergence and convergence sequence (Brown & Katz, 2011). That is, to generate ideas and possibilities in a participatory way (a sequence of divergence: normally done by forming sub-groups) and a later coming-together to select options (convergence sequence): through idea-sharing and decision-taking mechanisms.
• Stage A: Identifying the collective problem to be addressed. Initially, to stimulate the use of a range of skills within each working subgroup (formed of 6-8 participants), it was proposed that the students select a badge to identify a role they wished to adopt from a set of investigator roles and profiles. Later, the students were invited to brainstorm types of problems for which an experiment on human behaviour could generate evidence of requiring actions to be taken for the improvement of a neighbourhood or city. The parameters used to reach a consensus within each group-class were concepts like the “viability” of the experiment, the “social impact” of the results or the “motivation” necessary to carry it out. Students’ opinions were posted on the walls and compared using thermometers.
• Stage B: Generation of research questions. For the co-creation of specific research questions, each subgroup used a template on which they could stick Post-its enabling them to complete at least three questions that started: What would happen if…? What is the relation between? How…? In this cooperative way, they completed predefined syntagms that, in modular form, contained the different research question options: descriptive, comparative and relational (Onwuegbuzie & Leech, 2006). Later, a moment of convergence based on discussion and the visual selection of the best options helped to filter the most relevant research questions for the group as a whole.
• Stage C: Conceptual diagram of the experiment. The third co-design stage took the research questions selected by each group to a more exploratory and creative level, linking a sequence of concepts around the experiment like action flows through a chronogram. This dynamic followed the premise that the investigation should be designed collectively from its initial steps to ensure the commitment and alignment of all those involved (Barnes & al., 2006). The participants chose icons from a wide range of images that reflected the key aspects of a potential experiment: research methods; logistics; key concepts or variables; participants; and other elements to visualize. These “dense diagrams” reopened debate and conversation about viability and motivation, and helped in the selection of an experiment co-design from among the various “finalists”.
• Stage D: Task planning and logistics. Based on the final selection of objectives and sequences (one related to perceptions of public space and infrastructure in the city, another to gender and discrimination, and a third experiment concept centered on inequality and immigration), each session aimed to move on from the co-design paradigm to preliminary planning. Here the participants dealt with the logistics and tasks required to execute each experiment, in this way ensuring scientific rigor by gathering data and obtaining relevant results for all the agents involved. In this instance, the toolkit provided a surface divided into columns like a basic “kanban” table, which made tasks that might have gone unnoticed both agile and explicit (Hines & al., 2004).
Each column focused on a category of tasks derived from the icons used in the prototype selected, in which the participants brainstormed ideas that they considered appropriate for an experiment (the performance of which, following the earlier participatory stages are seen in Table 2, took place in various public spaces).
We present the main results of the study based on the student survey and interviews with the team of scientists. These results link the research questions to the theoretical fundamentals and key concepts of citizen science and co-design (Table 1) following the co-design sequence in the four stages previously described (Table 2).
The researchers were convinced that student participation in the design of the experiment was vital from the first moment. In the interviews, they were critic with the role of the expert in citizen science (RS, MC) and expressed a desire “to make science truly participatory” (LD). They were initially concerned about whether the subject chosen by the participants would belong to a setting in which they, as researchers, were sufficiently experienced (RS).
Before beginning the co-design process, the researchers’ intention to boost the participation of other actors in the design of the investigation had given rise to doubts: drawing up research questions in collaborative fashion (RS) could be a more complex process than letting the researchers do it themselves (RS, LD); the complexity of not knowing how a co-creation experience could evolve and end (LD); the casuistry of the schools and the populations, which could at times make managing the activity more complex (MC, LD). However, after various co-design stages had been completed, there was a consensus that the initial expectations had been more than satisfied (RS, CP, MC), and that motivation was considerably higher when the non-expert was involved from the beginning (AF, LD). The high level of motivation and commitment achieved through co-creation is also reflected in the responses of the 79 participants to the survey (Figure 1, question Q11), and clearly connects with the scientists’ assessments, such as the “engagement of the citizens with citizen science projects is key for ensuring the success and sustainability of the projects” (RS).
The contribution to the research of the visual material in the co-design toolkit was also analyzed. The material was adapted to the needs of each phase of discussion (MC, CP), and fulfilled the main objective to provide a common language (LD) that reflected ideas that would later be selected (AF, RS). The material was considered essential by 49% of the student participant in conceiving the experiments, and 35% thought it was relevant for enabling the acquisition of new knowledge; 15% found it quite useful and 0% thought it was of no use (Q8).
In terms of the quality of the results, the interviewees stated that the investigation had been democratized (AF, CP) and it yielded perspectives that had not been considered before (RS), including unforeseen circumstances: “the students took a critical stance on many occasions, more than I expected” (LD). The participants expressed satisfaction with the scope of the definition of the experiment design (Figure 1, question Q5), and declared that the experience had been enriching, while also emphasizing (compared to other forms of research design) the challenge to maintain this spirit of co-creation and transparency alive in the following collaborative phases of production, execution and analysis of results (RS, LD, MC).
Despite discrepancies regarding the extent of definition in some of the final conceptual maps (CP, AF), or on the level of detail in the tasks to be undertaken that were identified collectively (MC), the interviews reveal that the co-designing done with the students produced themes, research questions and experiment preparation that were useful (LD) and, in some cases, contained a level of detail that was unexpected (RS).
The interviews with the research group show that the collaborative method described helped integrate the participants’ local concerns into the investigation (RS, LD, CP). For example, RS stated that “the design process arose when the themes were decided and a genuine concern emerged; the connection with local problems has been very clear”.
In the survey, most students agreed that they had been able to get involved by expressing their personal points of view (Figure 2, question Q1). The interviews also reveal that the level of involvement, when dealing with a subject close to their concerns and interests, increased student commitment to carrying out the experiment (RS, LD, AF). The students acknowledged the usefulness of the toolkit in discussing and contrasting their concerns (RS, CP), and how the result of the dynamics established to delimit the subjects of the experiments “was closely related to the way in which the participants perceive society and the problems of their surroundings” (CP).
The generation of an environment of credibility and mutual confidence was considered essential for the various stages of co-creation (CP, MC), since the dynamizing agent and the scientists could have been perceived as intruders in the classroom, which could have diminished motivation and contributions. The survey showed (Figure 2, question Q6) that the majority of students had no problem in freely expressing their opinions and only a few felt insecure.
Various interviewees considered that interdisciplinary work and the collaborative production of knowledge can be facilitated thanks to this type of co-design: “each can take a step back from their individual discipline and establish peer-to-peer dialogues” (RS), “many people with different viewpoints have generated knowledge together, beyond one single disciplinary field” (LD). In addition, sharing ideas was highly rated by the researchers after each session (RS, LD, CP), in which preliminary results were put in order, and they could try to predict the outcome of the next ones, thereby bringing coherence. The impression that the different phases of the co-design process were connected as an ordered sequence was confirmed by the majority of participants (Figure 3, question Q7).
It is also important to note that the team of scientists defined themselves as session co-facilitators exercising a support role to clear up doubts (LD, AF, CP), unblocking discussions that occurred in specific groups (RS) or making initial presentations to help students contextualize the investigation (MC). It is also relevant that the students did not appreciate any difference in the influence of the figure of the main dynamizing agent and that of the co-facilitators of the research group (Figure 3, question Q4). This understanding also underlines the interdisciplinary question and the importance of combining scientific knowledge and specific facilitation skills for co-creation.
The research team mainly agreed that these co-creation techniques can be transferred to any scientific project design (RS, LD) and can help to channel scientific discussion (AF); however, most recognize the need for some experience and competence in conducting the co-creation dynamics in citizen science. “In the end, it is a question of finding a balance between democratizing science and the experience of the scientists” (AF), and that in terms of interdisciplinary work, “the researchers did not establish a knowledge hierarchy over the students” (LD).
The collaborative development of knowledge was based on the crucial participation of the students. RS describes the process as a design “validated by the participants themselves”. In the relation between the team of scientists and the students, the former describe this experience as an adaptive process (RS, CP) that is highly flexible (LD) and eminently cyclical: “when you begin the sessions, you realize that is not such a good idea to be so linear; and if you allow them a certain amount of freedom and open up options, then new things can be introduced at the last minute. Allowing for some room for manoeuver is a good idea” (MC).
Regarding the materials, the scientific team considered that the combination of toolkit activities and their facilitation “generated debate and dialogue by integrating diversity through co-design, gathering different opinions and introducing them into the discussion and moments of reflection” (CP). This observation connects with the result in the survey for the question related to the quality of the participation: a clear majority of students agreed that the process allowed them moments for discussion and debate (Figure 4, question Q10).
Despite the fact that some interviewees referred to the complexity of carrying out collective decisions and of managing this requirement within the time limits of each session, which were perceived as very intense (RS, MC, AF), involvement with the co-facilitation dynamics and the associated toolkit materials meant that the entire process was more open (RS); another achievement was a “visualization of difficult concepts” (LD) and, in general terms, the necessary adaptation of the materials and mechanisms in order for decisions to be taken (AF, CP), which again was acknowledged in the survey (Figure 4, question Q9).
With the team of scientists agreeing that the following stages of the investigation required further processing of the co-design results (RS, MC, LD), another key aspect to emphasize is the recognition that the design of the experiment, as was intended, faithfully reflected the work performed by all the participants at all times, with no single influence prevailing at any time, with the team of scientists declining to adopt a position of power (LD, CP, AF). This perception was supported by the result in the survey (question Q3) in which 77% of those polled stated that the design of the experiment reflected the work carried out by all the participants in the work sessions with the team of scientists, against 23% who declared that the design was very much influenced by the team (and 0% who said the results were only the work of the scientific team). This connects to the recurring question of the degree of influence exerted by the experts during the sessions, about which CP states: the themes discussed were not influenced by the scientific team, which is very positive as the students could feel part of the process”.
Finally, another aspect that stands out was the team of scientists’ generalized perception that the co-design techniques applied here could be transferred to other forms of citizen science (RS, MC, LD) and even to other types of scientific research projects (RS, CP). In this sense, the co-design of the experiments can be understood as another experiment in itself, in this case one of participation and consensus generation (MC), and as a good initiation experience in the co-created model of citizen science (RS, LD) whose results enable an exploration of even greater levels of participation in the collaborative design of an investigation (CP).
By describing the process and analysing the results of this case study, we have tried to address the question of how collaboration in citizen science can be strengthened by co-created designs for investigation, attending to a wide range of interests and joining social and scientific objectives (Bonney & al., 2014). We describe the mechanisms that enable clear and specific objectives to be fixed for each experiment, identifying various possibilities by iterative design processes (Dickinson & al., 2012). For example, we describe how the research questions can be formulated as a process driven by the participants themselves, instead of the usual top-down schema dictated by the expert scientist (Newman & al., 2012).
The data obtained from the research questions enable us to draw the following conclusions:
• Co-creation, adopting visual material and participatory design techniques that allow the generation and selection of ideas provide quality results for a science that is more open to citizens, and which is more collaborative. In particular, co-creation is perceived as a fundamental factor in participants’ motivation and commitment, a key aspect in citizen science projects.
• Citizen science can integrate social needs and concerns into its design and communication dynamics if, at the start of the co-creation process, it can generate the actors’ trust in the process. Initiating the mechanisms for decision taking preliminary to any investigation is valued by the participants as an important aspect of successful integration.
• Good coordination of interdisciplinary work is very important for achieving good collaborative generation of knowledge. In this context, coordination requires coherent sequencing of the various co-design phases in which scientific experts fully integrate their expertise with roles of facilitation of group dynamics.
• A key question is a good balance relations of power during the entire process, ceding the initiative to the amateur participants in a structured way while retaining the role of scientific expert, but as guide and reference point at key moments, using as support mechanisms and material that generate reflection and debate.
The results suggest that the toolkit functioned well as a support of design techniques to integrate the diversity of viewpoints and opinions in visual form (Brown & Wyatt, 2015). As both the survey and interviews reveal, this material also encourages interdisciplinarity and can channel co-creation onto a structured visual canvas, something that, despite exceptions (Nagle & Sammon, 2016), constitutes an innovative contribution this study to the design of research processes.
Among the limitations and the need for greater analysis of this type of co-creation developed in this citizen science experience, it is important to mention complications arising from time management in the development of co-design by phases. A recurring comment in the interviews was the complexity of managing each session compared to traditional research design processes, in particular satisfactorily combining the moments when ideas are generated with collective decision taking. Also, certain deficiencies were detected in some co-design sequences during analysis such as in the initial identification of roles (not adequately applied when forming groups), or in the final phases in which the interface and protocol of the experiments were defined in greater detail. Future research that analyzes similar co-creation dynamics in the design of the investigation, whether in the citizen science environment or other settings involving public participation in knowledge, should consider these aspects when planning the development of co-design activities.
As well as the key questions posed at the start of this study, the responses of the team of scientists also suggest that this type of co-design can be extrapolated to scientific and academic interdisciplinary settings where the general public, the non-expert or so-called amateurs are absent from a terrain occupied by experts from various fields. In other words, the possibility of adopting similar co-creation dynamics for the design of research projects in professional teams with different scientific challenges.
1 It is important to underline that the analysis centers on the initial phase of the co-creation of these citizen science experiments, before the following phases of organization and subsequent execution of each of the experiments, which also count on direct student involvement.
2 The version of the toolkit used is available for consultation online or use by third parties, to promote the reproduction of the co-designing processes of experiments: https://goo.gl/xoU8vJ.
This study has received support from the European Union’s Horizon 2020 Reserach and Innovation Programme, under grant agreement, No. 710577 (StemForYouth), the Spain’s State Research Agency (AEI-MINEICO), and the European Regional Development Fund (FEDER) (FIS2016-78904-C3-2-P), from the Spanish Ministry of Science and Innovation project “Open Access to Science in Spain” (CSO2014-52830-P), as well as from the Dimmons group, IN3/UOC.
Barnes, T.A., Pashby, I.R., & Gibbons, A.M. (2006). Managing collaborative R&D projects development of a practical management tool. International Journal of Project Management, 24(5), 395-404. https://doi.org/10.1016/j.ijproman.2006.03.003
Bela, G., Peltola, T., Young, J. C., Balázs, B., Arpin, I., Pataki, G., ... & Keune, H. (2016). Learning and the transformative potential of citizen science. Conservation Biology, 30(5), 990-999. https://doi.org/10.1111/cobi.12762
Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., & Wilderman, C.C. (2009a). Public Participation in scientific research: Defining the field and assessing its potential for informal science education. A CAISE Inquiry Group Report. Online Submission. (https://goo.gl/wsZYQn)
Bonney, R., Cooper, C.B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K.V., & Shirk, J. (2009b). Citizen science: a developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11), 977-984. https://doi.org/10.1525/bio.2009.59.11.9
Bonney, R., Shirk, J.L., Phillips, T.B., Wiggins, A., Ballard, H.L., Miller-Rushing, A.J., & Parrish, J.K. (2014). Next steps for citizen science. Science, 343(6178), 1436-1437. https://doi.org.10.1126/science.1251554
Brown, T., & Katz, B. (2011). Change by design. Journal of Product Innovation Management, 28(3), 381-383. https://doi.org.10.1111/j.1540-5885.2011.00806
Brown, T., & Wyatt, J. (2015). Design thinking for social innovation. Annual Review of Policy Design, 3(1), 1-10. (https://goo.gl/1pgucj)
Cohn, J.P. (2008). Citizen science: Can volunteers do real research? BioScience, 58(3), 192-197. https://doi.org/10.1641/B580303
Delfanti, A. (2016). Users and peers. From citizen science to P2P science. Cell, 21, 01. https://doi.org/10.1641/B580303
Dickinson, J.L., Shirk, J., Bonter, D., Bonney, R., Crain, R.L., Martin, J., ... & Purcell, K. (2012). The current state of citizen science as a tool for ecological research and public engagement. Frontiers in Ecology and the Environment, 10(6), 291-297. In G. Papadopoulos (Ed.), Proceedings of the 9th International Conference KICSS. Limassol, Cyprus, 6-8 Nov 2014, 446-451. https://doi.org.10.1007/978-3-319-27478-2
Ferran-Ferrer, N. (2015). Volunteer participation in citizen science projects. El Profesional de la Información, 24, 6, 827-837. https://doi.org/10.3145/epi.2015.nov.15
Follett, R., & Strezov, V. (2015). An analysis of citizen science based research: Usage and publication patterns. PloS One, 10(11), e0143687. https://doi.org/10.1371/journal.pone.0143687
Gura, T. (2013). Citizen science: Amateur experts. Nature, 496(7444), 259-261. https://doi.org/10.1038/nj7444-259a
Hand, E. (2010). People power. Nature, 466(7307), 685-687. https://doi.org/10.1038/466685a
Hines, P., Holweg, M., & Rich, N. (2004). Learning to evolve: A review of contemporary lean thinking. International Journal of Operations & Production Management, 24(10), 994-1011. https://doi.org/10.1108/01443570410558049
Irwin, A. (1995). Citizen science: A study of people, expertise and sustainable development. London: Taylor & Francis. https://doi.org/10.4324/9780203202395
Jasanoff, S. (2003). Technologies of humility: Citizen participation in governing science. Minerva, 41(3), 223-244. https://doi.org/10.1023/A:1025557512320
Jasanoff, S. (Ed.). (2004). States of knowledge: The co-production of science and the social order. London: Routledge. (https://goo.gl/xBNB3C)
Kellogg, W.K. (2004). Using logic models to bring together planning, evaluation, and action: Logic model development guide. Battle Creek, Michigan: WK Kellogg Foundation. (https://goo.gl/pCPB52)
Kensing, F., & Blomberg, J. (1998). Participatory design: Issues and concerns. Computer Supported Cooperative Work (CSCW), 7(3-4), 167-185. https://doi.org/10.1023/A:1008689307411
Krippendorff, K. (1990). Metodología de análisis de contenido: teoría y práctica. Barcelona: Planeta.
Manzini, E., & Coad, R. (2015). Design, when everybody designs: An introduction to design for social innovation. Cambridge, MA: The Mit Press (https://goo.gl/FRJV3B)
Mindell, J.S., Jones, P., Vaughan, L., Haklay, M., Scholes, S., Anciaes, P., & Dhanani, A. (2017). Street Mobility Project: Toolkit. (https://goo.gl/yxDuXk)
Nagle, T., & Sammon, D. (2016). The development of a Design Research Canvas for data practitioners. Journal of Decision Systems, 25(sup1), 369-380. https://doi.org/10.1080/12460125.2016.1187386
Newman, G., Wiggins, A., Crall, A., Graham, E., Newman, S., & Crowston, K. (2012). The future of citizen science: Emerging technologies and shifting paradigms. Frontiers in Ecology and the Environment, 10(6), 298-304. https://doi.org/10.1890/110294
Onwuegbuzie, A.J., & Leech, N.L. (2006). Linking research questions to mixed methods data analysis procedures 1. The Qualitative Report, 11(3), 474-498. (https://goo.gl/dJZ7An)
Perelló, J., Ferran-Ferrer, N., Farré, S., & Bonhoure, I. (2017). Secondary school rubrics for citizen science projects. In C. Heredotouand, E. Scanlon, & M. Sharples, (Eds.), Citizen inquiry: Synthesising science and inquiry learning. London: Taylor and Francis. [In press].
Sagarra Pascual, O. J., Gutiérrez-Roig, M., Bonhoure, I., & Perelló, J. (2016). Citizen Science Practices for Computational Social Science Research: The Conceptualization of Pop-Up Experiments. Frontiers in Physics, 3(93), 1-14. https://doi.org/10.3389/fphy.2015.00093
Sanders, E. (2006). Design serving people. Cumulus Working Papers, 28-33. (https://goo.gl/JXqodm)
Sanders, E., & Stappers, P.J. (2008). Co-creation and the new landscapes of design. Co-design, 4(1), 5-18. http://doi.org/10.1080/15710880701875068
Shirk, J., Ballard, H., Wilderman, C., Phillips, T., Wiggins, A., Jordan, R., ... & Bonney, R. (2012). Public participation in scientific research: A framework for deliberate design. Ecology and Society, 17(2). https://doi.org/10.5751/ES-04705-170229
Tweddle, J.C., Robinson, L.D., Pocock, M.J.O., & Roy, H.E. (2012). Guide to citizen science: developing, implementing and evaluating citizen science to study biodiversity and the environment in the UK. London: Natural History Museum and NERC Centre for Ecology & Hydrology for UK-EOF. (https://goo.gl/9QSBHr)
Wiggins, A., & Crowston, K. (2015). Surveying the citizen science landscape. First Monday, 20(1). https://doi.org/10.5210/fm.v20i1.5520
Wylie, S.A., Jalbert, K., Dosemagen, S., & Ratto, M. (2014). Institutions for civic technoscience: How critical making is transforming environmental research. The Information Society, 30(2), 116-126. https://doi.org/10.1080/01972243.2014.875783
Yañez-Figueroa, J.A., Ramírez-Montoya, M.S., & García-Peñalvo, F.J. (2016). Systematic mapping of the literature: Social innovation laboratories for the collaborative construction of knowledge from the perspective of open innovation. In Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 795-803). ACM. https://doi.org/10.1145/3012430.3012609