Volume index - Journal index - Article index - Map ---- Back


Comunicar Journal 51: E-Innovation in Higher Education (Vol. 25 - 2017)

Online Questionnaires Use with Automatic Feedback for e-Innovation in University Students

https://doi.org/10.3916/C51-2017-05

Ana Remesal

Rosa M. Colomina

Teresa Mauri

M. José Rochera

Abstract

Technological tools have permeated higher education programs. However, their mere introduction does not guarantee instructional quality. This article presents the results of an innovation project aimed at fostering autonomous learning among students at a Pre-School and Primary Teacher Grade. For one semester all freshmen students used a system for autonomous learning embedded in the institutional online platform (Moodle), which included automatic formative feedback. The system was part of a complex formative assessment program. We present results of the experience concerning two aspects: the students’ actual use of the system, and their final appraisal of it. The quantitative descriptive analysis focuses on the students’ perspective to evaluate the adequacy of the instructional decisions. Results indicate that students need certain limits to be able to manage their learning better if we pursue the quality of innovation. These limits refer mainly to the time of accessibility and the limitation of attempts of practice. With respect to time, an appropriate span of time (neither too long nor too short) must be chosen; with respect to the number of attempts, it is expedient to limit rather than promote free endless access.

Keywords

Autonomous learning, formative evaluation, automatic feedback, higher education, online questionnaires, blended learning, innovation, students’ perspective

PDF file in Spanish

PDF file in English

1. Introduction

This work presents teaching innovation results concerning the use of online questionnaires to promote autonomous learning among university students. A teaching team with a long trajectory of collaborative work at different Teacher Degrees developed the innovation project. This joint trajectory allowed elaborating a shared comprehension of teaching, learning and assessment processes, of the incorporation of new information technologies (ICT) through online learning management systems, and of the resources and materials that the team designed. This shared vision of the teaching team affords innovation which requires a high degree of dedication and would be mostly inaccessible for individual teachers. The continued teaching collaboration is one of the keys to joint professional development (Mauri, Clarà, Ginesta, & Colomina, 2013). Through this article, we want to share our experience of this innovation to contribute to the collective reflection on university teaching, towards reflective and sustainable practice (Guskey, 2002).

The innovation project focused on the search for educational strategies to foster autonomous learning in a blended context. Previous studies report results that highlight the importance of promoting an active use of ICT to support learning efficiently (Collis & Moonen, 2011). This efficient use does not only depend on technological features of the instrument, but also on pedagogical decisions which ground it and allow transforming traditional practices (Coll, Mauri, & Onrubia, 2008). One of the key transforming axes, promoted by the European Space of Higher Education, is autonomous and self-regulated learning to form competent learners for the 21st century (Cernuda, Gayo, Vinuesa, & al. 2005). In this context, assessment emerges as an essential ingredient of this desired change, and particularly, formative and continuous assessment (Coll, Mauri, & Rochera, 2012; Sánchez-Santamaría, 2011). Hence, there is also an increase of interest in seeing how ICT efficiently contributes to these assessment processes.

1.1. Innovation: autonomous learning as a goal

This study is grounded on three complementary antecedents. First, the need for teaching assistance for learning in virtual and blended contexts (Coll, Mauri, & Onrubia, 2008). Second, feedback as the nexus between learning and assessment to offer this teaching assistance (Carless, Salter, Yang, & Lam, 2011; Nicol & Mcfarlane-Dick, 2006). And third, the potential of online questionnaires with formative feedback as a situated and contextualised task (Guo, Palmer-Brown, Lee, & Cai, 2014).

Innovation implies complex and demanding processes for instructors, which need to be evaluated. When instructors observe that the changes introduced in their practice produce an improvement in learning, a shift in attitude towards innovation and beliefs about it follows, and not the other way around (Guskey, 2002). From the socio-cultural perspective of teaching and learning, propounding good practices for innovation with ICT in higher education involves the understanding that placing the student at the centre of the process leads us to take three aspects into account. First, the students need to be mentally active to learn. Second, they should participate in as “authentic” and contextualized tasks as possible. And third, they should receive the instructor's guidance (or help from classmates) in the process. Thus, placing the student at the centre is not at odds with defining the instructor's role as a “necessary guide”, beyond the role of facilitator of learning with ICT (Coll, Mauri, & Onrubia, 2008).

1.2. Formative assessment and automatic feedback

One of the challenges to improving autonomous learning lies in the resources to monitor the learning progress and offer feedback. In that sense, feedback is key according to certain conditions of implementation (Carless, Salter, Yang, & Lam, 2011).

Expectations put on ICT are not fully met because the overflow of data frequently exceeds the capacity of instructors in terms both of time and dedication. Therefore, we need to look for strategies which will make assessment, and specifically feedback, into sustainable processes for all parties. Moreover, recent studies highlight that the tasks designed to support autonomous learning and its assessment increase efficiency as they drip into the general teaching plan. The set of assessment tasks comprises then an "assessment system" or "assessment program" (Mauri, Ginesta, & Rochera, 2016). In this context, feedback on online tasks is an element of confluence to support both autonomous and self-regulated learning, and the assessment of learning results and processes (Hattie & Timperley, 2007; Hatziapostolou & Paraskakis, 2010; Shute, 2008). Thus, to enhance the effectiveness of autonomous learning, it is necessary to consider three relevant features of feedback (Carless, & al., 2011; Mauri, Ginesta, & Rochera, 2016). First, feedback must be written, specific and clear. Second, it must come at the appropriate time. And third, it must inform about reasonable steps for the student to take afterward (feed-up, feed-forward).

1.3. Online questionnaires and formative feedback

Online questionnaires have become a commonly used “ICT resource” for learning and assessment (although they are not the only resource). They are instruments requiring hard work in their preparation (Morales, 2012; Moreno, Martínez, & Muñiz, 2015; Rodríguez Garcés, Muñoz, & Castillo, 2014). Among the advantages one can count is that they offer objectivity and rigour; they also guarantee reliability for the measure of learning setting one and the same standard for all students. Also, they provide immediate results. Finally, they make the monitoring sustainable for instructors, avoiding likely errors or biases of correction, among other advantages (Morales, 2012). We used, thus, this sort of instrument to offer the students formative feedback, immediate to their use of the questionnaire. The formative feedback differs from accreditative and verifying feedback in offering the student information which goes beyond providing the right answer or a numeric result. It offers hints about the error, on how to correct it, and also metacognitive suggestions which help to reflect about their own knowledge (Jolly & Boud, 2015; Williams, Brown, & Benson, 2015). In essence, the key is utilising feedback to make learning visible (Dysthe, Lillejord, Wasson, & Vines, 2011; Havnes, Smith, Dysthe, & Ludvigsen, 2012).

1.4. General and specific goals

The general goal of this work is to share with the teaching community the reflection about the efficiency of an innovation project. For this general purpose, the following specific goals are set:

• To investigate the reported use of a support system designed to promote autonomous learning.

• To characterise the actual use of this system.

• To analyse students’ appraisal of this support system.

• To explore the differences of autonomous study behaviour and learning results, according to particular pedagogical conditions.

2. Material and methods

2.1. Design of the innovation experience

The Project (founded by the Universidad de Barcelona) was carried out in a compulsory course of the Teacher Degrees (Pre-School and Primary School). The instructor team, composed of 13 instructors specialized in the area of Developmental and Educational Psychology, designed a bank of multiple choice questions with four response options and the respective feedback, referring to content on child development.

With this bank of questions the instructors built a set of questionnaires in the Moodle online platform (the institutional virtual campus). The questionnaires were piloted during one semester before the final implementation to revise and improve the questions, responses, and feedback, following the standard procedure for the elaboration of automatic multiple choice tests (Moreno, Martínez, & Muñiz, 2015).

Students accessed the questionnaires as part of a complex assessment program which included a wide variety of assessment activities distributed throughout the course (Coll, Mauri, & Rochera, 2012). The assessment program included the response to a test on contents related to children's development (motor, cognitive, communicative and socio-affective). The students sat the final test after two months of practice with the online questionnaires. The practice questionnaires had the twofold goal of 1) being an instrument for support of autonomous learning of factual and conceptual contents, and 2) preparing for the final summative assessment of these topics. The practice questionnaires had 40 questions and were organised in three levels of increasing complexity, which could be accessed accomplishing minimal conditions: an average mark of at least five over 10 points, and a delay lapse of 24h between attempts. In each resolution access, the system randomized the questions as well as the items of response within each question. Level 1 comprised four questionnaires (one for each development area); level 2 comprised six questionnaires (combining contents of two areas of development) and level 3 offered a single questionnaire which emulated the final exam conditions (40 random items with a time limit of 20 minutes).

Figure 1 shows the system of questionnaires for practice, as it ran along with face to face lectures. After every attempt, students got a numeric result with merely informative value, but without accrediting value. Also, in levels 1 and 2, they received an automatic formative feedback response to each of the responses they had selected, regardless of their being right or wrong. Feedback could present the following features (Guo, Palmer-Brown, Lee, & Cai, 2014; Hatziapostolou & Paraskakis, 2010):

In the case that the answer was wrong, they received motivational comments and tips for searching for the right information in the learning materials, for reflecting on the error, or on the concept or data at hand.

In the case that the answer was right, they received endorsement of the right answer.

2.2. Context and participants

Thirteen classes took part in the project, with 687 students of the second semester of the Teacher Degrees (Pre-school and Primary School), which constituted the whole population who started the Bachelor Degree in that academic course. The sample consisted of 88% women and 12% men; 50% were aged younger than 20, 43% were between 20 and 25 years old, 7% were older than 25 years old; 49% worked part-time, 5% worked full-time, and finally 46% did not work.

In this article, we present data and results of four groups with four different instructors and a total of 224 students. The selection of these four groups responds to a search of the best possible representativeness and displays the following conditions:

• Four groups without technical problems which did not report missing data.

• Two of these groups were of Pre-School teacher students and two of them were of Primary School teacher students.

• From each section, there was both one morning and one evening group.

• One of the groups, the control-group, followed the basic design for the use of questionnaires, while instructors in the other three groups altered the pedagogical conditions concerning the limit of individual access and the total time-span of access.

Specifically, the groups are as follows:

• Group Z (n=49), the basic design for reference: Access is delayed 24h between attempts; free time for response in levels 1 and 2, and time limited to 20 minutes in level 3; system freely accessible during two months (parallel to the usual face to face lecture).

• Group A (n=57), alternative design: System accessibility “constrained”, limited to one month (in parallel to the second month of usual face to face lectures).

• Group B (n=64), alternative design: System accessibility “overexpanded” to three months (opened one month in advance to the usual face to face lectures).

• Group C (n=54), alternative design: Basic system accessibility but "limited" to three individual attempts at level 1, one single attempt at level 2 and level 3.

According to the goals, the blended characteristics of the course and the methodological requirements usual in these situations, the collection of data focused on three different, complementary axis:

• Data of reported use: final, anonymous Likert questionnaire.

• Data of actual use: frequency of access for resolution of questionnaires in level 1 during the whole time defined for autonomous learning, collected utilising the automatic tracking of the online platform Moodle.

• Appraisal of the material and its use: final, anonymous Likert questionnaire.

In all cases, the participants’ consent was attained, the data of use were anonymised, as well as the final appraisal response, conserving merely the group code.

3. Analysis and results

We applied several complementary techniques of analysis. After a first descriptive analysis, data were contrasted statistically with the Kruskal-Wallis test, according to the characteristics of the sample and the data, to verify the existence of significant differences between the design conditions of each group of participants and the basic instructional design. We discarded contrasting variables such as gender and age because the sample was very biased in these aspects (in natural correspondence with the course and the degrees). In the following subsections, we present the procedure and results of each of the goals of our study.

3.1. Reported use

First of all, the students' responses to the questionnaire of reported study behaviour allowed us to draw the following robot sketch of their study method:

• Combining reading the materials and direct practice with questionnaires (61% fully or mostly agree).

• Individual use (82% fully or mostly agree).

• Taking notes of errors (91% fully or mostly agree).

• Taking notes of the right answer (86% fully or mostly agree).

• Taking notes of the feedback comments (feedback) (64% fully or mostly agree).

The analysis reveals significant differences between the basic design and two of the alternative groups.

Table 1 shows these differences, which refer to two aspects:

• Reading the learning materials in anticipation of practice (less frequent in group B, with “expanded” time).

• Using the questionnaires individually (more common in group C, with "limited" access).

In summary, we interpret that the group with the widest span of time (B) for the study, did access the learning support system more frequently with a strategy of trial–and-error practice, without a previous reading of the corresponding texts. In contrast, the group with limited access (C) must ponder the attempts more carefully, and hence they proceeded with a previous, off-line reading of the learning contents to maximize results. Hence, the design affected the study behaviour of these two groups of students in opposite directions.

3.2. Actual use

The same analysis in two phases was carried out for the actual use of the questionnaires. This analysis was performed only on data of level 1 of practice since it was focused on one single topic and presented new questions. In the following levels of practice, the students found the same questions again in combined questionnaires. We assume, thus, that a single attempt of response to the questionnaire does not provide evidence of feedback use (nor of its efficiency for improvement). We rather would need at least two attempts to potentially use feedback in an efficient manner (which benefit would increase with an increasing number of attempts). Thus, we proceeded to classify the attempts of resolution as follows:

• No-use: up to four accesses of resolution altogether (one per section of topics).

• Minimal use: up to eight accesses of resolution altogether (two per section of topics).

• Moderate use: up to twelve accesses of resolution altogether (three per section of topics).

• Frequent use: up to sixteen accesses of resolution altogether (four per section of topics).

• Very frequent use: more than sixteen accesses of resolution altogether (more than four per section of topics).

Table 2 shows these result (see Table 2).

Globally speaking, one has to reckon a limited use of the questionnaires. In group Z, with the basic instructional design, over one-third of the students accessed the questionnaires less than four times altogether, showing a null use of feedback (37%). Another third showed a minimal use of the questionnaires (33%), whereas merely 16% showed frequent to very frequent use. The use of questionnaires increases slightly among the groups with an alternative design. In fact, we find some possible effects of design on the study behaviour: the group with a reduced time of access (A) increases its use of feedback significantly (26% moderate, 31% frequent or very frequent, and only 14% of no use). Thus, it is an "intensive" practice carried out by those students. In group C, in contrast, we do not find a “very frequent” use due to the instructional restriction (maximal three attempts for each questionnaire), but the other three options are balanced (around 30% for each of them), and the “frequent” use is high compared with the other groups (three times greater). This suggests that the questionnaires are used purposefully in this group.

3.3. Appraisal of the support system and learning results

The final questionnaire included questions on two different levels of detail: a) about the general conditions to access the questionnaires and thus the feedback; b) on the diverse forms of feedback designed by the instructors. The corresponding results are presented in Tables 3 and 4. Eventually, Table 5 presents the final learning results.

3.3.1. Appraisal of general conditions

In Table 3 we can observe significant differences between two of the groups with alternative designs, in contrast with the basic design concerning four aspects: 1) the possibility of identifying doubts and errors; 2) difficulty; 3) sense of repetition; 4) motivating potential. The results show that in group C with constrained access, students consider the questionnaires altogether easier and more useful to identify doubts and errors.

Table 3 shows as well that the existence of three levels of practice was more positively valued by the students in group C, due to the chance of “organising the individual study” and “reducing tension before the test”. In contrast, in group B, with expanded time, students were less positive regarding the chances of organising individual study.

The 24h delay between attempts is, generally speaking, the least valued feature of the design. However, we still find a higher appraisal by students of group C, despite being low, as an organisation aid, for motivation and lowering of tension before the test.

3.3.2. Appraisal of specific conditions of feedback

For the specific conditions of feedback (Table 4), there are sharply significant differences in group A, with reduced time, where students perceived the tips as more "repetitive". In contrast, group C stands out again for positive evaluations of multiple aspects of the feedback. Those students value more highly the help provided for the identification of errors, first; second, they value feedback as more motivating, funnier, and more useful to reflect or laugh. Moreover, their appraisal is significantly lower when indicating the feedback as “confusing”.

3.3. Learning results

Finally, the learning results were collected via the online platform, through the final exam which was administered to assess learning (Table 5).

Results show that group C, again, stands out above the other groups with significantly better results, which allows us to interpret improved effectiveness of instructional conditions in that group.

4. Discussion and conclusions

The results of this innovation experience show that the most favorable conditions to foster autonomous learning of students focus on a moderate time span of system accessibility (two months versus just one or up to three) and a constraint to the number of attempts (versus no limits to individual attempts). Before discussing these results, it is important to note the data of significance reported in the previous section, related to the reported use of the students, their actual use, their appraisal of the support system, and eventually their results at the final exam. Very frequently, projects presenting assessment of innovation experiences limit themselves to reported use and posthoc appraisals (Gómez-Escalonilla, Santín, & Mathieu, 2011; Zaragoza, Luis-Pascual, & Manrique, 2009). In this sense, our work offers data of complementary triangulation that we consider indispensable in evaluating the processes of innovation from its very context of complex practice, which is even more important when the implementation of ICT allows tracking of the actual use. Concerning the time of accessibility as provided by design, it is noteworthy that the access occurs parallel to the face-to-face sessions which were dedicated to working through the learning contents (Figure 1).

In face-to-face sessions, the students had the chance (if they wanted to) to share doubts about the questions, responses, and feedback in the questionnaires. This allows supporting the autonomous learning of the students outside the classroom with the instructor's guidance. Following Carless and colleagues (2011), one of the quality criteria of feedback consists in it being a dialogue process with the instructor, not only received as a unidirectional message. In blended instructional proposals, the dialogue about feedback can proceed in a particularly adjusted manner in the face-to-face sessions. In fact, the blended component added to the learning processes has been identified in recent works as a desirable option to support the students' learning, even if one departs from a pure e-learning model. On the other hand, learning, as a process, needs time, and so we can interpret that the fast response of the questionnaires during just one month (group A) was not sufficient.

Concerning the number of attempts of resolving the practice questionnaires, from a critical point of view, one could consider that the innovation experience was not successful since the students' use is certainly light overall. However, a second reading allows some important conclusions for university teaching. To begin with, one of the benefits that are usually allotted to the online context is that of complete and free accessibility; the user is free to choose the moment and the location of access. Yet, the results of our work show that this is only partially true. It is true that students do not appreciate the restriction of 24h delay to access between attempts because they perceive it as a limitation of freedom of action. Nevertheless, now we know that an unrestricted limitless access, with as much time as possible –in other words, the absence of whatsoever restrictions to access–, did not imply a greater use of the instrument, nor did it throw better learning results. It seems sensible, thus, that it is not the absence of restrictions but the presence of certain conditional constraints which favors the organization and the autonomous regulation to self-adaption to these conditions, parallel to other contexts and requirements that join in time for students (parallel courses, personal work, family life). It is also important to consider the phenomenon of “false sense of security” (Petersen, Craig, & Denny, 2016), which may evolve with the use of instruments of multiple choice, jeopardising the final results, blurring actual learning, provoking in the user a cessation of practice ahead of time, thus limiting the practice really needed.

As for the characteristics of the automatic feedback included in the online questionnaires, this work also confirms that its potential to foster learning depends on the technological conditions of use, led by pedagogical criteria (Carless, 2006; Nicol, & Mcfarlane-Dick, 2006). The results of the use of feedback and the learning results of group C (two points higher than the other groups) highlight the importance of timely feedback, provided immediately following the response to the questionnaire. Feedback also should complete the summative information about the results (marks) with information to continue learning. Also, especially, feedback should endorse the reflection on performance; it should also sustain students' motivation to overcome difficulties, which is something feedback may address by means of humor, a fact only significant in the case of students with the highest competence (Del-Rey, 2002).

Finally, e-innovation can (and ought to) put the student at the centre but only with pedagogical scaffolds that adjust to the students’ need in their learning process. Not everything “doable” with technology contributes equally to this goal. The instructor’s interventions are essential as well as the improvement of processes through professional development.

Despite the fact that current institutional conditions privilege research activity above teaching efforts, it is also true that the university community values and optimizes day by day opportunities to discuss advances to confront the most urgent challenges that we face. The research presented on the use of practice questionnaires for autonomous learning with the help of more productive e-feedback, which might also allow instructors to monitor the processes in a sustainable way, is part of this broader goal of understanding and better tackling of the complexity of the teaching action with ICT in higher education. Obviously, online questionnaires with automatic feedback are not the only resource for e-innovation, but the work we present allows using them with pedagogical criteria of improvement. Future studies should aim at a deeper understanding of possible influencing variables such as gender, prior education before accessing graduate studies, and students' age, which was beyond the reach of this study due to the natural conditions of the sample.

Funding agency

Teaching innovation project «El uso de cuestionarios con feedback como instrumentos de evaluación para la autorregulación del aprendizaje en Psicología de la Educación en los grados de maestro», founded by the University of Barcelona (2014PID-UB/046).

References

Carless, D. (2006). Differing Perceptions in the Feedback Process. Studies in Higher Education, 31(2), 219-223. https://doi.org/10.1080/03075070600572132

Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing Sustainable Feedback Practices. Studies in Higher Education, 36(4), 395-407. https://doi.org/10.1080/03075071003642449

Cernuda, A., Gayo, D., Vinuesa, L., Fernández, A.M., & Luengo, M.C. (2005). Análisis de los hábitos de trabajo autónomo de los alumnos de cara al sistema de créditos ECTS. Jornadas sobre la Enseñanza Universitaria de la Informática, Madrid. (https://goo.gl/Z4hze5) (2016-06-25).

Coll, C., Mauri, T., & Onrubia, J. (2008). El análisis de los procesos de enseñanza y aprendizaje mediados por las TIC: una perspectiva constructivista. In E. Barberà, T. Mauri, & J. Onrubia (Eds.), La calidad educativa de la enseñanza basada en las TIC. Pautas e instrumentos de análisis (pp. 47-62). Barcelona: Graó.

Coll, C., Mauri, T., & Rochera, M.J. (2012). La práctica de evaluación como contexto para aprender a ser un aprendiz competente. Profesorado, 16(1), 49-59.

Collis, B., & Moonen, J. (2011). Flexibilidad en la educación superior: revisión de expectativas. [Flexibility in Higher Education: Revisiting Expectations]. Comunicar, 37, 15-25. https://doi.org/10.3916/C37-2011-02-01

Del-Rey, J. (2002). La risa, una actividad de la inteligencia. CIC, 7, 329. (https://goo.gl/WLrsT4) (2016-06-14).

Dysthe, O., Lillejord, S., Wasson, B., & Vines, A. (2011). Productive e-Feedback in Higher Education. Two Models and Some Critical Issues. In T.S. Ludvigsen, A. Lund, I. Rasmussen, & R. Säljö (Eds.), Learning across Sites. New Tools Infrastructures and Practices (pp. 243-258). New York: Routledge. (https://goo.gl/8P9ufZ) (2016-07-29).

Gómez-Escalonilla, G., Santín, M., & Mathieu, G. (2011). La educación universitaria online en el Periodismo desde la visión del estudiante [Students' Perspective Online College Education in the Field of Journalism]. Comunicar, 37, 73-80. https://doi.org/10.3916/C37-2011-02-07

Guo, R., Palmer-Brown, D., Lee, S.W., & Cai, F.F. (2014). Intelligent Diagnostic Feedback for Online Multiple-choice Questions. Artificial Intelligence Review, 42, 369-383. https://doi.org/10.1007/s10462-013-9419-6

Guskey, T.R. (2002). Professional Development and Teacher Change. Teachers and Teaching, 8, 3, 381-391. https://doi.org/10.1080/135406002100000512

Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77, 1, 81-112. https://doi.org/10.3102/003465430298487

Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the Impact of Formative Feedback on Student Learning Through an Online Feedback System. Electronic Journal of e-Learning, 8(2), 111-122. (https://goo.gl/4sPtbL) (2016-05-15).

Havnes, A., Smith, K., Dysthe, O., & Ludvigsen, K. (2012). Formative Assessment and Feedback: Making Learning Visible. Studies in Educational Evaluation, 38(1), 21-27. https://doi.org/10.1016/j.stueduc.2012.04.001

Jolly, B., & Boud, D. (2015). El feedback por escrito. Para qué sirve y cómo podemos hacerlo bien. In D. Boud, & E. Molloy (Eds.), El feedback en educación superior y profesional. Comprenderlo y hacerlo bien (pp. 131-152). Madrid: Narcea.

Mauri, T., Clarà, M., Ginesta, A., & Colomina, R. (2013). La contribución al aprendizaje en el lugar de trabajo de los equipos docentes universitarios. Un estudio exploratorio. Infancia y Aprendizaje, 36(3), 341-360. https://doi.org/10.1174/021037013807533025

Mauri, T., Ginesta, A., & Rochera, M.J. (2016). The Use of Feedback Systems to Improve Collaborative Text Writing: A Proposal for the Higher Education Context. Innovations in Education and Teaching International, 53(4), 411-424. (https://goo.gl/QGIO8u) (24-12-2016).

Morales, P. (2012). Análisis de ítems en las pruebas objetivas. Madrid: Universidad Pontificia Comillas.

Moreno, R., Martínez, R.J., & Muñiz, J. (2015). Guidelines based Validity Criteria for Development of Multiple-choice Items. Psicothema, 27(4), 388-394. (https://goo.gl/NyvRIi) (2016-05-20).

Nicol, D.J., & Macfarlane-Dick, D. (2006). Formative Assessment and Self-regulated Learning: a Model and Seven Principles of Good Feedback Practice. Studies in Higher Education, 31(2), 199-218. https://doi.org/10.1080/03075070600572090

Petersen, A., Craig, M., & Denny, P. (2016). Employing Multiple-Answer Multiple-choice Questions. ITiCSE '16 Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, 252-253. New York. https://doi.org/10.1145/2899415.2925503

Rodríguez-Garcés, C., Muñoz, M., & Castillo, V. (2014). Tests informatizados y su contribución a la acción evaluativa en educación. Red, 43, 1-17. (https://goo.gl/c7wkKx) (2016-12-24).

Sánchez-Santamaría, J. (2011). Evaluación de los aprendizajes universitarios: una comparación sobre sus posibilidades y limitaciones en el Espacio Europeo de Educación Superior. Revista de Formación e Innovación Educativa Universitaria, 4(1), 40-54. (https://goo.gl/ufYG3p) (2016-06-30).

Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78, 153-189. https://doi.org/10.3102/0034654307313795

Williams, B., Brown, T., & Benson, R. (2015). Feedback en entornos digitales. En D. Boud, & E. Molloy (Eds.), El feedback en educación superior y profesional. Comprenderlo y hacerlo bien (pp. 153-168). Madrid: Narcea.

Zaragoza, J., Luis-Pascual, J.C., & Manrique, J.C. (2009). Experiencias de innovación en docencia universitaria: resultados de la aplicación de sistemas de evaluación formativa. Revista de Docencia Universitaria, 4, 1-33. (https://goo.gl/O4KHqG) (2016-06-30).