Volume index - Journal index - Article index - Map ---- Back
E-learning, e-evaluation, assessment, videoconferencing, higher education, teaching method innovations, synchronous environments, qualitative research
There has been a remarkable increase in international studies on formative assessment, and the concept of assessment has aroused interest not only from a pedagogical point of view, but also from the strategic and even economic perspective, leading to a redefinition of the concept.
In our study we focus on online formative assessment, and our initial research reveals elements that are identical to those found in any assessment method and which have to be contextualized according to the specific learning situation to be observed, measured and improved. In their studies, Gikandi, Morrow & Davis (2011) state that assessment (whether formative or summative) in online learning contexts includes characteristics that differ from face-to-face contexts, especially due to the asynchronous nature of the participant’s interactivity, which means that educators must rethink pedagogy in virtual settings in order to achieve effective formative assessment strategies.
As we pointed out in a previous analysis (Blázquez & Alonso, 2006), prior to 2005 the most common topics in e-learning settings were the categorization of formative and summative assessments (Birnbaum, 2001; Wentling & Jonson, 1999), models such as the Input-Process-Output Model (Mehrotra & al., 2001) and others with similar elements (Stufflebeam, 2000; Rockwell & al., 2000; Potts & al., 2000; Forster & Washington, 2000; Moore & al., 2002).
More recent studies have focused on formative e-assessment which, as Rodríguez & Ibarra (2011:35) point out, «relies on the open, flexible and shared conception of knowledge, emphasizing the use of assessment strategies that promote and maximize the student’s formative opportunities». In this sense, Oosterhoff, Conrad & Ely (2008) stress the importance of formative assessment in online courses. In our context, Peñalosa (2010) states that to identify the progress of interactive and cognitive processes in formative virtual settings it is necessary to formulate a valid, sensitive strategy to assess performance, together with a series of tools that enable us to identify changes in the complexity of knowledge-building on the part of the students.
Weschke & Canipe (2010) present assessment guidance aimed at teachers, in which they highlight an interactive assessment process that uses indicators such as assessment of student courses, self-assessment, submitted activities and rubrics, all of which makes cooperative assessment more valuable for professional development.
More recent studies present two innovative tools for the assessment of virtual settings: eRubrics and videoconferences. Serrano & Cebrián (2011) are developing an eRubric system in Higher Education in which the student becomes the main assessor of the process. This implies a methodological change in the conception of the e-assessment agent who by tradition has always been the teacher. Regarding the use of videoconferencing, Cubo et al (2009) urge the installation of Virtual Synchronous Classrooms as learning environments in Spain, and Cabero & Prendes (2009) pointed out that initial assessment (debate on previous knowledge), processual assessment (monitoring the students’ interaction) and final assessment (oral presentations, oral exams…) could all be done via videoconference.
We consider it necessary to continue with the innovative educational proposal, since it is the formative methods and strategies that will establish the use of technology and not the other way round since, as Sancho (2011) states, technology in itself does not entail formative innovation and it can even reinforce conservative behaviour and discourage participation. E-learning is enabling teacher and students to explore new formative and collaborative methods with the flexibility that conventional educational structures are unable to offer.
Our interest in trying out an innovative formative e-assessment system led our research team to design a specific proposal for an online Secondary Education Specialist course aimed at graduates from Spanish, Portuguese and Latin American universities wishing to train as secondary education teachers. Our virtual learning proposal considered an environment that was designed to be flexible, based on access to original and varied sources, and with the students as active participants in their own training, accompanied by a team of coordinated and inter-functional teachers who shared responsibility for the process both individually and in a group setting. In this process the interaction formulas were negotiated and priority was given to problem solving, alternating individual and collaborative work, and offering a rich diversity of materials (media) and continuous formative and comprehensive assessment with dialogue as the main premise.
Throughout the course, the aim was for students to acquire the knowledge and skills necessary to reflect, analyze and criticize the main contents that shape the formative aspects of a secondary or middle school teaching in different Western countries and train them to carry out their functions at that level of education.
The constructivist approach implemented in the course was based on a virtual training design executed in the Moodle platform that combined a learning activity that fostered collaborative work with cooperation among students and between students and teachers. One of the training focal points was the learning and support-mentoring model. The tutor was responsible for the entire training process and assessment of the student who in turn received input from specialist teachers and continuous and distance assessment reinforced by self-assessment, co-assessment and interviews via videoconference.
The assessment and qualification system proposed for the course is a flexible model, adapted to the circumstances of the course and the students, where learning was assessed throughout the training process itself and included online tasks which were evaluated from the perspective of individual and group learning. All this was embodied in individual activities, collaborative activities, a final monograph project, interview via videoconference and in other issues such as active and quality participation.
For the interview via videoconference as assessment element, we used Adobe Connect’s Synchronous Virtual Classroom whose functions include, among others, online and live conferences between users. The meeting room includes several visualisation panels (pods) and components, and also allows several users, or meeting attendees, to share computer screens or files, chat, transfer live audio or video and participate in other interactive activities online.
With this context and the pedagogical experience stated, we aimed to carry out research to develop and check the viability of a formative assessment system for reliable online teaching with knowledge accreditation guaranteed, without the need for the physical presence of the higher education student. At the same time we aimed to:
• Collaborate in the innovation and development of e-learning as an educational change agent, particularly in line with the proposals for the European Higher Education Area.
• Experiment with the suitability of the Moodle open software platform for the students’ individual and collaborative work and for the final interview via videoconference as part of the online assessment we are testing.
The study examines assessment in virtual learning settings within a specialist university course for students who want to train as secondary education teachers. The distinguishing feature of the course for the students, together with the singularity of the proposed methodology and assessment evaluation ), is the basis of the qualitative research proposal; as Rodríguez, Gil & García (1999) assert, it is about studying reality in its natural context to give meaning or interpret the phenomena according to the meanings they have for those involved. This study aims to give special meaning to the subjective aspects of the actors of the action, and is interested in the impressions and observations of the participants who can deduce the theories inductively.
Following Rodríguez, Gil & García (1999), the research has been developed in four stages: a preliminary stage, fieldwork, an analytical stage and a informative stage. In this section we will describe the preliminary stage and the fieldwork with the results and discussion to follow.
• Preliminary stage: This stage is developed out of the research’s conceptual theoretical framework. The field to be studied is defined in this stage, together with the different stages of the qualitative research design and the description of the object to be studied, the triangulation, the data collection tools and techniques along with the analysis to be developed.
• Field work: This was the data collection, which was carried out at the beginning of the specialist course and after it had finished.
• Teachers or tutors are essentially characterized by their university and psycho-pedagogical training, so they are particularly familiar with teaching/learning models. There were eight participants.
• The students were graduates who had gained a variety of university degrees and who wanted to obtain a diploma to certify their psycho-pedagogical knowledge and training to educate secondary school students. Twenty students participated.
• The group of experts was made up of four teachers, highly specialised in teaching/learning systems. Three were specialists and members of distance learning institutions, two of whom were from the Open University (United Kingdom) and a third was from the UNED (Spain). The fourth member is a renowned Spanish expert in virtual training.
A constant throughout the process has been the rigorous methodology in the design and development of the study. This enabled us to generate evidential data, i.e., consolidate the research’s rigour and relevance. To do so we controlled the four concepts that according to Rodriguez, Gil & Garcia (1999) are essential: credibility, transfer, consistency and validation.
3. Results
To divide this study into units we followed a thematic approach that considered talks, events and activities taking place in the same situation studied and the possibility of finding segments that speak about a similar subject. This procedure enabled us to synthesize and create group units of meaning that match the study objectives.
For the identification and classification of items we devised a system of categories and sub-categories following a deductive-inductive classification. Deductively, since it was based on a previous research study (Alonso & Blázquez, 2009) that helped define the initial macro-categories. Inductively, because we then proceeded to devise new codes, categories and sub-categories from the recorded data. The final category system is described below (the sub-categories that arise from the study are described in Table 3):
For this stage we used the NVIVO qualitative data analysis software, which made the coding and analysis of the transcriptions or documents much easier. It helped us to store, organize and extract summarized reports of the most significant data emerging from the analysis, in addition to combine two dimensions in our analysis by integrating a narrative perspective and a more analytical one.
In this phase of the analysis process the results in Figure 1 are presented by category and technique, indicating which category was addressed in each technique and if the assessment was positive or negative:
Results that answer the data in Figure 1 are:
a) Open questionnaire for the students. In the General Aspects of the Assessment (GAA) category, the students surveyed (15 of the 17 questionnaires analyzed) were pleased overall with the type of assessment proposed: «From my point of view, the assessment includes all four dimensions of the assessment process, that is: prior design of the criteria, a comparison of information to obtain a balanced judgment, a decision-making process and the communication of results» (student).
Some students suggested changes in the qualification system motivated by purely subjective aspects. They define assessment as follows: «Continuous training has been very good and we have been able to discuss and develop it with thematic forums for each issue. The activities proposed and implemented have been useful to consolidate the acquired knowledge or to work with it in a slightly more practical way, not so theoretical. With the final project, we have been able to work with that acquired knowledge, while we have studied it more in depth and through which we have been able to demonstrate it, as with the final interview» (student, questionnaire).
Regarding the Assessment of Individual Activities (AIA), 12 of the 17 students surveyed valued these activities very highly for enhancing the teaching-learning process and, therefore, the assessment: «With the completion of activities you learn a lot, because to be able to carry them out you must study and understand the theory and then put it into practice. And that is how you best learn the content» (student).
Likewise, there are many references in the text in relation to continuous assessment, with students especially valuing this point (all 17 course students). In fact, this is why most of them consider it a good assessment system: «Overall I thought it was a good assessment system, taking into account the course’s characteristics. The system used has made us work on the content on a daily basis. With the individual and group activities, and participation in the forums, you can achieve such a goal» (student).
The students generally consider the individual activities requested by the teachers as a very effective tool to ensure the quality of the course’s assessment system. As for the Assessment of Group Activities (AGA), students mainly ignored this category, but those who answered (3 out of 17) were very positive in their judgment, as one student says: «With the final project I was able to consolidate the contents dealt with on the course, and to study in depth other related issues».
The students know that these activities do not work on their own and they point out as positive aspects both the maintenance of the platform where the activities were found and the tutoring by the course teachers-tutors.
In relation to the tools used for the videoconference (AVID), the students said their experience of the forums was positive, and the tools themselves were highly rated; once again they emphasized the continuous feedback from the teachers-tutors. Regarding the Assessment via Videoconference category, students want more eye contact throughout the course, with occasional videoconferences every so often.
b) Interview with a course student. In the assessment of Individual Activities Requested from the Student (AIA), the interviewee appreciates the fact that more value or weight is given to the activities since she considers that «it is the best way to learn».
As for the Assessment of Group Activities (AGA), the online chat experience was not rated very highly because the students found it difficult to establish effective written communication, and so the tool was not well-received. In Assessment via Videoconference (AVID), the student addressed the assessment issue to ensure that the person behind the screen and the person carrying out the exercises was really the person studying for the diploma, which is closely related to the reliability sub-category of General Aspects of the Assessment. In a very subjective manner the student says: «In my case the grade obtained is reliable, but I do not know if some people could be cheating, I hadn’t even thought about that. Yesterday I told my grandmother, that somebody could do another person’s exercises, but maybe I am not very clever and had not thought of that».
c) Focus group. In our analysis of the two focus groups set up with the course teachers-tutors the teachers rate the course highly in terms of the General Aspects of the Assessment (GAA). A recurring topic was that of assessment criteria, namely the percentage allocation criteria, as several of the teachers (6 out of 8 teachers-tutors) believe that the individual activities should have more weight due to the work done by the students and their personal participation.
Regarding the Assessment of Individual Activities Requested from the Student (AIA), the teachers-tutors admit that these activities enabled to them to make a more accurate assessment, since they helped them to get to the students they were evaluating: «What happens is that on a course like this, with so few students, and intense supervision, the assessment has been continuous. We carried out an extremely accurate assessment of who they were, what they were doing, why they were not doing it, why they were late submitting the activities» (course teacher-tutor).
As for the Assessment of Group Activities (AGA), the views varied: 60% of the teachers argue that no work was carried out 100% collaboratively, because the students simply divided up the work and then joined it together. Other teachers-tutors, however, stated that the group activities had truly given them the criteria to get to know the students better and evaluate them.
In Assessment via Videoconference (AVID), we found that some teachers (2 of the 8 tutors) were reluctant to use the videoconference systems. «They are not reliable, at least not for me (...) in my case, the entire course went great, but as I was telling tutor 2, for me the assessment... in fact, I had only two students, because two left, one via webcam, the other via telephone. The one over the phone, I questioned him about activities, about the subject, to see what had been done and how. Well, imagine when he could not answer me directly because the communication kept cutting off, and I called again and asked the same question. And with the webcam, exactly the same, it didn’t work...» (teacher-tutor). However, another group (65%) was inclined to use the videoconference systems as a communication and assessment tool, and were very satisfied with this completely virtual experience since they were able to accredit the knowledge acquired, emphasizing the need for more virtual interviews during the course.
Finally, the teachers said that this course had been the most innovative they had worked on, despite the need for more work to be done on certain pedagogical aspects.
d) Interviews with experts. Experts provided information on three of the categories, first on the Assessment of Individual Activities Requested from the Students (AIA), where they emphasize the importance of continuous assessment: «I think a continuous assessment of the individual and group contributions is essential in these settings» (assessment expert).
As for the Assessment of Group Activities Requested from the Students (AGA), the experts focus on the need for a good online chat system for synchronous communication and even for the final assessment.
In the Assessment via Videoconference (AVID), one of the experts focused on its reliability: «Many times in a face-to-face setting we tell the students to start creating an electronic portfolio, in the end they will submit a project or a research report. What we know is that the students have given it to us physically, but we don’t have the mechanism, we have it when we are talking to them. This can be done online, but you have to find another criterion and the criterion is that the student-tutor ratio cannot be very high» (expert). In addition, another expert clarifies that «If you are doing an online course properly you know your students well, and it would be impossible for them to ‘cheat’ in their work. The important thing is that if it is well-designed I assure you they do not lie» (expert).
Finally, experts have also called attention to the need to establish coherence between the training model and the assessment system, so that the necessary means to enable online and distance assessment are arbitrated whenever the formative model follows these parameters.
In this study we have presented an online assessment model that does not require the presence of the students, based on a constructive consideration of knowledge where learning can and should be assessed and evaluated throughout the training process itself, with tasks that can be assessed from the perspective of individual and group learning. The assessment of this study has helped us establish the following conclusions:
1) Progress has been made in achieving an innovative model for feasible and effective e-assessment, which ensures its use in distance higher education, which is valued as a highly beneficial contribution for those following e-learning models anchored in standard summative assessments, arising from traditional teaching processes.
2) It can be asserted that the assessment we propose is formative: it is part of a process and enables improvement throughout. We therefore follow the line of argument of Rodriguez & Ibarra (2011) who defend that e-assessment must be a learning opportunity designed to improve and promote meaningful learning and which is currently not employed in universities because their system continues to place emphasis on the teachers’ workload rather than in the students’ learning.
3) Most students consider the assessment followed as a highly motivating method since, in addition to the different techniques and tools used, the assessment is considered to be part of the teaching-learning process and not only an activity that takes place at the end of the course.
4) According to the degree of satisfaction of the students, experts and teachers, the results show that progress has been made in several key directions for a much more active teaching that relies less on memory and is more focused on the students’ workload and, importantly, with an assessment model that does not require a face-to-face setting. This corroborates the contributions made by Sloep & Berlanga (2011), who propose the creation of learning networks beyond the universities’ borders.
5) The results of this study bring together teaching innovation intended for universities with e-learning as agent for educational change. The use of interviews via videoconference is the most significant innovation in our study, although it could be improved as an assessment method since it still generates insecurities when used as an assessment tool. However, the experience is considered highly positive and, although we must continue to sharpen the technique, it seems we are on the right track. As suggested by Blázquez (2004) we must use these technologies to innovate and not to repeat ineffective traditional models, misusing synchronous resources (such as videoconference) at the expense of asynchronous resources (website, e-mail, discussion forums, etc.).
6) The formative assessment model tested provides regulated university activities for students who are unable to attend face-to-face final exams due to reasons of distance, which points to a huge potential educational market for our universities, especially in Latin American countries.
7) As noted by Solectic (2000), the specificity of the teaching materials demands a series of activities that help students put their resources, strategies and skills into practice, and which encourage them to participate in the knowledge-building process. This, from the beginning, was the purpose of the individual activities and, as we have established, the students have also understood individual activities in the same way, since they receive the highest rating as a continuous assessment standard.
8) In general, the implementation of this online assessment method has been positive, especially when focusing on continuous assessment throughout the activities, projects and interviews (via videoconference), together with a tutorial model that ensures the supervision of the student’s learning progress, enhanced by a manageable ratio of five students per tutor. In turn, this is reinforced by the distinctiveness of the pilot scheme in which the teachers were selected for their desire to participate and motivation.
9) There is always scope for improvement, with minor changes concerning the flexibility of the assessment activities and the singular valuation of collaborative activities, which have to go beyond simple task distribution. An increase in the number of interviews should also be encouraged, which in turn will give teachers greater confidence when dealing with Synchronous Virtual Classroom technology.
10) We conclude by pointing out that, based on qualitative transfer criteria, we encourage the teaching community to build similar scenarios to implement formative e-assessment processes and to follow the principles presented in the study.
Alonso, L. & Blázquez, F. (2009). Are the Functions of Teachers in E-learning and Face-to-Face Learning Environments Really Different? Educational Technology & Society, 12(4), 331-343.
Birnbaum, B.W. (2001). Foundations and Practices in the Use of Distance Education. Lewiston, NY: Mellen Press.
Blázquez, F. & Alonso, L. (2006). Aportaciones para la evaluación on-line. Tarraconensis, Edició Especial, 207-228.
Blázquez, F. (2004). Nuevas tecnologías y cambio educativo. In F. Salvador, J.L. Rodríguez & A. Bolívar (Eds.), Diccionario En ciclopédico de Didáctica (pp.345-353). Málaga: Aljibe.
Cabero, J. & Prendes, M.P. (2009). La videoconferencia. Apli caciones a los ámbitos educativo y empresarial. Sevilla: MAD.
Cubo, S., Alonso, L., Arias, J., Gutiérrez, P., Reis, A. & Yuste, R. (2009). Modelización didáctica-pedagógica, metodológica y tecnológica de las aulas virtuales: implantación en la Universidad de Extremadura. In J. Valverde (Ed.), Buenas prácticas educativas con TIC. Cáceres: SPUEX.
Forster, M. & Washington, E. (2000). A Model for Developing and Managing Distance Education Programs Using Interactive Video Technology. Journal of Social Work Education, 36(1), 147-59.
Gikandi, J.W., Morrow, D. & Davis, N.E. (2011). Online Formative Assessment in Higher Education: A Review of the literature. Computers & Education 57, 2.333-2.351.
Mehrotra, C.M., Hollister, C.D. & McGahey, L. (2001). Distance Learning: Principles for Effective Design, Delivery and Evaluation. Thousand Oaks, CA: Sage Publications.
Moore, M., Lockee, B. & Burton, J. (2002). Measuring Success: Evaluation Strategies for Distance Education. Educause Quarterly, 25(1), 20-26.
Oosterhoff, A., Conrad, R.M. & Ely, D.P. (2008). Assessing Learners Online. Upper Saddle River, NJ: Pearson Prentice Hall.
Peñalosa, E. (2010). Evaluación de los aprendizajes y estudio de la interactividad en entornos en línea: un modelo para la investigación. RIED, 13 (1), 17-38.
Potts, M.K. & Hagan, C.B. (2000). Going the Distance: Using Systems Theory to Design, Implement, and Evaluate a Distance Edu cation Program. Journal of Social Work Education, 36(1), 131-145
Rockwell, K., Furgason, J. & Marx, D.B. (2000). Research and Evaluation Needs for Distance Education: A Delphi Study. Online Journal of Distance Learning Administration, III.
Rodríguez, G. & Ibarra, M.S. (2011). e-Evaluación orientada al e-aprendizaje estratégico en educación superior. Madrid: Narcea.
Rodríguez, G., Gil, J. & García, E. (1999). Metodología de la investigación cualitativa. Málaga: Aljibe.
Sancho, J. (2011). Entrevista a Juana María Sancho Gil. En Edu cación y Tecnologías. Las voces de los expertos. Buenos Aires: An ses.
Serrano, J. & Cebrián, M. (2011). Study of the Impact on Student Learning Using the eRubric Tool and Peer Assessment. In Varios: Education in a Technological World: Communicating Current and Emerging Research and technological efforts. EDIT. Formatex Research Center (In press).
Sloep, P. & Berlanga, A. (2011). Redes de aprendizaje, aprendizaje en red. Comunicar, 37, XIX, 55-64. (DOI: 10.3916/C37-2011-02-05).
Solectic, A. (2000). La producción de materiales escritos en los programas de educación a distancia: problemas y desafíos. In L. Litwin (2000), La educación a distancia. Temas para el debate en una nueva agenda educativa. España: Amorrortu.
Stufflebeam, D.L. (2000). Guidelines for Developing Evaluation Checklists. (www.wmich.edu/evalctr/checklists) (15-12-2011).
Wentling, T. & Johnson, S. (1999). The Design and Develop ment of an Evaluation System for Online Instruction. www.universia.pr/cultura/videoconferencia.jsp. (30-09-2011).
Weschke, B. & Canipe, S. (2010). The Faculty Evaluation Process: The First Step in Fostering Professional Development in an Online University. Journal of College Teaching & Learning, 7(1), 45-58.