Volume index - Journal index - Article index - Map ---- Back
Interaction analysis in virtual and hybrid learning environments is a complex issue, since it is necessary to go beyond a quantitative approach (number of messages) and obtain information about interaction dynamics in the context of educational activities. This article presents a set of interaction analysis strategies, which were designed during the development of a doctoral thesis in response to the two challenges identified: First, how can interaction be observed? And second, how can interaction be related to academic performance? The strategies designed provide elements for the analysis of educational activities, of asynchronous on-line discussions, of interaction representation and of the relationship between interaction and academic performance. For the analysis of educational activities, elements of sociocultural activity theory were used. For asynchronous on-line discussions, a content analysis of discussion transcripts was performed using a group of categories reflecting the knowledge construction process. Interaction was represented using the forograma technique, which is an alternative strategy for evaluating on-line discussion forums. The relationship between interaction and academic performance was established by comparing interaction dynamics and the academic performance results of the groups selected. Finally, an example is given to show how the strategies are applied.
Interaction, interaction analysis, hybrid environments, asynchronous discussions, forogramas, blended learning
Hybrid learning environments are those that combine face-to-face learning and instruction mediated by information and communication technologies (ICTs) (Reay, 2001; Rooney, 2003; Sands, 2002; Ward & LaBranche, 2003; Young, 2002; Osorio, 2010). Dziuban, Hartman and Moskal (2004) consider that this combination optimises both environments, provided that the best of both worlds is put to good use. This concurs with the results of a study by Hinojo, Aznar and Cáceres (2009) on students’ perception of this approach. It is important to explore new forms of interaction analysis in hybrid environments to ensure that they account for the quantity and, above all, the quality of participation, processes and conditions that favour knowledge construction (Gros & Silva, 2006; Meyer, 2004; De Weber, Schellens, Valcke & Van Keer, 2006; Rourke, Anderson, Garrison & Archer, 2001).
Interaction can be defined as cognitive and social actions among actors of the educational process (student-lecturer, student-student) while undertaking learning activities. Understood within cognitive and social frameworks, interaction requires an analysis of various aspects and several levels. According to Barberà and Badia (2004), interaction analysis should consider:
- An analysis of a joint activity in which a lecturer and students, and students alone, participate and interact while undertaking learning activities.
- In order to understand social interaction, account should be taken of the knowledge that is activated and produced by the group.
- While a joint activity includes various times and forms of interaction, various authors concur in recognising the value of asynchronous conversations as important expressions and manifestations of interaction. Thus, interaction analysis should be complemented with a careful analysis of asynchronous conversations.
This article presents a set of interaction analysis strategies designed during the development of a doctoral thesis, the purpose of which was to analyse interaction in hybrid learning environments on a case study of a hybrid learning postgraduate programme (Regional Development Management [RDM]) offered by a Colombian university.
In the context of the research undertaken, the approach to interaction analysis raised the following challenges: First, how can interaction be observed? And second, how can interaction be related to academic performance? The strategies employed to deal with these challenges are presented below.
According to Onrubia (2005), in order to analyse interaction, it is necessary to identify the joint activity in which it occurs so that the context and meaning of interactions can be recognised. It is also necessary to identify how asynchronous discussions occur in the context of educational activities, in the knowledge construction process (Barberà & Badia, 2004).In the case study of the RDM programme, the following strategies were employed to approach those two aspects:
a) Educational activity analysis: The design of learning activities was analysed in accordance with the components identified in activity theory (Jonassen & Ronrer, 1999). Recognition of an activity and its structure allows learning sequences to be identified in various hybrid learning space-times: face-to-face, e-learning and independent. In addition, bearing in mind that authentic educational activities foster the generation of spaces and times for individual and collaborative knowledge construction, the categories proposed by Oliver, Herrington and Reeves (2006) were used for the analysis of learning activities as authentic activities.
Analysing and, therefore, designing educational activities in accordance with the components of activity theory implies a revision of: The system: object, subject, mediating artefacts (instruments, resources), rules and division of labour (organisation and methodological design), activity structure (learning sequences), system dynamics (interactions).
In order to analyse activity authenticity, the criteria proposed by Oliver et al. (2006) were used. According to these authors, authentic activities: are relevant to the real world; are not very defined; students need to define the tasks and sub-tasks required to complete an activity; include complex tasks that cannot be undertaken over short periods of time; foster opportunities for students to examine a task from several perspectives; foster collaboration; promote reflection; go beyond a specific domain or result; are integrated into assessment; generate outcomes that have value in their own right; allow multiple solutions and diverse results.
This set of characteristics constitutes the categories on the basis of which learning activities can be analysed as authentic activities.
b) Asynchronous online discussion analysis
In accordance with various authors (Rourke et al, 2001; Schalk & Marcelo, 2010), content analysis is presented as a technique for analysing information obtained from transcripts of asynchronous discussion groups. In the literature, it is possible to find several publications on this issue, with different theoretical underpinnings and different conceptions of interaction (Henri, 1992; Zhu, 1996; Gunawardena, Lowe & Anderson, 1997; Garrison & Anderson, 2003).
In the context of the RDM programme case study, two sets of categories were used for the analysis of asynchronous conversations: those proposed by Gunawardena et al. (1997) and by Garrison and Anderson (2003). These were selected because their theoretical frameworks are based on knowledge construction. After applying the categories to several forums, it was found that they did not allow interaction dynamics and group work to be reflected, both of which are inherent to this set of courses. It was for this reason that a decision was taken to analyse interaction dynamics in order to identify the knowledge construction process that students followed on the RDM programme’s forums. The transcripts of 17 forums of seven programme subjects were analysed in order to identify the process that students followed for collaborative knowledge construction, and it was found that messages of the three types suggested by various authors could be identified in the groups’ interaction dynamics: affective/motivational, informative/organisational, academic/knowledge construction (Barberà & Badia, 2004).
When analysing the collaborative knowledge construction process on forums, a set of sub-categories emerged in the academic/knowledge construction category. These sub-categories allowed messages to be classified as follows:
- Isolated contribution: a participant makes a contribution to the group without establishing any relationship with other messages.
- Opinion contribution or comment on other participants’ contributions: this is when participants begin to read each others’ messages and to give their opinions on other participants’ contributions. These comments may be opinions, questions, replies or clarifications. The aim of this category is to reflect the process of a group’s dialogue and negotiation.
- Contribution collecting and summarising a group’s contributions: this is when, once a group has made its contributions and, in some cases, has had a discussion, one or several of the group’s members collect those contributions and generate a group outcome based on them.
- Contribution completing and enhancing a group’s construction: when a group has a collaborative construction outcome, this outcome goes through a process of enhancement through contributions made the group’s participants.
Following the protocol of analysis techniques for online discussions (Neuendorf, 2002; Rourke et al., 2001), the set of categories that emerged in the research was subjected to a process of validation by three researchers in order to identify the mean percentage agreement reached, which turned out to be 70%. This percentage agreement is considered acceptable and reliable for an analysis of asynchronous discussion content.
After managing to identify the interaction observation strategies, the challenge was to identify elements to represent those interactions in order to ensure that first they were comparable, and second that they could be related to academic performance results. To that end, the following strategies were employed:
a) Interaction representation. The forograma (Salazar, 2006) was used as a tool for discussion representation and analysis. The technique proposed by Salazar (2006) was adapted to the interests of the study. The main input for the elaboration of forogramas is asynchronous discussion transcripts. To begin the graphic representation, each discussion participant is represented by a circle with his or her initials inside it. Each contribution is represented by a circle around its author, with a line colour representing the message type, which is classified in accordance with the categories identified. Messages are organised chronologically in the forograma; an arrow pointing from the author to the message allows the contribution time and author to be identified. When a message is addressed to another participant, the author and addressee of the message are connected by an arrow. If a message is addressed to the whole group, it is represented by a horizontal line that encompasses all participants. In the forograma, a representation colour is associated with each of the categories identified. Below is a table of the conventions used in forogramas:
b) Relationship between interaction and academic performance. An analysis of asynchronous discussion interaction dynamics was performed on the basis of a comparative analysis of the groups’ forogramas. The comparison criteria emerge when analysing the results obtained in a set of forogramas, such as those aspects that discriminate and allow differences to be identified. The comparison criteria were:
- Quantity of messages: total number of a group’s messages.
- Classification of messages: quantity of messages, discriminated by the categories identified and by messages sent by the lecturer.
- Group work dynamics: an analysis was performed of forum progress, of organisation and interaction dynamics, and of the spokesperson role (in cases where this role was present).
- Times: time spent on undertaking an activity.
- Activity assessment: academic performance results achieved by a group while undertaking an activity.
Shown below is an example of the type of analysis performed in the context of the RDM programme case study. This example shows how each of the strategies described earlier was applied. The first two strategies allowed the activity design to be analysed, while the other two provided elements of analysis of the activity and, in particular, of times of greatest interaction while that activity was being undertaken, in order to relate interaction to academic performance. The example is based on an activity of subject S2, which forms part of the RDM programme.
Shown below are the general characteristics of the activity design (table 1), as is the action sequence when the activity was being undertaken in the face-to-face, e-learning and independent working spaces (table 2) of the hybrid environment. The activity was analysed on the basis of activity system components.
Below is the activity analysis in accordance with the categories proposed by Oliver et al. (2006) for authentic activities.
While undertaking the activity, the times of greatest student-student interaction and student-lecturer interaction were: the face-to-face session and times when there was group discussion of documents for each region.
In order to understand group work dynamics, two of the four groups were selected so that their asynchronous group discussions could be observed. The two groups selected corresponded to those that had the highest (group 1) and second lowest (group 2) grade for their respective group work outcomes. In this case, the group dynamics included the group spokesperson role, a specific role requested by the lecturer. The spokesperson was in charge of mobilising the group and guaranteeing the dynamics that would lead to the production of the group document. Shown below are the forogramas for the two groups.
The analysis based on forograma comparison criteria was as follows:
- Quantity of messages: Group 1 had a higher number of messages than group 2.
- Classification of messages: When observing the classification of messages, it was found that the two groups had a similar number of academic contributions. Regarding isolated contributions, group 1 had six messages of this type, while group 2 had three. Regarding outcome completion and enhancement contributions, group 1 had two messages of this type, while group 2 did not have any. There was a considerable difference in the quantity of informative/organisational messages; group 1 had 11 messages of this type, while group 2 had two. Each group had two affective/motivational messages.
- Group work dynamics: The spokesperson role was included in the instruction given to the groups. The spokesperson was in charge of facilitating, organising and summarising the group’s outcome. Between the two groups, there were major differences in these roles. The spokesperson for group 1 made several organisational contributions and proposed the group work dynamics. This spokesperson also took charge of the two versions of the summary documents. The spokesperson for group 2 only made one organisational contribution, and the group did not achieve good interaction for the production of the document.
- Times: Both groups had the same time available for undertaking the activity.
- Activity assessment: The following aspects of the activity were assessed: the group work outcome, the presentation in a face-to-face session, the spokesperson role.
Seeking the relationship between interaction and performance, the results of the two selected groups’ final outcomes (analysis documents) were taken; by doing so, the result was not related to the interaction dynamics that the groups managed to achieve. It was found that: the group with the highest rate of interaction and better group work dynamics obtained the highest grade (5 out of 5), while the group with the lowest rate of interaction and greater difficulties in group work dynamics obtained the lowest grade (3.85 out of 5).
During the course, an individual exam was held after the activity analysed; this provided individual evidence of performance. When correlating this assessment with participation in the group’s internal forum, a Pearson correlation coefficient of 0.76722215 was obtained. As this result shows, there is a significant correlation between the variables for participation in a group’s internal forum and the grade students obtained in the assessment done by the lecturer to check the attainment of learning objectives. Bearing in mind that the students’ participation was in the form of discussions, communication and group work, these data show the relationship between interaction and the attainment of learning objectives, and specifically between interaction and academic performance.
Recognising the interaction phenomenon in the context of educational activities, and not just the messages exchanged in discussions, allows a closer, more detailed approach to be taken to hybrid learning environments, as environments that promote interaction.
Interaction in the context of authentic activities can strengthen individual and collaborative knowledge construction, and thus, in turn, can generate the conditions necessary for greater learning and better academic performance results. However, for this to be potentially so, a prerequisite is the presence of certain conditions and characteristics in the design and implementation of such activities in order to ensure that the greatest advantage is taken, not only of hybrid environments, but also of certain student and lecturer practices and characteristics in group work dynamics.
Examining joint activity and all its components from a sociocultural activity theory viewpoint allows aspects that determine conditions for interaction development to be identified, such as: the ultimate goal of interaction (interaction outcome), the actors and roles involved, and the mediating artefacts (instruments, resources), as well as the dynamics or sequences before, during and after times of interaction, thus recognising the continuous process (between face-to-face and e-learning times) within which interaction occurs.
As set out in the theoretical underpinnings, online discussion analysis needs to go beyond a quantitative focus and allow an approach to the dynamics, to the whys and wherefores of interaction and interaction process results. It was necessary to have a discussion analysis mechanism in order to identify group work dynamics and, in particular, to find relationships between these dynamics and academic performance results.
The set of categories used allowed group interaction dynamics or processes to be recognised, which identified individual participation, negotiation and exchange, and a group’s progress towards group construction and synthesis. Following the protocol of analysis techniques for online discussions, the set of categories that emerged in the research was subjected to a process of validation by three researchers in order to identify the mean percentage agreement reached. A set of forums was selected and coded by the three researchers, and a percentage agreement of 70% was obtained. While these categories were constructed in the specific context of the case study, they may be an alternative for the analysis of knowledge construction processes in other contexts.
Forogramas allowed information about discussions at various levels to be represented graphically in a single schema: the individual recognition of participants, the type and quantity of contributions, the chronological progress of discussions, the senders and addressees of messages, the dynamics and progress of negotiations, and the process of group construction and synthesis. All of this information expands the possibilities for analysing, comparing and contrasting discussions. The forograma technique and the categories identified may be used in different contexts and other research projects.
A limitation of this discussion representation technique is that the graphic representation becomes very complex when the number of participants is higher than eight.
The forograma technique for representing online discussions, and the categories that emerged from the technique for analysing the content of a large number of forums, allowed virtual interaction processes to be analysed, comparisons between group dynamics to be made and the relationship between those dynamics and academic performance results to be identified.
The four strategies as a whole offer the potential to observe and analyse interaction dynamics in the context of educational activities, and to establish relationships between those dynamics and academic performance results.
Barberà, E. & Badia, A. (2004). Educar con aulas virtuales. Orientaciones para la innovación en el proceso de enseñanza y aprendizaje. Madrid: Machado Libros.
De Weber, B.; Schellens, T.; Valcke, M. & Van Keer H. (2006). Content Analysis Scheme to Analyze Transcript of Online Asynchronous Discussion Groups: A review. Computer & Education, 46; 6-28.
Dziuban, C.; Hartman, J. & Moskal, P. (2004). Blended Learning. EDUCAUSE Center for Applied Research, Re-search Bulletin, 7; 1-12.
Garrison, D.R. & Anderson, T. (2003). E-learning in the 21st Century. A Framework for Research and Practice. London: Routledge Falmer.
Gros, B. & Silva, J. (2006). El problema del análisis de las discusiones asincrónicas en el aprendizaje colaborativo mediado. Revista de Educación a Distancia, 16; 1-16. (www.um.es/ead/red/16) (04-06-2008).
Gunawardena, C.; Lowe, C. & Anderson, T. (1997). Interaction Analysis of a Global Online Debate and the De-velopment of a Constructivist Interaction Analysis Model for Computer Conferencing. Journal of Educational Computing Research, 17; 395-492.
Henri, F. (1992). Computer Conferencing and Content Analysis, in Kaye, A. (Ed.). Collaborative Learning through Computer Conferencing. The Najaden Papers, Springer-Verlag, Berlin.
Hinojo-Lucena, F.J.; Aznar-Díaz, I. & Cáceres-Reche, M.P. (2009). Percepciones del alumnado sobre el blended learning en la universidad. Comunicar, 33: 165-174.
Jonassen, D. & Ronrer, L. (1999). Activity Theory as a Framework for Designing Constructivist Learning Environ-ments. ETR&D, 47; 61-79.
Meyer, K. (2004). Evaluating Online Discussions: Four Different Frames of Analysis. Journal of Asynchronous Learning, 8(2); 101-114.
Neuendorf, K.A. (2002). The Content Analysis Guidebook. Thousand Oaks: Sage Publications.
Oliver, R.; Herringtron, J. & Reeves, T.C. (2006). Creating Authentic Learning Environments through Blended Learning Approaches. The Handbook of Blended Learning: Global Perspectives, Local Designs. San Francisco: Pfeif-fer, 502-515.
Onrubia, J. (2005). Aprender y enseñar en entornos virtuales: Actividad conjunta, ayuda pedagógica y construcción de conocimiento. Revista de Educación a Distancia, 4, Monograph 2;1-16
Osorio, L. (2010). Características de los ambientes híbridos de aprendizaje: estudio de caso de un programa de posgrado de la Universidad de los Andes. Revista de Universidad y Sociedad del Conocimiento, 7(1). (rusc.uoc.edu/ojs/index.php/rusc/article/view/v7n1_osorio/v7n1_osorio) (06-08-2010).
Reay, J. (2001). Blended Learning - a fusion for the future. Knowledge Management Review, 4(3); 6.
Rooney, J.E. (2003). Blending Learning Opportunities to Enhance Educational Programming and Meetings. Asso-ciation Management, 55(5); 26-32.
Rourke, L.; Anderson, T.; Garrison, D. & Archer, W. (2001). Methodological Issues in the Content Analysis of Computer Conference Transcripts. International Journal of Artificial Intelligence in Education, 12; 8-22.
Salazar, A. (2006). Forograma, una estrategia alternativa para la evaluación de espacios virtuales de discusión. Memorias VIII Congreso Colombiano de Informática Educativa. Cali, Colombia.
Sands, P. (2002). Inside Outside, Upside Downside: Strategies for Connecting Online and Face-to-face Instruction in Hybrid Courses. Teaching with Technology Today, 8; 6.
Schalk, A.E. & García, C. (2010). Análisis del discurso asíncrono en la calidad de los aprendizajes esperados Asynchronous. Comunicar, 35; 131-139.
Ward, J. & LaBranche, G.A. (2003). Blended Learning: The Convergence of e-Learning and Meetings. Franchising World, 35(4); 22-23.
Young, J.R. (2002). ‘Hybrid’ Teaching Seeks to End the Divide Between Traditional and Online Instruction. Chro-nicle of Higher Education; 48 (28), A33.
Zhu, E. (1996). Meaning Negotiation, Knowledge Construction and Mentoring in a Distance Learning Course. Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology. Indianapolis.