Central Institute for Distance Education Research - FernUniversität in Hagen
30.9.1998
Helmut Fritsch
DOCUMENT DONE, 1998.
see also:article by Bernath and Rubin in: INFORMATIK FORUM, Vol 12, No. 1, March 1998, p. 18 - 23;(ammendments after a comment of Ulrich Bernath (Sept.1998) in Italics!)
After last years work with the title "Host contacted, waiting for reply" this is the second approach to evaluate the virtual seminar by Eugene Rubin and Ulrich Bernath which took place from Jan- March 1998. There will be a comparison of questionnaire results and some more enquiries as to the communicative structure of such seminars.
More and more universities offer teaching in so called virtual seminars. Such seminars take place on the Internet, are not open to the public, lead to some kind of certification and are esteemed as the most prominent form of advanced communication among experts of teaching and learning. It is thought of as an appropriate form for further education or continuing education and training on the job. So in some cases these seminars are offered by teaching staff as an alternative for on-campus professional development and have to be paid for .The elements of such a seminar could be
- the readiness / experience to use the WWW for browsing and commenting
- restricted number of participants
- fixed curriculum within a time span
- contact with peers / experts and tutors
- a number of necessary own activities
- the readiness to contribute to the evaluation of the seminar itself .
A. Differences in my role
The first seminar of this series has been evaluated as a whole in 1997 the results have been presented in June 1997 at the ICDE world conference. Main finding was the didactical element of witness learning being more prominent than even „own activities" in the seminar and being rated as second important of all the didactic elements of such a seminar ( second after the seminar readings on the Web).The difference now is that I did not participate in the second seminar, neither as someone who had access to the mailings, nor as someone who helped the experts with technical issues in answering questions of the participants; nor as external evaluator who also participated in direct seminar sessions in Germany and USA. I only received the data from the final questionnaire. We still had the procedures to process these data into an Excel file and that is why we could present the final evaluation results a day after we got the data. We put these results on the web and there they are: http://www.fernuni-hagen.de/ZIFF/ev-ddfinal.htm
Since the results of the previous seminar are on the web, too, http://www.fernuni-hagen.de/ZIFF/evarep3.htm it should be possible for participants and the "web-public" to draw their own conclusions. Response numbers of last year changed a little bit after reminding participants of sending in their questionnaires, so in the www-version of the results we find N=28 - now we have data of N=33) All I want to do here is to draw my personal conclusions out of these data:B) Similarities of results
Already the first glance over the tables of the final questionnaire struck me - Are these the data of a new cohort of people? Is the questionnaire so bad, that it will not even give way to small differences? Now anyone may object to this procedure, since the instrument we used has never been tested in terms of measurement scales, reliability etc.! But in the first round there was no time to evaluate the instrument itself. At one point I proposed to confront new participants, people interested in such a seminar ( there had been a long waiting list!) with that instrument and then to make all the cross validation of the questionnaire items. We did not do this, so now we are in the position of interpreting data collected in a rather superficial way in order to get an impression about a seminar which took place for the second time - and is about to take place again. So when I was confronted with the similarities of answers of the two groups, I doubted in the first instance that the questionnaire would give sound information about this second special seminar.Will any group of participants of a virtual seminar answer the same questions in the same patterns? What will happen, if we send this questionnaire to future participants - we would have two studies for the same cohort ( ex-ante and ex-post questionnaire). Are these questions worth more effort to validate anything? My position when constructing the questionnaires for this seminar (in the first seminar we had three questionnaires which evolved parallel to the time schedule of the seminar itself) was, that it would not be worth the effort, since we do not have measurable learning goals with which to confront the results of the individual answers.
So now this is the situation:
We have data about two seminars which had been presented by the same people, with the same experts "on board", the same prescribed readings and with the same platform used for interaction: Two different cohorts answered the same questionnaire. The assumption is, that differences in the answering behaviour will indicate differences in the cohorts. The danger is that we do not know about the possible range of differences that might occur within any cohort, and that with such a small number of participants interpretation must not be generalized. Moreover this general reserve Ulrich Bernath indicated to me that " Although all mentioned aspects were in both seminars the same, there has been the difference that
a) in seminar #1 four weeks with four different experts were followed by week 5 and 10 for general and orientation matters and by week 7,8,9 for applications in DE
b) in seminar #2 eight weeks with the same four experts - and the experts came in in week2 of each module - were followed by only two weeks on applications, while the second week in parallel was the final week for wrapping up.
c) in the seminar #1 face-toface meetings took place; nothing of that kind in #2"
B.1 General objectives of the seminar
There had been 13 items out of which participants could pick any number and rate and assign to the item whether it had been reached or not: Both cohorts chose as the most prominent goal "experience a virtual seminar myself" There are only slight differences in the rating of the other elements - probably one "to learn how a virtual seminar is organized" (which has the highest rating of "reached" in the second cohort) tends to surmount the more academic goals like meeting peers or learning about theory. Anyhow, both groups are most interested in technical skills, in learning by experience. Here are the two lists:1998
1. General objectives of the virtual seminar
| When you joined the seminar, how important was it for you, to reach the following goals and which ones did you reach? | 1=most important | 3=not important | ||||||||
| 1 | 2 | 3 | reached | not reached | ||||||
| n | % | n | % | n | % | n | % | n | % | |
| Enlarge my knowledge about distance education. | 18 | 66,67 | 9 | 33,33 | 0 | 0,00 | 23 | 85,19 | 2 | 7,41 |
| Learn about different systems of DE. | 14 | 51,85 | 11 | 40,74 | 2 | 7,41 | 22 | 81,48 | 3 | 11,11 |
| Have direct contact to experts in DE. | 14 | 51,85 | 10 | 37,04 | 3 | 11,11 | 19 | 70,37 | 4 | 14,81 |
| Experience a virtual seminar myself. | 22 | 81,48 | 4 | 14,81 | 1 | 3,70 | 24 | 88,89 | 1 | 3,70 |
| Learn about the state of the art in DE-technology. | 13 | 48,15 | 9 | 33,33 | 4 | 14,81 | 17 | 62,96 | 8 | 29,63 |
| Learn about how a virtual seminar is organised. | 18 | 66,67 | 9 | 33,33 | 0 | 0,00 | 25 | 92,59 | 0 | 0,00 |
| Contribute to the field of DE-research. | 7 | 25,93 | 7 | 25,93 | 13 | 48,15 | 8 | 29,63 | 14 | 51,85 |
| Get rid of my personal critical questions about DE. | 1 | 3,70 | 9 | 33,33 | 16 | 59,26 | 11 | 40,74 | 10 | 37,04 |
| Get involved in the global perspective of DE. | 14 | 51,85 | 9 | 33,33 | 3 | 11,11 | 19 | 70,37 | 4 | 14,81 |
| Get in touch with colleagues. | 10 | 37,04 | 11 | 40,74 | 5 | 18,52 | 17 | 62,96 | 5 | 18,52 |
| Improve the level of my skills in DE. | 13 | 48,15 | 10 | 37,04 | 4 | 14,81 | 19 | 70,37 | 6 | 22,22 |
| Acquire skills that I could use in the near future. | 17 | 62,96 | 6 | 22,22 | 3 | 11,11 | 19 | 70,37 | 5 | 18,52 |
| I hoped to enrich my DE-related vocabulary. | 7 | 25,93 | 11 | 40,74 | 8 | 29,63 | 19 | 70,37 | 5 | 18,52 |
| When you joined the seminar, how important was it for you, to reach the following goals and which ones did you reach? | 1=most important | 3=not important | ||||||||
| 1 | 2 | 3 | reached | not reached | ||||||
| n | % | n | % | n | % | n | % | n | % | |
| Enlarge my knowledge about distance education. | 21 | 63,64 | 8 | 24,24 | 3 | 9,09 | 29 | 87,88 | 2 | 6,06 |
| Learn about different systems of DE. | 16 | 48,48 | 13 | 39,39 | 4 | 12,12 | 28 | 84,85 | 4 | 12,12 |
| Have direct contact to experts in DE. | 21 | 63,64 | 8 | 24,24 | 3 | 9,09 | 31 | 93,94 | 1 | 3,03 |
| Experience a virtual seminar myself. | 30 | 90,91 | 1 | 3,03 | 1 | 3,03 | 31 | 93,94 | 1 | 3,03 |
| Learn about the state of the art in DE-technology. | 18 | 54,55 | 13 | 39,39 | 1 | 3,03 | 23 | 69,70 | 8 | 24,24 |
| Learn about how a virtual seminar is organised. | 22 | 66,67 | 7 | 21,21 | 3 | 9,09 | 25 | 75,76 | 6 | 18,18 |
| Contribute to the field of DE-research. | 6 | 18,18 | 7 | 21,21 | 19 | 57,58 | 12 | 36,36 | 17 | 51,52 |
| Get rid of my personal critical questions about DE. | 3 | 9,09 | 12 | 36,36 | 17 | 51,52 | 16 | 48,48 | 13 | 39,39 |
| Get involved in the global perspective of DE. | 16 | 48,48 | 11 | 33,33 | 5 | 15,15 | 27 | 81,82 | 5 | 15,15 |
| Get in touch with colleagues. | 14 | 42,42 | 17 | 51,52 | 1 | 3,03 | 30 | 90,91 | 2 | 6,06 |
| Improve the level of my skills in DE. | 14 | 42,42 | 14 | 42,42 | 3 | 9,09 | 24 | 72,73 | 6 | 18,18 |
| Acquire skills that I could use in the near future. | 23 | 69,70 | 5 | 15,15 | 3 | 9,09 | 26 | 78,79 | 4 | 12,12 |
| I hoped to enrich my DE-related vocabulary. | 6 | 18,18 | 13 | 39,39 | 12 | 36,36 | 22 | 66,67 | 8 | 24,24 |
the list reads
| 1998 | 1997 | ||
| Learn about how a virtual seminar is organised. | 92,6% | Experience a virtual seminar myself | 94% |
| Experience a virtual seminar myself.: | 88,9% | Have direct contact to experts in DE | 94% |
| Enlarge my knowledge about distance education: | 85,2% | Get in touch with colleagues. | 9o,1% |
| Learn about different systems of DE. | 81,5% | Learn about different systems of DE | 84,5%. |
B 2 Statements
The following comments are about single statements out of the list of 18 statements to which participants could fully, partly or not agree. After the above argument about an average of impact it is only natural, to try the same thing with this list of 18 statements: the cohort of 1998 has a lesser „agreement" quotient than last years group:
| 1997 | fully agree: 6.7 | partly agree: 6.8 | don’t agree: 3.8 |
| 1998 | fully agree: 5.9 | partly agree: 6.1 | don’t agree: 4.9 |
The first two questions of this list are combined: It is about self-perception and perception of others in the same dimension: having reached the cognitive :learning goals of the seminar.
The distribution of answers is not so different in both cohorts: about half of the participants are sure to have reached the cognitive learning goals themselves but only 20-25% are sure that fellow participants did as well. To avoid this discrepancy of self-perception and perception of others in this context we formulated the question about the others in a more hesitant way: „Most of my fellow participants reached the cognitive learning objectives of the seminar" I don’t know why the participants were so much distrusting the learning process of their fellows.
| fully agree | partly agree | don’t agree | |||||
| 1997 | n | % | n | % | n | % | |
| I think I reached the cognitive learning objectives of the seminar. | 16 | 48,48 | 14 | 42,42 | 2 | 6,06 | |
| Most of my fellow participants reached the cognitive learning objectives | 7 | 21,21 | 21 | 63,64 | 4 | 12,12 | |
| 1998 | |||||||
| I think I reached the cognitive learning objectives of the seminar. | 13 | 48,15 | 12 | 44,44 | 1 | 3,70 | |
| Most of my fellow participants reached the cognitive learning objectives | 7 | 25,93 | 15 | 55,56 | 0 | 0,00 |
language problems
In 1997 the rate of (partly) language problems was almost 40% whereas in this year we find less than 30%; probably this is due to a changing rate of non-native English participants. (Which, as Ulrich Bernath told me cannot be, since the rate of native speakers in 1997 was much higher!)Groups
This aspect of virtual seminars seems to me most important; that is the reason why we bluntly asked whether participants realized that groups were emerging, whether they wanted to belong to a group and whether they actually belonged to a group.Lets start with the last aspect: almost half of last years participants say that the belonged (partly) to a group and two third had stated that they wanted to belong to a group. So there was a discrepancy with possibly frustated people in 1997.
In 1998 the picture is a little bit different: One third of the participants say that they (partly) wanted to belong to a group and the same proportion said that they (partly) belonged to a group. In other words - some 60% of 1998 participants explicitly did not want to belong to a group whereas 24% of 1997 members did not agree to the statement „ I wanted to belong to a group". ( Again here is a comment of Ulrich Bernath to this aspect: " We have explicitly encouraged internal groups for professional development in seminar #2; in seminar #1 we had "groups": the German, the Marylander and the others...)
Photos and biographies
Both groups unanimously vote for the inclusion of such personal elements. Even the 1998 group, and that is very interesting, has the highest ranking in agreement on the topic „At the beginning of the seminar I went through the biographies".(77.8%) So this result might induce the notion that this group isn’t so anti-personal or withholding as indicated above. The answer comes with one of the questions of our list:| 1997: | Social contacts with fellow participants are important to me. | 19 | 57,58 | 9 | 27,27 | 4 | 12,12 |
| 1998 | Social contacts with fellow participants are important to me. | 8 | 29,63 | 11 | 40,74 | 6 | 22,22 |
Now this is a clear difference between the two groups: 1997 seems to be much more sociable in its „personality" This result very well goes along with the result of the last three questions of that block:
More than half of the participants of 1997 plan to keep in touch with some participants of the group whereas less than one fourth of the 1998 group wishes to do so.
For two third of the 1998 group communication was „personal enough" the same percentage did not have any e-mail contact : the 1997 group shows much more interest in personal communication.
Probably all these results are overinterpreted now, because we have no information about the perception of self and others at the beginning of the 1998 seminar- but one thing is for sure: speaking of „personality traits" of a cohort of a course, the 1997 course seemed to me far more looking after personal relations than the 1998 course.
Again a comment from Ulrich Bernath: "No, because in 1997 we gathered participants from social contextes: The German group, the Maryland group!"
B.3 Witness learning
The core of the 1997 study „Host contacted" was the didactical result of the element of „witness-learning" having proven to be of extremely great importance to a virtual seminar. Sometimes we still read the expresssion „lurking" as something voyeuristic, with a touch of indecency: On the contrary, witness-learning seems to me to be of utmost importance especially in Net-based seminars. In our questionnaire we asked the participants in question No.3 to rate 15 didactical elements of the course with respect to their personal success: More than half of the 1997 participants stated that chatting did not help to the personal success of the seminar . Also „projects"had been rejected by one third of that group- the same tendency is towards additional readings on the Web. The „hit-list" of elements not contributing much to the personal success of the seminar reads almost the same in the 1998 group - „projects" seeemed to be almost unknown: half of the group rejects these. But lets go top - down:When participants had problems with linking to the web they were provided with printouts of the major articles via surface mail most likely the ratio of participants needing such a service was less in 1998 than in 1997 - The second item in the list "Readings on the Web" turns out to be the most important didactical elemnt of the seminar. In both years.Although participants of the 1998 seminar have been characterized as not being so much interested in others also these vote for witnessing the message interactions as second important The same result as in 1997!
The next most prominent element in both cohorts is actively participating in message interactions with experts and organizers. So the top three elements turn out to be important for both groups alike.
3. Rating of the elements
| How much did the elements contribute to your personal success in the seminar? | a lot | a bit | not | |||
| n | % | n | % | n | % | |
| The seminar readings delivered by mail. | 5 | 18,52 | 1 | 3,70 | 11 | 40,74 |
| The seminar readings on the web. | 20 | 74,07 | 7 | 25,93 | 0 | 0,00 |
| Additional recommended readings (not on the web). | 9 | 33,33 | 6 | 22,22 | 9 | 33,33 |
| Recommended URLs. | 13 | 48,15 | 11 | 40,74 | 2 | 7,41 |
| Actively participating in the message interaction with the experts and organizers. | 14 | 51,85 | 6 | 22,22 | 5 | 18,52 |
| Actively participating in the message interaction with fellow participants. | 11 | 40,74 | 7 | 25,93 | 6 | 22,22 |
| Witnessing the message interactions. | 18 | 66,67 | 5 | 18,52 | 1 | 3,70 |
| Availability of biographies. | 11 | 40,74 | 11 | 40,74 | 4 | 14,81 |
| Availability of a photo. | 11 | 40,74 | 8 | 29,63 | 7 | 25,93 |
| Direct email contact. | 4 | 14,81 | 11 | 40,74 | 10 | 37,04 |
| Conference Assistance. | 7 | 25,93 | 8 | 29,63 | 10 | 37,04 |
| Seminar weekly archives. | 12 | 44,44 | 6 | 22,22 | 4 | 14,81 |
| Chat group for the virtual seminar. | 4 | 14,81 | 7 | 25,93 | 12 | 44,44 |
| Projects. | 1 | 3,70 | 5 | 18,52 | 13 | 48,15 |
| Evaluation seminar. | 3 | 11,11 | 15 | 55,56 | 2 | 7,41 |
Now to the differences in the rating of didactical elements:
Whereas the fourth position clearly goes in 1997 to mail delivered readings and then to the weekly seminar archives, we find position 6 to be held by actively participation in message interaction with fellow participants and after that going to "recommended URLs" it is this element, that moved up to a higher rank position in 1998.
So something has changed: less than 30% in 1997 rated recommended URLs as important whereas in 1998 we find almost 50% .
The rest of the final questionnaire results is not so much different from the results of 1997- so I will stop here and hope that the next seminar of this kind will allow us to present a questionnaire again.
Hagen, Mai 11th 1998