While the aim of the first questionnaire reconnoitering European distance teaching institutions was mostly to gather information (tangible and easily verifiable data on where the institutions were and how they operated), the aim of the second questionnaire is to explore how the different institutions deal with the problem of evaluation in order to sketch a comparative picture.
Generally speaking, questionnaires are used to throw light on a situation about which little is known when there is no exact model against which to compare the data to be collected. This research team did not have a model a priori with which to analyse the various interpretations of testing in different countries and in different training contexts and with which to foresee the different techniques, procedures and tools needed in practical terms. Each member of the team had several possible models in mind, but it was decided to start by formulating wide-ranging hypotheses that would then be narrowed down and further detailed when the survey was already underway. The risk was annulling the authenticity of the survey.
The initial hypotheses were the result of years of experience in the field. The team has shared their experience working together in Rome's Third University Laboratory of Experimental Pedagogy where specialisation courses in distance learning have been held for more than ten years. The Research Centre in Distance Teaching (CRIAD), and the journal IAD which plays a significant role in the distance teaching debate at an international level, are the concrete results of this experience.
The testing system used in our distance learning specialisation courses has been the object of systematic reflection and analysis for years(1), and constitutes the central focus of our preliminary survey. This system envisages numerous control procedures, accompanying every stage of the course and offering students various opportunities for self-evaluation and evaluation by others. Its most important feature is the high level of automation and personalisation of the correction and learning compensation procedures.
The specific nature of our experience in testing led us to consider in particular a few essential characteristics. These include the periods when testing takes place, the kinds of tools, the existence of teaching feedback and the data collected from the tests.
The vast research activity that rotates around the IAD journal and the CRIAD(2) has provided a very strong stimulus for a more theoretical approach that has contributed to developing a theoretical basis for distance teaching.(3) This work has led us to consider the issue of testing as an intrinsic part of the whole training process and not as an isolated problem.
Moreover, exchange programmes with other insitutions in other countries through the journal have consolidated our knowledge and experience and provided a solid basis of information on which to build. This flow of information on how other distance teaching bodies work, together with theoretical reflection on the issue of testing, has helped us identify and define the various aspects to be covered by the questionnaire.
As we have already stressed, we have attempted to plan and draft the questionnaire without recourse to pre-defined hypotheses, withholding judgement in order to avoid a forced interpretation of the data. Our aim was to conduct a detailed analysis of all the possible features of a testing system and then verify whether or not these systems were in force in the various European insitutions through the questionnaire. Once this atomization was accomplished, the various features were classified in order to estimate the greater or lesser attention of each institution for each class of variable once the data was collected.
The following are the results of the preliminary research conducted by the research tema on the features of testing systems in distance learning.
| A. Collecting information on students |
| 1. Periods: - beginning - middle - end |
| 2. Frequency |
| 3. Object: |
| - transversal skills (reading, writing etc) |
| - general knowledge |
| - specific skills |
| - specific knowledge |
| - study skills |
| - study self-sufficiency |
| - motivation |
| - interest |
| - attitude towards conventional teaching |
| - attitude towards distance teaching |
| - time available for study |
| - resources availbale for study (computer, library etc.) |
| 4. Tools: |
| - structured: - tests, semi-structured tests, questionnaires, outlines of interviews, observation grids |
| -non-structured: - interviews, open tests (exercises, simulations, thesis writing, project work, research on a theme etc), observations in context, practical tests |
| - construction: intenral or delegated to experts |
| 5. Treatment of data: |
| - archiving (paper, computer) |
| - analysis: - quantitative - qualitiative - individual (synchronic, diachronic) - collective (synchronic, diachronic) |
| objectives: - measuring learning results - evaluating quality (effectiveness of materials, indices of interaction, efficiency of manaement, effectiveness and efficiency of media, effectiveness of tools |
| - utilisation of data collected |
| B. Interaction |
| - type of interaction - direct - mediated |
| - means of communication used |
| - frequency |
| - aims |
| C. Personalisation |
| - feedback |
| - flexibility of learning path |
| - support for learning |
| - tutoring |
| D. Certification - exams |
| - exam centre |
| - tools |
| - type of testing |
| - type of certification |
The questionnaire was formulated on the basis of this analysis. These basic categories were obviously further divided and reclassified according to considerations that emerged while the questionnaire was being written, as is often the case in the transition from the planning phase to implementation. The structure and the organisation of the final version of the questionnaire was slightly different from the original draft as new variables were found during the research.