International Confererence on
Supporting Students in e-learning systems Rome, Italy, September 24th 2004
Quality
assurance from the students’ point of view
Helmut
Fritsch ZIFF,
FernUniversität in Hagen
_______________________________________________________________
contents:
1. Quality as perceived by society at large and quality as perceived by
students
2.
The legal
situation
2.1 Evaluation as a prerequisite for the accreditation of curricula
according to the Bologna-process
2.2 The necessity for student counselling after the first year
2.3 Current issues with these prerequisites
2.4 Student satisfaction with connectivity of the supporting institution
3. staff and student perspectives
________________________________________________________________________
1.
Quality
The notion
of "quality" ranges from any adjective to detailed descriptions of components
and up to 30 specified dimensions[1].
In the times of consumer protection you want to know what you will get but who
dares to apply the judgement of "best quality" to a product when
there is nothing at hand to be compared with. Before quality can be compared,
detailed description of the components is needed. For products it is the detailed
description first and then the question whether this description finds its true
counterpart in the product. Only after this "internal" process you
might compare the quality of two products.
Lately one
of the big issues in e-learning is the description of all different aspects , or ingredients, or possible outcomes of a learning
object. Data about such an object in this specific, descriptive manner is
called "metadata" (Learning
Object Metadata, LOM as proposed by IEEE, SC36/WG5 Terms of Reference[2].).
Such metadata are proposed for the better future marketing of learning objects,
which makes sense, not having to invent the wheel each time anew. So the
mission of that group is:
Scope: To
describe and characterize processes, components, and attributes related to the
quality and architecture of IT-supported environments in the field of learning,
education, and training.
There is a
phenomenon that "quality", being discussed by all players in the
field is understood differently from each perspective: the ministry of
education of NorthRhine Westfalia, where the FernUniversität in Hagen is based
and by which it is financed, tries to screen the quality of universities
according to two dimensions:
output of students
leaving the university with final exams and intake of "third party's money" for research
projects.[3]
They take the curricula in each subject area and assess it as follows: If a
department fails to meet at least the average of the other state universities
in both dimensions, discussion starts with the goal that the curriculum will no
longer be acceptable and the department will be closed within a time-span of
3-5 years. There are small niches for new curricula where such comparisons do
not work as yet, but the strategy as a governance concept has led to frustration
among universities.
There is a
forced consensus introduced by the Bologna process which means for all German universities
to restructure their curricula according to the new European models of Bachelor
and Master studies and PhD studies. Quality control as
part of accreditation together with the introduction of the European Credit
Transfer System was meant to ensure compatibility within Europe. It is about
the precise description of what learning outcome can be expected from a
curriculum and, with a so called "diploma supplement", precisely what
has been offered and what grading the learner in his cohort has reached. ECTS
compliancy will be the legal situation from 2006 on. No enrolments into
"older" curricula will be possible after that date. Accredited
curricula will be installed all over Europe.
So almost all American universities do undergo several accreditation
procedures- except the MIT which claims that there is no need to do this
because no one would doubt the inherent quality of all MIT curricula.
The notion
of quality, and its debate shocked the universities, reminiscent of colorful associations
to 19th and 20th century university life, such as having
studied in Bologna or graduated from London School of Economics, having passed
exams in Tübingen or at
the Ecole Supérieure in Paris , a feeling of belonging to the academic light
these universities shed on societies of
that time.
2.
The legal situation
2.1
Evaluation as a prerequisite of accreditation
"As a result of the amended Framework Act
for Higher Education (HRG) of 20 August 1998 German higher education
institutions are able to introduce the wellknown and internationally accepted
degree programmes leading to Bachelor's and Master's (BA/MA) degrees. According
to the resolution of the Association of Universities and Other Higher Education
Institutions in Germany (Hochschulrektorenkonferenz – HRK) of 6 July 1998 this
reform is intended to enlarge the creative scope of the higher education
institutions, to improve the international compatibility of German university
degrees, to enhance the student mobility and to increase the quotas of foreign
applicants for a place at university."[4]
Conformity with the structural
guidelines is reviewed within an accreditation process; this specifically
relates to
These are
the texts of the national accreditation board. Several accreditation agencies have
been set up from which universities may choose.
As a rule
evaluation is one of the issues to be looked for within the accreditation
process, done by an accredited agency but necessarily conducted by different
bodies, e.g. so called "external evaluation".
The term
evaluation does not yet play an important role in the next hierarchical layer,
the accreditation agency, for which my institution has opted, at least not
today. But The philosophy of this agency is to ask for
evaluation data together with the forms for "re-accreditation". This
gives us time since the first accreditations have been done on the basis of the
existing applications. Without evaluation no re-accreditation[5].
Curricula not accredited will have to be discarded or amended. Ammendment
processes should not last for longer than one year. So we do have a timeframe of 3-4 years.
2.2. student counselling
the
ammendment of the North Rhine Westfalian law for universities introduced the duty
of universities to actively give feedback to the students after the first year
of study. This new process has not yet started systematically, but will be of major
importance for the so called "drop-out" prevention especially for
late drop-out. Because drop-out has its
peak during the first year, for the university it must be of major importance
to prevent drop-out or at least to have it as early as possible because funding
will look at such rates very intensively. It would be no problem, if the situation
stayed like in earlier times, when the institution receives money for all
enrolled students and need not care about completion rates. So here again comes
the perspective of the student- let alone the perspective of the tax-payer.
It is not
so difficult - although hard enough - to convince the faculty about the process
of evaluation in times when different obligations have to be encountered,
different problems to be solved. Curricula should be compliant with
international standards, they need to be accredited,
which means lots of additional work especially for the deans of departments. So
they had been quite happy with the procedure to get the first accreditation
upon application. But the present problem is one of reform: It is something new
to staff and faculty to get screened by peers from outside. And it is something
new to care for systematic evaluation: "as if staff could be made responsible
for the learners outcome"! So in the first round
of re-accreditation we may witness a procedure that has to be adopted by all
those departments which have no systematic procedures for evaluation provided
until 2006/2007. There will be a
specific documentation of interviews with the students who went through these
curricula. Such interviews may be structured and need not be representative
data. The open question, though, is how
long will this strategy last? When, and here again we have to take the students perspective, will
students ask for such evaluations before taking the decision to enrol? As
of today the elected students representatives of FernUniversität in Hagen
already ask for detailed evaluation concepts and questionnaires because they
want to take part in evaluation.
NorthRhineWestfalian
Law provides five instances of different
concepts of evaluation:
1) generally the law
says that assessment (regular evaluation) has to be carried through in the
fields of research, teaching, promoting the young academics, and gender mainstraming. This has to be done in cooperation
with other universities and with the state govenment.
2) students have to be
asked for their evaluation of seminars and of curricula
3) external experts
have to be included
4) until the end of
the second semester,at the latest, the university evaluates the course of study
of the individual student and, if
necessary, starts a counselling process.
5) the dean presents a
teaching report biannually with 4 requirements:
a) statistical data about staff, offices, exams, length of
study, interim assessments,drop-out rates
b) content and structure of teaching offered, organisation,
tuition and counselling
c) results of the yearly students opinion poll od student
satisfaction
d) evaluation of strengths and weaknesses and proposed
improvements
The council of the department and the Senate comment on the
teaching report which has to be published.
2.4 student
satisfaction with connectivity[6]
During the
last couple of years FernUniversität encountered some complaints about low responsiveness
of the institution towards queries of its students. Staff looked for could not
be reached via telephone. In order to find out about the scope of the problem
we conducted an empirical survey among a sample of all enrolled students in
March 2003. Not only telephone-reactivity but also the question
of satisfaction with the student support service of the institution and its
different departments have been in the centre of this research project.
2.4.1
Method:
Feasibility aspects of the study restricted us
to a mailed questionnaire because a net-questionnaire seemed not to be suitable
to reach all students in the stratified sample that included the group
refraining from modern media and would turn to the telephone in search for help
or even send a written snail-mail letter.
The sample we drew was N=5351 enrolled students. We received 700 (13%)
feedback. This very low response rate as well as a slight over-representation
of students of the liberal Arts faculty (coinciding with an overrepresentation
of female students) restricted the representativity of the results.
Cross-checking of the results made clear, though, that there has not been a
systematic distortion of the results due to gender.
Some further attributes of the respondents characterize our students: Slightly
more than half of the respondents come from early semesters (55%), meaning that
they have been studying for less than four semesters.
Two thirds rate their "status" as being part-time student (66%).
Among answering students we also find quite a number of advanced students,
shortly before their exam.
Two third of the respondents know other university systems from previous own
experiences and 60% of the respondents can be called "active"
students in that they have participated last semester by sending in assignments
for correction or taking part in written exams. Although the results were not
representative we will interpret them in an explorative way.
2.4.2 Results
Telephone-contact:
There is only a small number of our students who
during the last semester tried to make a phone-call. Mostly two targets have
been called: the department (responsible for all teaching issues, and
assessment) and the students-secretariat (responsible for enrolment, fees,
address-change etc.).
Adding up all contact instances we find that 30% have never tried to contact any
of the offices via telephone. On the other hand we find a huge variance, more
than 50% of the students being successful in the first try and two thirds
successful in trying twice. There are some instances where students tried and
tried, up to 8 times and had no success.
The students being content with their effort to have telephone contact with the
department or the administration ("very content", or
"content") always outnumber the students answering "not
content" or "disappointed".
Also the rating of success in clearing their problem is very positive: 85% felt
felt they were successful..
2.4.3. Turn-around
time for e-mails,
fax and mail
Small
numbers of instances are the basis of interpretation. Mail Letters:
More than
50% receive answers in less than one week. Only one department took longer time
in responding. In seven of 12 units we find 90% of the queries answered within
two weeks. FAX:
In less
than 4 days most Fax queries are answered. while there are individuals who report little
or no success, though.
e-mail: within
2 days 50% of the students trying to solve their problem via e-mail are
successful - In 10 of the 12 units 90% of turn-around time is less than 7 days.
Less than 1% report to have received no answer. In open
comments students report instant interaction with one department and no
reaction with another one.
2.4.4 Overall evaluation of responsiveness
There are
generally less than 10% of our students who report responsiveness-difficulties
and then only in the case of two targets (their study centre and their official
course faculty member where we find up to 15% dissatisfied students).
Generally, for all units, we find the number of satisfied students surpassing
the number of dissatisfied students. When ranking the
units we find the official course representative in the faculty in top
position, then the student's secretariat and then the study centre. The same
ranking order is found when we sort according to averages of satisfaction. For
all departments between overall evaluation of the services and telephone-interaction
there are correlations of medium size (r=.2 to r=.5).
2.4.5. General
evaluation of their studies with the university
78% of the
respondents are satisfied or very much satisfied with their study situation at
FernUniversität. Some 7% are not and 2% are very dissatisfied (the rest undecided).
Except between general evaluation and evaluation of responsiveness of the
department (not higher than .45) we found no other significant correlations.
3. staff and student perspectives
Student
support in e-learning
affects both sides, the staff and the student body. During the last decade or so our students have
been asked regularly about their computing equipment and about their
competencies in using computers- and access to computer systems and the
internet. By now we may assume that only small numbers of students do not have
access at home. In these cases students might have either access during their
work or in a study center. So in the Senate meeting early 2004 it was announced that in the near
future the university will require the students to have access via internet to the
university.
Now having
access and continuously using
the internet is something different. Not only for the students,
of which only 10-15% make full use of the offer of the virtual university platform
but just as well for the staff. So we conducted a short inquiry[7]
about internet use for teaching purposes among our staff: all in all it is the
same situation, most know and use www but only small numbers of staff really
"teach" via the internet.
In the
framework of the Sokrates Minerva project we also asked the students having had
contact with our virtual system LVU.[8]
We
presented 38 items in an online questionnaire and asked for answers in two
dimensions: we were interested in the student´s ratings of importance and in a separate
list their satisfaction with the same element.
7 sociodemographic items and one item on their
use of electronic support systems and fianlly one on their satisfaction in
general.[9]
Within the
first 24 hours we received 130 answers- after one week we had to close the
questionnaire because of a virus attack. So the results of roughly 200 students
answering will neither be representative nor replicable. But there are some
insights which already can give hints to our virtual university system.
- Before
working hours and instantly after 16:30 we find most students answering, this
gives a hint that they will use their office computers either before daily
responsibilities or towards the end of working hours.
- There are
four outstanding items where students are quite dissatisfied with,
here is the place to improve, seen from the students point of view:
1) information about required working hours for study
2) information about course choice and possibilities of
individual advice
3) information on cost
4) advice on internet-based group work possibilities
So these
items should not be too difficult for the system of LVU to resolve. It shows
that students asking for support will mainly ask for information. Confronted with the concept of a virtual university using a fully
comprehensive system it shows that deficiencies are more on the side of the
institution than on the side of the student. We have to ask ourselves
whether this situation is typical of virtual universities or rather typical for
institutions with a bureaucratic administration. I think it reflects the aspect
of organizational difficulties, difficulties of the kind "how to find the
application form for the application".
[1] QUALITY IN E-LEARNING FROM A LEARNER’S
PERSPECTIVE by Ulf Ehlers, University of
Duisburg-Essen, (best paper award at the 2004 research workshop
of EDEN March 2004).in: Proceedings of the 3rd EDEN Research workshop, ed.ISBN
3-8142-0902-8 ,bis Oldenburg 2004, p.130-137
[2] http://frameworks.jtc1sc36.org
[3] Berlin uses a different, more elaborated
system of evaluation for financing parts of the budget:
http://www.his.de/Service/Publikationen/Kia/pdf/Kia/kia200404.pdf
[4] http://www.accreditation-council.de/
[5] http://www.aqas.de/Material/Aqas_Folien.pdf
[7] http://www.fernuni-hagen.de/ZIFF/wwwteach.htm
[8] http://www.fernuni-hagen.de/ZIFF/sssfb.htm