9.3 Evaluation
To evaluate the resource centre, ‘qualitative data’
is needed, as well as the ‘quantitative data’ collected
by monitoring.
9.3.1 Collecting qualitative data
Qualitative data, such as users’ opinions, can be gathered
in different ways. For example, users could be asked to complete
a questionnaire, or they could be interviewed using the questionnaire
as a structure for the interviews. Questionnaires could be given
to a representative selection of visitors and enquirers, to keep
the number of questionnaires down and make them easier to analyse.
Questionnaires sent to enquirers should be accompanied by a copy
of the original request and the reply, including details of the
materials that were supplied. This is particularly important if
the questionnaire is sent some time after the enquiry was made
(such as more than six months).
Focus group discussions (small group discussions) could be set
up for staff or students of the parent organisation, visitors
and more distant users. They could be organised as part of a national
or regional meeting attended by users of the resource centre.
Users could discuss their opinions of the resource centre and
its services, what impact information from the resource centre
has had their work, and what subjects and services they require.
Information gathered through questionnaires and focus group discussions
can be used both for evaluation and as part of ongoing needs assessment
(see Section 1.1).
9.3.2 Using the data
The data collected through monitoring and questionnaires or discussions
can be used to answer questions such as:
1. What impact does the resource centre have on users?
Has using the resource centre or the enquiry service
helped users to carry out their work or studies? Has information
from the resource centre led to any particular action or change
in work practices?
This information can be obtained from the monitoring information
and comments made by users through questionnaires and discussions.
2. How far does the resource centre meet users needs?
Do users believe that the resource centre is meeting their needs?
It is also useful to consider whether the resource centre was
established in response to a demonstrated need. Many resource
centres grow from an individual campaign and are based on staff’s
view of what is needed, rather than the views of the community
that they aim to serve. It is worth checking whether a needs assessment
was carried out, and whether the collection and services reflect
the needs identified in the assessment.
This information can be obtained from documents about setting
up the resource centre, and by asking users how far the resource
centre is meeting their needs. Users can also be asked what they
expect of the resource centre and what services are available
elsewhere. It is useful to note how many users visit the resource
centre more than once, as repeat visits help to show how useful
the resource centre is. Similarly, it is useful to note how many
enquirers make more that one enquiry. A second request on a different
topic suggests that the user was satisfied with the response to
their first enquiry. A sample of enquirers could be contacted
to obtain more detailed feedback about how useful they find the
services.
3. What do users save by using the resource centre?
How much time, money and effort would users have to spend to obtain
information, if the resource centre did not exist? For example,
they might have to travel further, or they might have to visit
several different resource centres. If they could not obtain the
information they need from elsewhere, this would mean that the
resource centre was providing a unique service. Even if the resource
centre was not very efficient, being unique could be a key strength.
This information can be obtained through focus group discussions
and questionnaires, and through knowledge of what other information
services exist, what subject areas they cover and what services
they provide (see Section 1.4.2).
4. Are enough people being reached?
How clearly defined is the resource centre’s target
audience, and are enough of them being reached? For everyone who
comes to the resource centre, there are sure to be many others
who do not know that they need information or do not know where
to find it. Is the resource centre publicised in places where
potential users go, such as training institutions, religious centres
or community groups? Is it possible to identify sections of the
community who need the services but are not using them?
To answer these questions properly, it is necessary to know the
size of the target audience, such as the number of health workers
or trainers in the area covered by the resource centre. It is
also important to look at efforts to promote the resource centre
to members of the target audience who were not previously using
it, to see how well the promotional activities have worked.
This information can be obtained from the records of visitors
using the resource centre, enquiry services, and number of users
before and after promotional activities. It can also be obtained
from the opinions and ideas of users for whom promotional activities
are targeted, including those that have not used the resource
centre.
5. Does the resource centre meet the needs of funding
organisations?
This information can be obtained by checking funding
organisations’ goals and criteria for supporting the resource
centre, and comparing these with the results of monitoring and
evaluation.
6. How good are the materials?
How often are the materials used? Are there many materials
that are rarely used? Are some subject areas more popular because
they are more up-to-date? Are both resource centre staff and users
clear about the subject areas covered by the resource centre?
Have all the subjects or material formats requested in the needs
assessment been regularly used? If they are not, does the collection
policy need to be changed, do materials need updating, or is more
promotion required?
This information can be obtained from the monitoring information
and user questionnaires and discussion.
7. What does it cost users to obtain information?
How much time do users spend learning to use systems
such as the classification scheme, catalogue or database? How
easy is it for them to use these once they have learned how? How
do the advisory services and information skills training provided
by staff help users to find information in the resource centre?
This information can be obtained by providing advisory services
(see Section 7.4), and through user discussions and questionnaires.
8. How skilled are staff?
Can staff provide information as well as process materials?
Are staff friendly and helpful? Are they involved in planning
new developments and knowledgeable about what is going on? Do
they need more training?
This information can be obtained from user discussions and questionnaires,
and discussions with the resource centre advisory committee and
resource centre staff about staff’s capabilities and training
needs.
9. How well is the resource centre networking?
Are enquiries received from other organisations or individuals,
such as public libraries, research organisations, community groups
or individual experts? Are enquiries from users referred to other
resource centres? Have efforts been made to eliminate duplication
by sharing responsibilities, such as collection, processing and
storage, with other groups? Is there a file of people or organisations
who can provide information and share their expertise? Staff should
not simply add names of useful contacts as they hear about them,
but they should go out and ask people if they will collaborate
with the resource centre.
This information can be obtained from staff records, minutes
of resource centre advisory committee meetings and discussions
with staff.
10. How useful are the resource centre’s publications?
Are publications such as current awareness bulletins,
information packs, newsletters, articles, or resource lists produced?
Are they accurate, legible, appropriate to users and efficiently
distributed? Do users find them useful and timely? Are they a
worthwhile activity in terms of the time and resources available
to the resource centre?
This information can be obtained from staff records for preparation
and distribution, users’ discussions and questionnaires,
and staff comments about the time and effort taken to prepare
publications.
11. Are systems for selection, indexing, cataloguing
and retrieving information cost-effective?
How much does it cost to process each material in the
collection (in terms of both staff time and materials)? How long
does it take to process materials (for example, accessioning,
cataloguing and classifying, entering records onto the database,
and quality controlling)? Are these systems worth the staff time
involved, because they speed up the retrieval process, or do they
take more time than can be spared?
This information can be obtained by monitoring the time taken
to process materials, looking at the records of information searches
that have been carried out, how long searches have taken, and
how far they have met users’ needs. Opinions expressed through
staff and user discussions and questionnaires are also important.
12. How can the resource centre increase its collection
and improve its services in the most cost-effective way?
This information can be obtained by listing ways in which
the resource centre can increase its collection and improve its
services, and then identifying which of these are least expensive
in terms of money and staff time. Comments from users can be obtained
from the suggestions/comments box, monitoring forms, and questionnaires
and discussions.
13. What improvements are the most cost-effective and
beneficial?
This can be decided by comparing improvements that are
cost-effective with what users most need, and reaching a balance.
It makes no sense to offer services that the resource centre cannot
afford, but if there is a choice of services that can be offered,
the needs of the users should always come first.
9.3.3 Using the results of an evaluation
The purpose of carrying out an evaluation is to help improve the
resource centre and its services. The process of evaluation demonstrates
what is being done well and should be continued, as well as what
needs to change and what additional activities could be undertaken.
Poor results are as important as good ones, as they can point
to ways to improve a service.
Evaluation results should be used to identify new objectives,
and develop new action plans (see Sections
1.2 and 1.3). They may result in
changes to how the resource centre is run, what it collects and
what services it provides. They may also identify staff training
needs, to enable staff to carry out their work efficiently and
provide the services required.
The planning cycle
top of page
