People Dancing

Monitoring and evaluation - a general guide

(A PDF of this information sheet is available to download below)

This briefing looks at effective ways to monitor your work, evaluate what you do, and share results with your organisation’s partners, funders and staff. Monitoring and evaluation (M&E) is increasingly becoming a standard requirement of most funding bodies to demonstrate how well organisations are doing and what’s working for them. M&E is often seen as time-consuming, unrewarding, and a job for professionals, but done well, it can be another creative and enjoyable aspect of your work and an invaluable system for improving your services. By following our guide, you will be able to treat it as an integral part of what you do, using resources and skills you already have.

What do the terms mean?

Monitoring – is information-gathering work that takes place throughout the life of the project. Monitoring will help you answer questions about your project, and will be crucial when you want to evaluate. It is important you collect the information in a well-planned and organised way.

Evaluation – is the assessment of the project, usually once it’s complete. Evaluation uses the monitoring information you have gathered to make judgements about how well you are doing, i.e. what is and isn’t working. It will reveal how well funds have been used and the benefits to all those involved. It will also help you create more successful, well-managed projects in the future. This process should be open and honest for the best results.

Self-evaluation – is when an organisation uses its own people, skills and resources to carry out evaluation of their own work. This ought to be built into the everyday activities of a project so that it becomes a standard part of what you do. All those involved should be aware of what you are trying monitor and what the targets are.

Project - Throughout this briefing ‘project’ is used as a general term for the activity or event that you are evaluating. However, please bear in mind that these same principals can be applied to all sorts of aspects of your work.

Why do you need to monitor and evaluate?

• To review what you’ve done
• To measure progress and identify failures
• To make improvements to your work
• To show the impact on your community
• To show volunteers and staff the value of their work
• To develop the group’s strengths
• To keep control of your finances
• To involve stakeholders and partners
• To show your funders that their money has been spent in the right way. Many will ask you to monitor your work to provide them with ‘proof’ of success.

The M&E Glossary

Some of the terms used in M&E, which you might come across when communicating with funders, aren’t always familiar. A few of the most common are listed below (and marked with an * symbol throughout the Information Sheet):

• baseline - what exists at the start of a project. This can help show gaps in provision or prove the need for the project to happen
• demographics - details such as age, sex, ethnicity and place of birth
• inputs - time, money and resources that go into the running of the project 
• outcomes - changes that the project has made to the group or community. These can include increased skills, confidence and knowledge amongst the people in the group
• outputs - measurable evidence about what happened as a result of the project. They show how close to your original targets you were. For example, if you set out to do workshops with 20 young people, what numbers did you actually get? 
• quantitative or ‘hard’ data - this is the numbers bit – i.e. figures that show what happened during a project. You can find out how many people participated, of what age, where from, what they liked/disliked, what they spent money on etc. Statistics like these help you do an audience profile that can show up gaps in provision as well as new opportunities. This ‘measurable’ data is collected through ticket sales, tick box forms, clickers and other methods
• qualitative or ‘soft’ data - this shows people’s experience of the project, including the audience, participants, artists and organisers. Although it isn’t ‘measurable’ data, it can reveal the impact a project has on people’s lives or on the local community. Did people learn new skills, gain confidence or make new friends? Has the project made the community feel closer? Information can be gathered through photos, video and talking to people in an informal way
• target group - the main group or groups you are working with and the people your activity, service or event is for. Your project will probably attract different kinds of people but it is important to be clear about the people who are most likely to participate in your activity. Which groups will benefit or change as a result of the project?

The timescale 

During any arts project, time is one of your most valuable resources, and one that fast slips away the further into the action you get. The key to successful evaluation is to start the monitoring process even before the project begins. Perhaps the easiest way to divide up your time is to plan in three stages – before, during and after.

BEFORE

This is the time to assess what you already have and what you will need before the project begins. This means that you can establish a baseline* level from which the project is starting, allowing you to see how far you have travelled by the end.

Asking the following four questions will help you do this:

1. What is the demand?

• Have you asked people in the community if they want or need this project?
• Will they take part as an audience or participants?
• Do similar things already go on?
• Can you show a lack of provision in the area?
• Have previous surveys identified what is going on, and what might be needed?
• Are there potential partner organisations you should approach?
• Most funding applications will ask for evidence like this before giving you a grant.

2. What are your aims and objectives?

Funders and other partners often want to know what benefits will come from their investment in you, and so before you begin it is important to make your aims and objectives clear and focussed. This will also make the evaluation process much easier at the end, as you can measure whether you have met your initial targets.

Aims are the changes you want to see in your target group*

Use verbs that describe change, such as: to improve; to develop; to reduce; to promote; or
to enable. Your aims could include:

• getting people involved
• reaching out to new audiences
• teaching new skills
• developing partnerships
• establishing and maintaining your reputation.

Aims are often qualitative* (e.g. improving the quality of life for disabled people in the area).

Objectives are the practical activities you will carry out to bring about a change for your target group* – to achieve your aims.

Use verbs which show action, such as: to increase; to organise; to set up; to provide; to run; or to produce. Your objectives could be:

• to hold a discussion group
• to put on performances especially for schools or young people
• to increase the number of participants
• to provide a series of workshops
• to involve a local business
• to organise an on-going series of events.

Objectives are more likely to be quantitative* (e.g. increasing the number of folk dance groups by 15%).

Try to include as many people involved in the project as possible when setting your aims and objectives, and make sure that everyone is clear about them once they have been decided upon.

3. Your resources – what do you have and what will you need?

First you will need to look at the resources you already have – what money, equipment, and venues are available? Do you have volunteers and staff with the skills you will need? What partners are available to help, and what can they bring to the project? These are called your inputs*.

After this, you can carefully assess what and how much more you will need. What are your estimates for project or activity costs? Possible costs could include:

• expected expenditure on equipment
• fees for paid workers
• volunteer expenses
• promotion
• rent
• venue hire
• stationery
• phones.

If you keep a record of anything you’re being offered for free, such as use of a hall, or volunteers giving their time, it can be used as ‘match funding’ when you apply for grants.

Keep a check on your income and expenditure throughout the project. Compare this with your original estimates and make necessary adjustments.

4. What are your performance indicators?

Outputs and outcomes
This may sound like scary jargon, but once you break the words down they make simple sense. And once again, funders may well ask questions using these terms, so it’s worth getting clued-up. All it really means is taking each of your aims and objectives and asking ‘What signs (performance indicators) will show us that the change we hoped for has happened?’ Simply put, performance indicators guide you on the sort of information you’ll need to collect to help you make judgements and evaluations.

There are two main types of performance indicators that we will look at:

• output indicators – which help you assess the progress you are making towards your objectives
• outcome indicators – which help you assess the progress you are making towards your aims.

Output indicators help you measure the success of the activities, and what happened as a result of the project. They show how close to your original objectives you were. For example, if you set out to do workshops with 20 young people, what numbers did you actually get?

Outcome indicators help you measure whether you are really bringing about the changes you aim the project to make to your group and the people you are working with, like increased skills, confidence and knowledge. What will show that these things have changed?

Before you start you will need to decide what indicators will help you prove how successful the project is. But, make it easy on yourself:

• limit the number of indicators you want to look for – there is no point collecting information that you don’t need
• make your indicators user-friendly – will it be easy, or even possible, to collect the information you need against the indicators you have chosen?

The next section will look at the different information you can gather about your indicators, and the tools to measure and record it.

DURING

During the course of the project, you will need to carefully monitor your progress, looking at who is taking part and how many people are attending, who’s contributing what, and what they have gained from it. This information will help you evaluate the success of the project at the end.

1. Who’s involved?

Measuring attendance
Who is taking part, and how many of them are there? You can collect data on the participants’ demographics* – where they come from, their age, gender, and ethnicity; disabilities; how they found out about the event; how they got there; whether they’ve been involved before and if they’ll come again. Ask questions that reflect what is most important to you. You might want to use one of the following tools for measuring attendance:

Paper lapel stickers
A friendly, inexpensive and low-tech system to count people attending an event – just subtract your empty spaces from the number on the packet. This is also open to all sorts of variation, as you could use different sizes or colours to differentiate children from adults, men from women, returnees from first-time arrivals, and so on.

Number clicker
For counting entries through a gate or turnstile. This is even simpler than the stickers – just read the number, and re-set as you need to. However, it’s not as adaptable and cannot distinguish people who leave and then return.

Box office and gate returns
For counting entries at the gate. You could also use postal and on-line bookings. The more advanced your technology, the greater the range of information this can give you, down to block bookings, occupations, family size and post-codes. But remember, this will only tell you who paid for tickets or actually came in through the gates (not those who came in free of charge etc) and it will be no good for free events.

Camera and video
Photographs or video footage can give rough estimates of attendance – although it makes it less easy to get accurate numbers. Nevertheless, it is quick and easy, and easily fitted in with a number of other jobs. Vivid and colourful photos or film can also be great for future marketing, and enriching dialogue with funders. It can also be used for qualitative* analysis – see below.

2. How good do people feel?

Measuring responses at the time
Unlike numbers and statistics, qualitative data* can help you capture the essence of a project, and what people have gained from it. You can measure people’s responses in several ways during the project:

Graffiti wall, comments book and post-it notes
These techniques offer people the chance to make an immediate response on their own terms. Simple, open, informal, and relatively private, they often elicit honest and unexpected responses, and are very good for capturing witty, catchy comments and increasing a sense of participation. It will demand little staffing, but this means you have no control over the sorts of statements that will be made. It’s also not a representative method – as not everyone will want to contribute.

Archery target and stickers, etc.
An archery target divided into rings for ‘excellent’, ‘good’ and ‘fair’ (for example) is a simple, ‘gut-feeling’ response tool. Although not the best tool if you are looking for qualified or considered responses, it is low-tech, tactile and fun to use – especially but not exclusively with young people. It can be used with stickers, pins, post-it notes, marker pen marks or as a pin board, and targets can be anything from six inches square to twelve feet (why stop there?!) – a very adaptable technique indeed. The diagram below is quite a sophisticated version, with eight sections reflecting various elements in which people can register different responses – this one is designed for use at a music event.

(See PDF for diagram)

Questionnaires: closed questions
This means questionnaires that use tick-boxes and yes-no answers, basic questions of fact, or ask for a score, rating out of five or percentage. They can provide a range of basic information, can be answered quickly, and are relatively easy to process. Bear in mind that bad questionnaires can lead to confusion and misleading information, so think carefully about the questions you ask. Rates of return are often poor – 25% is high – so perhaps try linking the questionnaire to entry for a prize draw, or some other reward.

Questionnaires: open questions
These ask questions that encourage a fuller and more considered response – perhaps leading on from closed questions – giving more scope for people to express their feelings. They can provide phrases that ‘sum up’ an experience, and encourage a sense of participation and ownership, helping to develop long-term connections. On the other hand, they are slower to complete, often even fewer are returned, and they are likely to be much harder to design, process and evaluate.

The disadvantage of questionnaires is that they can be rather impersonal. The advantages, however, are that they can be done at the respondents’ leisure (at the time or later), and can be sent back by post or e-mail (saving on staff time).

Interview
An interview could be individual or group, recorded on paper, video or audiotape, and as with a questionnaire, you can offer open or closed questions (or both). Human and face-to face, this is more spontaneous, mobile and adaptable than many of the other techniques, and can draw responses from those who might not write comments or answer questionnaires, including the less literate. However, the disadvantage is that taped interviews can be difficult and time-consuming to process.

N.B. When you are collecting information from people, make sure you explain clearly why you are asking them to do it and what the information will be used for. Also, once completed, monitoring records must be stored safely and confidentially.

3. Who’s contributing what?

Simple records can give you information about how much time your staff and volunteers gave, what tasks they undertook, and the problems they faced. You can also keep track of the ‘help in kind’ you receive, whether it’s voluntary work, discounts, organisation time, or gifts of facilities. Remember that every person in the monitoring process is giving a great deal – their time, their feelings and their opinions. Try to ensure that the process is rewarding for them too, and allows them to express themselves. Then the process becomes everybody’s property. Monitoring is a human and creative activity for all of us, if we get it right.

4. Feedback time

Measuring responses afterwards
Ask your workers, artists and other participants for their considered responses as soon as possible once the project comes to an end. Ask local businesses, schools, residents groups or social clubs, for their thoughts. Here is a range of interview formats, forums, collaborative criticism techniques and questionnaires that can help:

Group feedback sessions
A means of consulting all participants soon after a project ends and involving the considered responses of the people on the inside, whose future commitment is essential. The group situation should stimulate broad discussion, but it may be necessary sometimes to consult different groups separately. It’s vital that these sessions are prepared and facilitated, and do not become an arena for competing views. All points raised should be recorded, the group should not be pushed towards unanimity, and the recorder’s opinions must not intrude.

Focus groups (one-off) and forums (longer-term)
Small groups can be selected at random, or from partners and other stakeholders, allowing you to reach groups and individuals that might otherwise be excluded, and compare the views of different target groups. Mixing perspectives can be very creative. Forum groups allow ideas to develop over a longer period, and can draw in new members.

Published and virtual consultations Magazines, newsletters, letters, e-mail discussion groups, and web-based chat facilities can provide you with feedback over a longer time-scale than any other techniques. This gives people time, privacy and freedom to participate as they see fit, and can be really useful in formulating new policies, managing debates and drawing in new members. However, this lack of immediacy can make it hard to integrate into the deadlines of an evaluation process.

The final analysis This is the collaborative evaluation of the entire process, done by the project’s steering group (if it has one), the organisers and managers of the project, or perhaps with your organisation’s board or committee. This is when a picture of the project – as others perceived it – should emerge, and working together on the evaluation will strengthen the group’s mutual understanding. However, do bear in mind any time constraints you have if working to a deadline.

AFTER

The evaluation

Once you have collated all the information you gathered during the project, you should begin to make comparisons with your estimates from before it began. This is when monitoring becomes evaluation – the time for a detailed examination of the material you’ve gathered, and for reflecting on what it’s telling you. To help you with this process, consider the following:

1. Did you meet your aims and objectives?

Hard data* first (statistics):

• How did the numbers involved relate to your estimates?
• Did you achieve the outputs* you set yourselves, whether it was more trained samba drummers, a higher proportion of older people or more disabled people taking part? Did you reach your target group?

Soft data* too:

• Does your material show the quality of the experience? This might include increased ambition amongst participants, or new skills and confidence among the organisers
• Did the wider community respond as you hoped?

2. How much did it all cost?

A detailed comparison of your real income and expenditure against your estimates will tell you a lot about the way the project developed, and how you can ensure value for money in the future. Where were the unexpected windfalls, and why did the bucket develop holes where it did?

3. Time for changes?

• What else has the evaluation told you about the way the project works? For example, your timing, staffing, management support or other resources?
• Did the project run as planned?
• What do you deserve a pat on the back for? What were the high points of the project, and what were the lows? What worked well and what not so well?
• What have you learnt for the future? What do you need to change and put right, so you can go forward stronger?

Also, think about how well your monitoring and evaluation worked. How might you do it differently next time?

4. Time to share it out!

The best evaluation package in the world is useless unless you share it. How you share and use the results is very important.

• You might well need to present a written report to your funders. Sharing an honest evaluation with them will increase their respect for you, and show them that you mean business
• Make sure that the information is also distributed to your organisation’s board, staff, volunteers, and the artists or leaders whose talents you drew on. Your partners in the project will appreciate being involved
• If it’s going to be widely read, it’s important that it’s as lively, colourful and professionally produced as possible. Use pictures, quotations and clear headlines as well as tables and text
• As well as a written report, use your creativity to think of other ways to present your findings, such as: newspaper articles; radio interviews; your website; newsletters; training events; conferences; meetings; videos
• Think about who your audience is – for example, if you had been running a series of workshops with school children, a video, colourful newsletter or display boards with pictures and photos might get much more attention and involvement than a stuffy written report.

No! Don’t put it away for a rainy day!

The most important thing about M&E is that once the process is complete, you learn from your findings, and put them into practice.

Remember that evaluation is meant to be used

The results must be seen to be acted upon – otherwise you will have wasted all your participants time, as well as your own – and the process becomes meaningless. Agree how you will feed back what you have learnt into your daily work and the project’s future development.

Further information

This briefing offers an abridged version of the new Voluntary Arts Wales publication ‘Monitoring and evaluating your arts event… Why bother?’. To order a copy, featuring case studies, example forms and evaluation packs, contact the VAW team on T: 01938 556 455 or email: info@vaw.org.uk

For more information on Monitoring and Evaluation, here are some useful websites:

• www.ces-vol.org.uk – The Charities Evaluation Service. They have produced a useful booklet, First Steps in Monitoring and Evaluation which you can download for free from the site.
• www.evaluationtrust.org – The Evaluation Trust

Published by Foundation for Community Dance, January 2011
© Foundation for Community Dance. All rights reserved.

Every care has been taken in the preparation of this publication, but is not intended to be legally comprehensive or to replace professional/legal advice. No responsibility can be accepted by the publishers, author(s) or contributors for any errors, omissions or changes not for any harm, however caused, which results from the information presented.

This information sheet was first produced and published by Voluntary Arts: www.voluntaryarts.org/briefings