The story so far: Since 1998, the government has charged Arts Council England (ACE), along with all its other quangos, with showing how it would address social exclusion issues - the linked problems of unemployment, low incomes, lack of marketable skills, poor housing, high crime, bad health and family breakdown. And so was born, during 2000, a "social inclusion research project". This project was to look at three models of social inclusion/exclusion:
- Model 1 envisaged community-led work, where the initiative for the arts project originated locally
- Model 2 identified arts organisations whose main intention was to work with disadvantaged groups
- Model 3 proposed special partnerships, brokered by ACE, between the specialist organisations above and so-called mainstream funded arts companies.
The first stage of the work was a useful literature review by Helen Jermyn. Then followed a self-evaluation exercise by Gerry Moriarty which was intended to clarify arts organisations' own evaluation processes as well as all community arts organisations and inform ACE itself about the issues; this turned into a helpful (See Animated Summer 2000). Finally - something like five years after the work began - we have The art of inclusion - Helen Jermyn's external evaluation "exploring arts practices and outcomes". Jermyn's brief was to:
- Gather evidence
- Develop and test methodologies
- Evaluate the three models
- Identify the characteristics of successful initiatives
- Identify and explain failures
- Develop measures of success that could be applied across a range of work and genres.
The study was carried out on 28 organisations across a wide range of artforms; Settings included sheltered accommodation, under-5s groups, prisons, hostels for the homeless and community centres. The age range was wide. Consequently, and in spite of the high number of organisations included, the projects observed were more remarkable for their unique characteristics than for their common elements - a situation that seems to have taken the researcher by surprise, though it would be obvious to anyone who has observed more than a handful of such projects. To add to the complexity, there were no consistent modes of delivery and some projects clearly struggled to survive with high levels of transience in casual drop-in situations, making the tracking of individual journeys especially difficult. Although there was clearly some forethought in the choice of companies, and the Model 3 partnerships were specially set up to aid the research, it appears there was little attempt to achieve any kind of consistency across the target participant groups. Nor was there any consideration given to limiting the outcome parameters in such a way as to enable valid comparisons to be made. Where was the first rule of science - observe only one thing at a time?
In reading Jermyn's diligent and detailed study, one is immediately made aware of the issues eating away at the work: fuzzy definitions, lack of common parameters, validity of the samples, legitimacy of the interview/questionnaire process - and indeed the entire proposition. Their piranha-like feeding frenzy is evident even within the first few pages and it seems as much as she can do to maintain a steady course, fixed on the original objectives, even as they are being devoured by a shoal of caveats, doubts, deficient methodology and false assumptions.
The exclusion/inclusion fish surfaces early. Naturally enough the artists were not happy with either word - though they saw the need to speak the language of the funders (if only they could work out what it meant). Jermyn rapidly concludes that all artists want (indeed need) to include as many people as possible; that they don't differentiate their approach according to their catchment groups; that "inclusive" in one direction usually means "excluding" in another; that not a single artist or project was working to the precise include/exclude criteria that the research was intended to explore. Her conclusions could easily apply to any activity-based arts project involving artists, participants and support partners.
The main body of the report is devoted to examinations of the research objectives: project outcomes, partnership, sustainability, evaluation methods and good practice. Readers long-used to these issues will be relieved to learn that everything is as it should be. No new revelations, either positive or negative, emerged. It was disappointing, however, to read little or nothing about the artefacts themselves. The only honest evaluation resides in the products - performances, stories, pictures, exhibitions, parades, etc. This is the whole point of the work (as Jermyn later concedes) and it actually holds the keys to many of the other issues infecting the process.
In her conclusions Jermyn begins, tentatively, to dismantle the premises on which the research was set up. The three Models turn out to be merely different aspects of the same model. Which is no great revelation: completely community-generated projects are not that common, as last year's Voluntary Arts England's report, Doing it ourselves made clear. And arbitrarily brokered high profile schemes have no greater chances of success than those which emerge from low-profile specialist companies or as a result of a solid arts development policy. Even attempting one-to-one cause and effect observations between the arts activities and improvements in personal or communal circumstances is well-nigh impossible. We long thought we'd had the answer with François Matarasso's Use or ornament? - but Eleonora Belfiore's trashing of its methodology last year shows how far we are away still. In any case, it's quite conceivable that the proverbial life-changing episode seen as the Holy Grail of arts evaluators might in fact be the post-project evaluation interview itself, or a casual remark in the pub afterwards.
Jermyn rightly alludes to the need for more longitudinal research here. But this presents its own problems. Unless you create some sort of Big brother isolation (ethically dodgy even if practically possible) how can you be sure that it's the arts activity which is building self esteem (for instance) and not the fact of getting a job, winning the lottery - or even committing a successful burglary. "The Arts Council," she says, "needs to communicate clearly what it views as social inclusion work and how this is different from access or audience development". And did I detect just a hint of irony in her final paragraph, where she suggests that "it might be possible ... to build up a pool of indicators for use in groups of projects that have explicit aims ... or that focus on practice"? Surely that was precisely what the present work was supposed to establish?
Half way through reading The Art of inclusion I broke off and chanced on a piece by David Mamet for his column in the Guardian Friday Review. His eloquent explosion of verbal vitriol gave voice to my own increasing frustration at how public arts funding appears bent on emasculating the very activity it purports to support, through a combination of energy-sapping and brain-deadening bureaucratic 'experiments' which ultimately treat artists, audiences and participants as laboratory rats. Mamet writes: "What is the gift the new-fledged bureaucrat holds? ... the knowledge that others are inferior. The young and privileged, saved from the transition from adolescence to adulthood, are the perfect soldiers of the arts bureaucracy - they are ignorant of artistic interaction, they have never had to please an audience, they have never had to tie their fortunes to their judgment, they are in thrall to their superiors and to their superiors' rendition of the world". If Jermyn seems caught up in this insidious virtual reality, it is no reflection on the quality or sincerity of her work, it's just difficult to imagine what useful purpose it will serve in the community of those who try to survive by committing art.
Her own - well-disguised - frustrations occasionally rise to the surface. She questions the ethics of dissecting participants' often fragile and poorly articulated engagement with their recent arts experiences: "there was an issue about the extent to which the researcher should 'intrude' into that environment and make demands of participants ... it might have been desirable to have administered a baseline questionnaire at the start of all projects and tests to provide scientific measures of participants' self-esteem." The dilemma for Jermyn is that, in hinting at the invalidity of her results, she is effectively invalidating the whole point of the research exercise itself.
But maybe we're beating ourselves up too much about this apparent deficiency in methodology. Most people in the social science world - think of the huge swathe of reports coming out almost daily from the Rowntree foundation - seem to be able to cope with partial methodologies; with studies that simply add another brick in the wall rather than try to create a whole edifice in one go (but, it has to be said, with knowing rather more about 'distance travelled' measurements that this report seems to). And the question is then perhaps begged - who benefits more by appearing not to be able to answer the question about what good community arts projects do: the community artists who might then feel themselves relieved of an added burden ... or the funding bodies who might then feel they don't have to worry about funding these projects?
And yet it's good to read what Jermyn says in her descriptive accounts, set out largely as a series of case studies. Arts projects in any context are invariably rather messy and unpredictable, though it would be impossible to find a single artist or support worker who was not committed to making the work as good as it can be. The sympathetic reader will empathise with both the successes and the failures since, in spite of the diversity of work, the journeys themselves are often very similar. It is unfortunate, though, that Jermyn wasn't able to approach the task with a more sophisticated understanding of how the real world is, as opposed to having to work with the rather naive constructs with which she was presented. She seems not to have used her own extensive literature survey to inform her methodology. One of the more surprising features of the whole affair is how little ACE itself seems capable of learning from its own previous work. In 1981 it published a report of mine - New approaches to new music - which tackled many of the present issues, albeit confined to a specific genre. I developed the themes of partnership and best practice working extensively in two subsequent publications by the (then) London Arts Board: Musicians go to school (1997) and All together now (2001). That an organisation responsible for developing and implementing national cultural policy should be operating on such a low level of corporate knowledge is both depressing and scandalous.
Andrew Peggie is a composer and community project leader. His report, Tuning Up - a new look at instrumental music teaching in London, was published by Sound Sense in 2002.
A version of this article first appeared in Sounding Board the magazine of Sound Sense the national organisation for Community Music, with whose permission it is reprinted. Subsequent to this research being initiated Arts Council England commissioned Dance included 6 projects reported in Animated, Summer 2003. Helen Jermyn will again be producing a report about these projects.