Francois Matarasso Evaluation, Talks, Thinking

Some thoughts on monitoring, evaluating and researching culture

Introduction

Last week, I participated in a round table discussion organised by the Cultural Value Scoping Project at Tate Modern. The initiative is a collaboration between the Arts and Humanities Research Council, the Paul Hamlyn Foundation,  Arts Council England and King’s College London and King’s College London. It aims to explore how research, evaluation, evidence-building and analysis into cultural value might best be supported in the future. Those present included people working in cultural institutions, academic researchers and others involved in the field. About 15 speakers were invited to contribute a 10 minute reflection during the day, with the rest of the time given over to some fruitful discussion. My contribution  – the notes are below – addressed the question:

In what concrete ways could the new platform dedicated to research and analysis into cultural value support the cultural sector?

tate-modern-2

Value is subjective

There is no universal agreement about what is valuable beyond a small number of essentials, such as life, shelter, education and so on. Because there is agreement about these things, they are described as rights, not values. But there aren’t many: the Universal Declaration of Human Rights has just 30 articles.

Everything beyond that (and nowadays, even some of that) is subjective, a matter of choice, including culture. Human beings do not all value the same things. We sometimes find what others value ridiculous or even offensive, including culture. But in a society where human rights are respected, we tolerate other people’s choices.

Voltaire said, ‘Dissension is the great evil of mankind and tolerance its only remedy’. And politics is the means by which we negotiate our disagreements about value and the limits of our tolerance.

Value is comparative

Value is also comparative. Even absolute value, such as the right to life, is evident only when compared with alternatives: being alive is better than being dead. Disagreements about value are also understood comparatively. A democratic society is better than an autocratic one. A kidney dialysis machine is worth more than a public sculpture. Both of these statements are choices that reflect comparisons in the context of values held.

One of tasks of politics is to negotiate the relative value that we give to different things, expressed in the abstract system of money. How we gain and spend money – individually and collectively – is an expression of values, choices made on the basis of comparison. It’s messy, sometimes ugly and often unfair, but we haven’t got a better way of preventing our disagreements about value being resolved by violence.

Cultural value is political

It is many years since I began trying  to understand the value of participation in the arts, first in my own work and then in the work of others. Often that has involved working with people who want to prove the value of art or culture. I understand that desire. If your work depends on comparative choices made by other people, it is natural to want to persuade them that what you do has relatively high value.

The problem is that value, which is subjective, cannot be proven because proof is an objective concept used in the natural sciences. We can prove that the melting point of iron is 1,538 degrees Celsius. Once proven, it is no longer a matter for debate, or an ‘alternative fact’. We cannot prove the value of culture. We cannot even agree a definition of culture.

Acting politically

Those who want culture to have a greater importance in society, public policy and budgets, must act politically, not scientifically. They must engage others – and particularly those who disagree with them about culture’s value – in persuasive debate. Data, evidence and knowledge may all help in that task, but so will the ancient tools of politics: rhetoric, argument, emotion and the rest.

Trying to prove cultural value is like hunting the Snark, and while we are occupied with that fruitless search, we fail both to engage effectively in political debate about culture’s value and to learn from the knowledge that our research does produce.

The purpose of cultural research

If we separate debating value from understanding value we enter into more fruitful territory.  Instead of trying to find proofs that can persuade others to change their beliefs – which, insofar as it can be achieved at all, is a political task best undertaken with political means – we can look for knowledge that can influence the actions of those who already hold certain beliefs about the value of culture. In this context, there is practical value in testing individual subjectivity to find the common ground for shared judgements.

The proper purpose of cultural research, I believe, is not persuasion but improvement.  Whether that research is undertaken independently by academics, commissioned by a cultural actor, required by a funding agreement or simply done by a cultural organisation as part of its everyday operations, its purpose should be to increase knowledge and understanding of cultural activity.

How and why

As such, its questions are concerned less with whether than with how and why. Whether something happens is important, of course, especially to funders and in political discussion. How and why it happens are much deeper and more powerful questions because they have the potential to challenge the cultural actor’s own beliefs and assumptions. They have the potential to influence the way art is created, managed and presented, the programmes offered and the ways in which people are able to engage with them, the interpretation and meaning of culture itself.

In my experience, however, it is not only politicians who are uninterested in how and why cultural experiences produce value for people. The cultural profession – with exceptions – is not very interested either, especially if the insights research produces might point to ways in which they could productively change. As far as cultural policy and management goes, it seems all but impossible to get beyond ‘whether’ and the mirage of proving value.

What might be done?

As we all know, the first thing to do when you’re in a hole is to stop digging. And I believe that research and evaluation into public cultural investment is in a deep hole. Let me clarify that I’m concerned here only with what is done within, and required by, the public cultural sector, whoever is doing it. I’m not referring to the growing body of independent, mostly academic research into the theory, policy, work, management or practice of culture.

The principal reason, as I’ve suggested, is that professional engagement with cultural value  is geared towards an unavailable proof, but over the years a host of subsidiary reasons – such as the adoption of inappropriate public management concepts and practices – have also contributed.

When I say, stop digging, I mean let’s take a pause and look around. The Cultural Value Scoping Project is an opportunity to do that, to ask what we are trying to understand and why. Then we might review what we are currently doing and apply a simple test to it: Does the use to which it is put justify the cost of producing it?

If it were possible to back out of the hole we’re currently in, I’d suggest a different approach to monitoring, evaluation and research, which would begin by distinguishing clearly between the three kinds of activity.

Monitoring

The foundation of knowledge about cultural life must be factual and at least where public spending is concerned, there is a great deal that can be known simply through the process of making grants and monitoring the resulting activity. It should be possible, especially now that the grant-making process is computerised, to design a system that can produce reliable data about the nature and extent of activity supported, say, by Arts Council England.

Accountability in grant making should be achieved mainly through monitoring, not evaluation. If the activity was assessed as having public value at application stage, and monitoring shows it to have been completed as expected, then it should follow that its value was delivered in terms of policy and spending decisions.

Refining the application and reporting requirements so that they are factual, consistent over time and used in across the sector, would enable ACE and independent researchers to analyse the resulting data from different perspectives and produce reports that highlight patterns or inconsistencies.

Evaluation

With monitoring processes that are capable both of demonstrating the public value of grants and providing big data that can be investigated for trends, the evaluation process can be freed from the need to justify spending decisions. It can then become a truly open-minded inquiry into how and why results have been produced from which things can be learned and changes to practice made.

In this context, the Arts Council’s recent focus on self-evaluation is very welcome. Evaluation is a critical part of creative work which artists and arts organisations should be doing naturally and in order to improve their work. It will be very difficult to change a culture that sees evaluation principally as a tool of self-justification. But with time, training support and a greater use of monitoring in assessments, some progress could be made.

At the same time, self-evaluation needs to be reported in a form that can be shared internally and with peers. Again, the collection and cataloguing of this material would provide a huge body of data that could be investigated by researchers.

Research

With more reliable data produced by reformed monitoring and evaluation activity, a body such as the Arts Council would be in a position to commission a range of external and independent studies of cultural activity. Some of that might test the self-evaluation process by looking at a representative sample. Some of it might use the data in order to answer specific questions about sub sectors or kinds of practice. This approach would allow the regular publication of good quality thematic research reports that can contribute to debate within the sector and at the Arts Council and thus lead to changes in policy or practice.

In conclusion

This is no more than a sketch of some ways in which research into cultural value might change to become more influential than it has been to date. This may not be the best approach and, to be honest, even if it were I’m doubtful whether it is achievable given the  entanglement of judgements about the value of individuals and of practices with financial decisions. Better research can give us better knowledge but in the end, politics will determine the decisions we make.

•

Post-script

I’ve been writing about cultural value in one way or another for a long time: some of my previous posts on the subject can be found here, though I’m aware that there’s a certain amount of saying the same things again and again:

Francois Matarasso Evaluation, Experiences, Thinking

Below the radar

Wrangle 1There’s a lot of interest at the moment in the value of culture and cultural value (not the same thing), both in academic and policy terms, and I’ve touched on it before. Research programmes, initiatives like the Arts Council’s Creative People and Places and political speeches can all enrich the debate about how we understand culture, its purpose, value and meaning.

For example, in reporting on a recent symposium in the US, Geoffrey Crossick, Director of the Cultural Value Project, observes that: ‘the standard surveys of participation – of which the DCMS/Arts Council England’s Taking Part is just one example – have become a necessary part of the evidence base for those seeking to make the case for public funding of the arts’, going on to ask some important questions about the conceptual and practical difficulties of assessing levels of arts participation.

But what is the relationship between a level of participation in the arts and a case for public funding? Would a high level of participation justify that funding? Surely not in itself, since people participate enthusiastically in all sorts of things that do not receive (or request) taxpayer subsidy. Perhaps a low level of participation would be a better justification, since it would be possible to argue that this was a bad thing – but on what basis? – which might be remedied by subsidising more art and better access to it. But if the level rose or fell, what would be the appropriate policy response?

I hope that questions such as these, which demand the attention of philosophers as much as statisticians, will get more attention in the present lively discussion of cultural value.

Their practical and concrete nature was brought home to me a few days ago, when I met a team of bell ringers at Wrangle, in Lincolnshire, for a project about the church’s place in artistic and community life. These men are practitioners and custodians of change ringing, an ancient, uniquely English musical tradition that makes equally high demands on the ringer’s stamina, concentration and intellect. Bell ringing is  unlikely to figure in most surveys about participation in art, though clearly not for reasons of cultural value: the drawing of boundaries is always revealing.

Francois Matarasso Evaluation, Talks

A Three Pipe Problem (MCV8)

‘What are you going to do, then?’ I asked. ‘To smoke,’ he answered. ‘It is quite a three pipe problem, and I beg that you won’t speak to me for fifty minutes.’

Arthur Conan Doyle (1891)

The term ‘cultural value’ appeared in British policy discourse about 10 years ago, notably in Capturing Cultural Value, a pamphlet by John Holden for Demos. Its use has grown quickly and it is now central to several campaigns, research programmes, and debates (an overview may be found on the website of The Warwick Commission on the Future of Cultural Value).

Whilst this extensive and varied engagement in the value of culture is very welcome, particularly in a British context where such discourse has previously been limited, there is neither clarity nor consensus about what cultural value actually means. The use of an ill-defined and consequently slippery concept to advance knowledge and inform policy is concerning because it risks obscuring the very tensions and complexities its advocates set out to address. Consequently, the term’s intellectual value is eroded even as more capital is invested in its ideological value.

Here’s a brief note developed from a talk given to an Expert Panel for the AHRC Cultural Value Project and supported by the University of Exeter. It considers four areas where discourse around cultural value seems to me confused, including the distinctions:

  • between value and values;
  • between capacity value and effect value;
  • between intrinsic and extrinsic effect; and
  • between what is observable and what is controllable.

There are others, and there is much more to be considered if a stable basis for discussion could be established, so this note should be considered as no more than a beginning.

Francois Matarasso Evaluation

The value of everyday evaluation

‘We all began with the best of intentions, and then ran slap into reality. Very quickly, it was evident that the project was altogether impossible.’

So writes Elizabeth White (Director, British Council Azerbaijan) about a production of Martin McDonagh’s Leenane Trilogy, performed in Baku by Azerbaijani actors in their own language. Her account of the project – which ended triumphantly against all the odds – can be read online here, and is full of insights into theatre, diversity and intercultural dialogue.

Fascinating as it is, though, I was also struck by how blogging and the Internet can enable new kinds of evaluation. Ms. White’s piece is, in effect, a participant’s evaluation of the project. It describes the rationale and development, the obstacles that had to be overcome (or sometimes circumvented), the performances and the responses of actors, critics and government. She writes candidly about the difficulties and there is a lot to be learned from the piece.

There will, I suppose, be a formal evaluation, describing all this again with more methodological rigour at greater length for fewer readers. But this critical reflection, arising spontaneously from a committed creative process is probably more valuable to the writer’s peers and other practitioners, despite having the appearance of  a simple blog post. Because it is public, she has taken care both to tell her story well and to focus on what are, for her, its key experiences and lessons.

There are good policy reasons for evaluating arts programmes although, since the reality is often burdensome and unproductive, it has far less influence on artistic and cultural practice than it might. A straightforward, public and reflective self-evaluation, of which this piece is a fine example, could do a great deal to nourish the creative practice of artists, managers and even funding bodies.

Francois Matarasso Essays, Evaluation

Quality in community art (MCV7)

The McMaster Review

Fashions sometimes change quickly in the arts, and in arts policy too. In July 2007, the then Secretary of State for Culture, James Purnell, commissioned Sir Brian McMaster to undertake a review of ‘excellence in the arts’. The report was published in January 2008 and for a few months, the art world buzzed with the idea. I wrote something myself about it at the time for the Voluntary Arts Network.

Then Mr Purnell began his rapid exit from politics, via a stint at the Department of Work and Pensions. There was a change of government, and the buzzwords became philanthropy and digital. The DCMS pages about Sir Brian McMaster’s work now reside in  the National Archives.

Elephant Yes

A new essay on quality in community arts

But the issue of excellence remains a central preoccupation of artists, whatever the policy world does. Indeed, one of the questionable aspects of the whole exercise was the implicit assertion that people had lost sight of excellence. I’ve never met anyone working in the arts or cultural sector who was not committed to producing the best work that could be achieved. The challenge has always lain in the fact that there are many ideas of what ‘best’ or ‘excellent’ means. 

I’ve just published a new essay looking at on some of the issues in the context of community arts practice in a UNESCO e-journal on Multi-Disiplinary Research in the Arts, at the University of Melbourne. Click here to download the essay or follow this link to the University of Melbourne website to download the whole journal.

Abstract

This paper takes Creative Progression, an arts programme whose aim was to support the progress of the homeless participants towards health, wellbeing and independent living, by Helix Arts (Newcastle-upon-Tyne, UK), as a case study through which to which to reflect on the meaning and assessment of quality in participatory arts. It considers the use of the word ‘quality’ by arts professionals, and the recent focus on ‘excellence’ in British cultural policy discourse, suggesting that the first term is often confused with ‘good’, partly because of uncertainty about concepts and partly because doing so may help to avoid potential challenges about values.

The paper then identifies five stages in a participatory arts process—conception, contracting, working, creation and completion—considering in turn some of the problems of defining or securing quality in performance, using the experience of the Creative Progression programme as a framework. It concludes by suggesting that, given the inevitably subjective nature of both arts practice and artistic experience, it is impossible to define fixed standards of quality in performance or outcome. Nor, indeed, would it be desirable to try to do so.

However, the quality of self-awareness and critical reflection exercised by artists working participatory contexts, and the extent to which that reflection is open to all participants, is central both to an ethically-defensible process and to the probability of programmes achieving their stated goals.

 

Download Creative Progression: Reflections on quality in participatory arts, François Matarasso 2013

Francois Matarasso Evaluation, Thinking

What is an artist? (MCV6)

On the face of it, it’s an odd question. After all, we know what a baker, a dentist or a banker is and, more to the point, it’s not difficult to say whether a person is, or is not, a baker, a dentist or a banker. In the past, the same was probably true of artists, because they were more commonly described in functional terms such as painter, actor, musician and so on.

But the term ‘artist’ is not functional. There is no verb ‘to art’ that one can set alongside  ‘to paint’ or ‘to play’. (‘To create’ goes some way towards it, but the act of creation is not exclusive to artists.) In our present concept, artists are: they do only incidentally. So Sibelius remains a composer through the  last, silent 30 years of his life.  Rimbaud is a poet though he wrote no poetry after the age of 21.

Today, we speak of being an artist, as if it were an existential condition. Many artists talk about themselves in ways that reinforce the idea that their art practice fulfils an essential quality of their being. Stories of precocious achievement or, less verifiably, of an early sense of mission, are common in interviews with artists. Indeed, the use of terms like mission, calling or vocation, are an instance among others of the transference of religious metaphors to the artist’s identity. There is an interesting study to be made of the similarities in the discourse of contemporary arts biography and medieval hagiography,

Biennial

The shift from doing to being in contemporary ideas of the artist is connected with the rise in status that artists have experienced during the past century. Being recognized as an artist has probably never been as desirable as it is today. It brings not only admiration but also a licence to transgress social norms not available to the rest of society. Indeed, the willingness to be different becomes a further sign of the moral and spiritual integrity that we ascribe, on somewhat tenuous evidence, to the artist today.

Many societies value artists to the point of awarding them distinct financial advantages, most commonly in the form of grants. Ireland goes further, allowing artists to earn up to €40,000 from their work without paying any income tax. Many artists understandably consider such measures a high form of political enlightenment, but cleaners and shelf-stackers on minimum wage might see things differently. Be that as it may, it is evident that a social role that brings status and cash, in addition to its lifestyle qualities, is attractive to many people. So it is in the interest of those who are recognised as artists—and of their admirers—to control who can join the club.

The problem they face is that there is no objective way of recognizing an artist than by their actions. Artists are people who pursue the creative activities that produce art. The degree of skill, originality, ambition, resonance or feeling involved is independent of the act. A person may be act as an artist well or badly, just as a baker can bake well or badly. The assessment of quality will include objective criteria (e.g. about handling of materials) and subjective ones (e.g. whether the result gives pleasure). But a person who produces an unpleasant loaf of bread is no less a baker than a person who produces a dull painting is an artist. They are just not good ones.

Bluecoat

Most professions are concerned with quality control, so that the work of poor practitioners does not undermine confidence in the standards of the rest. But the art profession—if such it is—seeks often to erect its barriers further upstream by denying many people who practice art, including the very large numbers of people who do so for the love of it, the status of artist. Art school education, social status, class, critical acclaim, commercial success—none of these are useful in determining a person’s right to describe themselves as an artist, or to ask others to recognize them as such.

To divide artists between professionals and non-professionals is not to make a quality (or status) judgement but to recognise a difference of social occupation. A professional artist is simply someone who acts as an artist most of the time, perhaps to the exclusion of other occupations, and in ways that other people are willing to support financially. A non-professional artist is someone whose artistic action is either only a small part of her daily life or not of a kind that other people admire enough to support financially. This distinction can be made without recourse to aesthetics. Being a professional artist is not equivalent to being a good artist—unless your measure of quality is monetary. A person can be a fine artist but not wish for various reasons to place their work in a market.

So what is an artist? Someone who enacts the creative processes that produce artworks. What is a good artist? That is a question for another day.

Yayoi Kusama

Francois Matarasso Evaluation, Research

Portraying the change

How can you describe a cultural programme that has extended over 14 years, 9 countries and 3,000 projects? How do you account for its outcomes, the change it may have contributed to, and the effects on culture or society?

“The project allowed people to spend their time in a more creative and constructive way, so we all contribute to enriching life in our own towns. We feel good both at home and across the border, at our friends’ homes.”
“The first thing I learned was to work in a group that was completely heterogeneous, to collaborate creatively with a group of people who were completely different. That was one of these wonderful things, and back then I thought the biggest problem was that we were totally different in the group”
“There is a saying which I strongly believe in: “If you want to go fast go alone, but, if you want to go far let’s do it together”. Projects like the Revival of City Squares in the Balkan Cities take some time to be implemented but once done they become a flagship for future generations due to them being done together with the community and not just “for” the community as a lot of politicians would say.”
“Our most important activity was making Kravica In Words And Pictures, a book and video that preserves and promotes our traditional customs. They are not the work of one person—all Kravica’s residents gave their contributions.”
“If we do not even talk on what we disagree about, we make space for a new conflict in the futre. That is what the project had taught me.”
“It’s not all about money; much depends precisely on the willingness of citizens to perceive the city and urban spaces as their own – as the place where they live, not as something belonging to somebody else.”
“For me it was very important to learn about the regional situation. Ever since I have felt not only as a performer from Belgrade or Serbia, but that I belong to the regional scene of the Balkans.”
‘The street is no longer my home: it’s my workplace. I know that I’m doing a good job. I’m not ashamed of this. People are nice to me, they are smiling – they are not threatening me like some beggar.”
This project has affected me because when you open an issue, the questions do not stop after the realization of one work, but the matter may be treated in different ways and the artist can keep returning to the same topic over and over again.
“NOMAD helped me find myself in contemporary dance. It is like finding yourself. I knew that choosing dance instead of philosophy was the right decision. It was a kind of revelation for me.”
“I always say, I can do everything with the peoples’ help. Doing the initiatives together is what makes this village so beautiful and developed”
“I don’t like projects that end in their last day of funding … I like projects that have a longer impact. And these theatre performances had a great impact across Europe … particularly, in breaking down the views of Roma people.”

That was the challenge faced by the small team of people who have managed the Swiss Cultural Programme in the Western Balkans in recent years, and which has just come to an end. I helped them with the review process and wrote a report that tries to show the wood, if not all the trees.  The report, Cultural Encounters, is available to download here.

But this work is about people, people of all ages and cultures and backgrounds, coming together to talk, learn, make art, share ideas, challenge themselves and each other, create, demonstrate commitment, think, perform, laugh, sometimes cry, and ultimately help build a better society one person, one place, one day at a time. So we had the idea of interviewing 24 people who had been involved in the programme in recent years – not the organisers or project leaders, but those who’ve taken part, given their time and benefited from the experience.

Alongside the interviews, conducted by experienced journalists in the relevant languages, we commissioned portraits from local photographers. The resulting exhibition, Portraying the Change, was shown at the National Gallery of Bosnia Herzegovina, in Sarajevo, last week, at the launch of the event marking the formal end of the programme. Half of those portraits, with short extracts from the interviews, can be seen here.

All the portraits, and longer extracts from the interviews, are included in Cultural Encounters. I hope that these voices speak for themselves.

SCP Report Cover 2

Francois Matarasso Evaluation

Arts and Health evaluation (MCV5)

Learning about evaluation

Fifteen years ago, at the end of Use or Ornament?, I rather naively raised the possibility of producing an evaluation handbook for arts projects. I’d been concerned with the conceptual, ethical, methodological and practical challenges of arts evaluation for some years, first as an arts practitioner and then a researcher. Even so, I was still only beginning to understand the true complexity of those challenges.

Today, it seems obvious to me that the idea of a single evaluation resource was not just unrealistic but misguided: there are many good routes through this territory (as well as some dangerous ones that lead people over cliffs or lose them in deserts).

Since 1997, I’ve produced notes to accompany evaluation workshops, given lectures and seminars on the subject, written a short book for Arts and Business (Did it make a difference?, 2001) and even tried an arts evaluation wiki. Of all this it is the workshops that are most valuable, because they enable people to question, debate and share experience. They demystify ideas that can seem obscure and even oppressive and, like good community arts practice, at their best they are empowering.

The arts in community health and wellbeing

Last year Liverpool Primary Care Trust (PCT) asked me to write some evaluation guidance to support the community groups funded through the Grass Roots arts programme they run with the City Council. This is one of the legacies of Liverpool’s year as European Capital of Culture in 2008. Grants of up to £5,000 are given to community groups for arts programmes that can support broad public health objectives, as part of Liverpool’s Decade of Health and Wellbeing

In 2010, the projects included theatre with homeless people by Collective Encounters, samba workshops and a parade, art with young disabled people, and work by Chaturangan with Liverpool School of Tropical Medicine among much else. The results have often been impressive, perhaps because the small grants have given voluntary groups just the resources they needed to do something they were excited about achieving.

Just another Arts and Health evaluation resource

In order to justify investing health funds in community arts projects, the PCT needed credible evidence of their value. But many of the groups are run by volunteers with little or no knowledge of formal evaluation. Their activities are often informal and open, especially if they are trying to reach people who have little or no contact with public services.

It was essential therefore to develop an evaluation approach that was workable and interfered with the activities as little as possible. It also had to be proportionate to a grant of a few thousand pounds and a project that might last only a few weeks.

The guidance and planning template were introduced to those who’d been awarded grants at a workshop at the Bluecoat in September 2011. They subsequently used the materials to plan and report on their work throughout 2011 and 2012.

It has proved to be a useful tool. The PCT have made some adjustments based on that first experience and the revised version will be used in the new round of projects, starting this autumn.

Download the guidance

Although these tools relate to a specific programme, they may also be helpful to someone intending to evaluate a community-based art in health project, so I’ve added them to this site. To download copies in Word files, click on the links below:

This is now only partly my work, but its development is just what should happen to such materials. They are not authoritative statements, such as I probably had in mind when I wrote of a handbook 15 years ago. They are just imperfect tools that can, for all their limitations, still help us understand what happens in community arts programmes better.

Francois Matarasso Evaluation, Research, Thinking

The problem of ‘impact’ (MCV 4)

A conversation with Peter Wright at the Creative Communities 3 conference made me realize that it might be useful to explain why I no longer use the word ‘impact’ in relation to people’s experience of the arts.

Peter, who teaches at Murdoch University in Western Australia, gave a rich presentation about his work with Big hArt, which he has been researching for some years. We spoke at length afterwards about the project and more generally about ideas of social arts practice and its evaluation. Not surprisingly, since conference speakers had often used it, we talked about the word ‘impact’. When Peter subsequently asked me where he could read more about this, I couldn’t point him to anywhere that I’ve set out this thinking, so here is a short explanation.

I should say first, though, that I haven’t always seen this. 15 years ago, when I worked on a research programme entitled ‘The social impact of participation in the arts’ I took up the phrase (which others had coined by adapting thinking about the ‘economic impact of the arts’) without hesitation. What I write here is the result of much subsequent thinking and research and many valuable conversations, such as the ones I had with Peter.

•

The word ‘impact’ is a concept in physics and specifically mechanics, that describes what occurs when force strikes an object or when two bodies collide. It has been transferred to many other fields, and it is now commonplace to speak of the economic impact, environmental impact or social impact of policy decisions.

In moving from mechanics to, say, public policy, the word changes from being a technical descriptor to being a metaphor. Nothing wrong in that: we use metaphors constantly to understand and describe experience. But language also shapes how we think and my doubt here is about the effect the use of this metaphor might have on how policy interventions in general—and social arts programmes in particular—are imagined by those who commission, deliver, experience and evaluate them.

Here’s the problem.

Impact suggests a force striking an object, as a die impresses itself on a blank.  It implies an active agent and a passive recipient, a subject and an object. One might even say that it unconsciously reflects the gendered imagination that divides the world into active male and passive female. And like that imagination, it’s freighted with potential violence.

In this thinking, the social art project is conceived as an experience whose ‘impact’ changes those who take part. (And for ‘change’ one should read ‘improve’, in the terms of the problem-solving mission jointly identified by the artist and the commissioner.) Participants are not seen as active, autonomous individuals, capable of interpreting, responding to or even rejecting the experiences of an art project or the intentions of those who have offered it.

But that, even if it were not an offence to ethics and democracy, is just a fantasy. Artists cannot control how others interpret their work, whether that work is a performance, a text or a workshop. Governments cannot command through social interventions guaranteeable changes in people’s lives or behaviour.

We all interpret our experiences, and especially the ambiguous ones we gain through art, in our own way. That is why two people can sit side by side for two hours watching a film or a play and come away with completely different responses to what they have seen. That is why no two people have ever read the same book.

Impact is not just an inadequate metaphor for imagining how people experience, and are affected by, the arts. It is a deceptive one, reinforcing unequal power relations that divide societies into those who know and those who don’t, those who are acceptable and those who are not, those who fit and those who need to be changed.

Since it is also a completely inaccurate account of what actually happens as a result of participation in the arts, it is dangerously misleading. It encourages politicians, commissioners and even artists in the delusion that because art and culture have profound effects on people those effects can also be commanded. The effects on policy can be exceedingly perverse.

Will any of this stop people using ‘impact’ as a metaphor to explain the effects of art and to justify investing in it? Hardly. The idea is far too embedded and too flattering to those who use it to be set aside soon. But it would be a step forward if the artists who work in this field and the researchers who write about that work questioned the term a bit more.

Francois Matarasso Evaluation

A Different Heartbeat (MCV 3)

The gold standard in evaluation

Twenty years ago, when I began thinking about how to improve the evaluation of community arts programmes, I turned to practice in health and medicine. I had been working with other artists in hospitals, care centres and other institutions for several years. Two large projects had been funded by the Department of Health, the Social Services Inspectorate and the Joseph Rowntree Foundation and required substantial evaluation.

But more importantly, perhaps, I shared the widespread perception that the health profession owned the gold standard in outcome evaluation: the randomised controlled trial (RCT). My assumption then was that policy makers would consider the value of participation in arts programmes only if the outcomes could be down in research that compared with that medical benchmark.

Like many people educated in the arts, I was willing to defer to scientific methods and epistemologies, without knowing exactly to what I was conceding authority.

The limits of medical research

But the more I read about health and medicine, about policy formation, evidence and evaluation, the more questionable that deference came to seem. In 1996, I published Defining Values, a working paper with some preliminary ideas (as part of the research eventually described in Use or Ornament?). In it I noted some gaps and inconsistencies in health evaluation, including the fact that RCTs are required only for drugs and not for procedures, which are much more unevenly evaluated. I also found that some drugs established before RCTs became universal have never been tested in this way: since they are known to be effective, such a trial would require denying them to patients who would benefit from them.

Most strikingly, I came to see how health policy – including major changes such as NHS reorganisation – was rarely based on RCTs, clinical evidence, or even piloting. It was principally driven by the beliefs of politicians and their allies.

In itself, that is not wrong. How can we act, except on our beliefs? But if we deceive ourselves about the evidence base on which those beliefs are based, if we claim evidential authority as a way to quash objections or stifle argument, we step onto a very slippery path indeed.

Looking for alternatives

In subsequent years, more experience in research and further reading have made me increasingly wary of the misapplication of scientific – and specifically medical – evaluation models to the complex experiences that people have through the arts. I value those methods in themselves. I applaud the efforts of people such as Ben Goldacre and Mark Henderson to defend the integrity of scientific method from misuse, ignorance and misrepresentation. But simplistic attempts to transfer them to art and culture are unwise and misleading.

So in recent years I have looked for alternative ways to account for (a concept I prefer to ‘evaluate’) the outcomes (preferred to ‘impact’) of people’s experiences of the arts. An account describes rather than judges; it is modest in its nature, allowing the reader space to form a response. The approach also recognises and values, though not uncritically, the particular ways in which textual and visual language make sense of human experience.

It has also encouraged me to test existing research norms and practices, something that may be easier to do from the position of a writer or an artist than that of a researcher bound by the procedures of her or his institution; that, however, is for another time.

A Different Heartbeat

The images that illustrate this post are taken from A Different Heartbeat, an account of a residency by musician Patrick Dineen at the Kidney Dialysis Unit, Royal Liverpool University Hospital, in spring 2011. They were done by Mik Godley, with whom I’ve since been working on a project about artists in old age, during the several visits we made together to the ward to speak to patients, staff and others involved in the residency.

A Different Heartbeat describes an intimate, small scale arts in health project, and places it into a wider context of questions about chronic illness, well being and the nature of benefits. It is an essay, a reflection on particular experience, and so about as far from a randomized controlled trial as it could be. But perhaps in that difference is something of value also.

In Defining Values I quoted the medical sociologist, Clive Seale, writing that:

The results of research may often influence policy makers in a more diffuse way, seeping into their thinking, in a manner that eventually provides enlightenment.

In the unavoidable absence of certainty, enlightenment is good enough.

Download 2012 A Different Heartbeat (PDF 940kb)