Introduction

Last week, I participated in a round table discussion organised by the Cultural Value Scoping Project at Tate Modern. The initiative is a collaboration between the Arts and Humanities Research Council, the Paul Hamlyn Foundation,  Arts Council England and King’s College London and King’s College London. It aims to explore how research, evaluation, evidence-building and analysis into cultural value might best be supported in the future. Those present included people working in cultural institutions, academic researchers and others involved in the field. About 15 speakers were invited to contribute a 10 minute reflection during the day, with the rest of the time given over to some fruitful discussion. My contribution  – the notes are below – addressed the question:

In what concrete ways could the new platform dedicated to research and analysis into cultural value support the cultural sector?

tate-modern-2

Value is subjective

There is no universal agreement about what is valuable beyond a small number of essentials, such as life, shelter, education and so on. Because there is agreement about these things, they are described as rights, not values. But there aren’t many: the Universal Declaration of Human Rights has just 30 articles.

Everything beyond that (and nowadays, even some of that) is subjective, a matter of choice, including culture. Human beings do not all value the same things. We sometimes find what others value ridiculous or even offensive, including culture. But in a society where human rights are respected, we tolerate other people’s choices.

Voltaire said, ‘Dissension is the great evil of mankind and tolerance its only remedy’. And politics is the means by which we negotiate our disagreements about value and the limits of our tolerance.

Value is comparative

Value is also comparative. Even absolute value, such as the right to life, is evident only when compared with alternatives: being alive is better than being dead. Disagreements about value are also understood comparatively. A democratic society is better than an autocratic one. A kidney dialysis machine is worth more than a public sculpture. Both of these statements are choices that reflect comparisons in the context of values held.

One of tasks of politics is to negotiate the relative value that we give to different things, expressed in the abstract system of money. How we gain and spend money – individually and collectively – is an expression of values, choices made on the basis of comparison. It’s messy, sometimes ugly and often unfair, but we haven’t got a better way of preventing our disagreements about value being resolved by violence.

Cultural value is political

It is many years since I began trying  to understand the value of participation in the arts, first in my own work and then in the work of others. Often that has involved working with people who want to prove the value of art or culture. I understand that desire. If your work depends on comparative choices made by other people, it is natural to want to persuade them that what you do has relatively high value.

The problem is that value, which is subjective, cannot be proven because proof is an objective concept used in the natural sciences. We can prove that the melting point of iron is 1,538 degrees Celsius. Once proven, it is no longer a matter for debate, or an ‘alternative fact’. We cannot prove the value of culture. We cannot even agree a definition of culture.

Acting politically

Those who want culture to have a greater importance in society, public policy and budgets, must act politically, not scientifically. They must engage others – and particularly those who disagree with them about culture’s value – in persuasive debate. Data, evidence and knowledge may all help in that task, but so will the ancient tools of politics: rhetoric, argument, emotion and the rest.

Trying to prove cultural value is like hunting the Snark, and while we are occupied with that fruitless search, we fail both to engage effectively in political debate about culture’s value and to learn from the knowledge that our research does produce.

The purpose of cultural research

If we separate debating value from understanding value we enter into more fruitful territory.  Instead of trying to find proofs that can persuade others to change their beliefs – which, insofar as it can be achieved at all, is a political task best undertaken with political means – we can look for knowledge that can influence the actions of those who already hold certain beliefs about the value of culture. In this context, there is practical value in testing individual subjectivity to find the common ground for shared judgements.

The proper purpose of cultural research, I believe, is not persuasion but improvement.  Whether that research is undertaken independently by academics, commissioned by a cultural actor, required by a funding agreement or simply done by a cultural organisation as part of its everyday operations, its purpose should be to increase knowledge and understanding of cultural activity.

How and why

As such, its questions are concerned less with whether than with how and why. Whether something happens is important, of course, especially to funders and in political discussion. How and why it happens are much deeper and more powerful questions because they have the potential to challenge the cultural actor’s own beliefs and assumptions. They have the potential to influence the way art is created, managed and presented, the programmes offered and the ways in which people are able to engage with them, the interpretation and meaning of culture itself.

In my experience, however, it is not only politicians who are uninterested in how and why cultural experiences produce value for people. The cultural profession – with exceptions – is not very interested either, especially if the insights research produces might point to ways in which they could productively change. As far as cultural policy and management goes, it seems all but impossible to get beyond ‘whether’ and the mirage of proving value.

What might be done?

As we all know, the first thing to do when you’re in a hole is to stop digging. And I believe that research and evaluation into public cultural investment is in a deep hole. Let me clarify that I’m concerned here only with what is done within, and required by, the public cultural sector, whoever is doing it. I’m not referring to the growing body of independent, mostly academic research into the theory, policy, work, management or practice of culture.

The principal reason, as I’ve suggested, is that professional engagement with cultural value  is geared towards an unavailable proof, but over the years a host of subsidiary reasons – such as the adoption of inappropriate public management concepts and practices – have also contributed.

When I say, stop digging, I mean let’s take a pause and look around. The Cultural Value Scoping Project is an opportunity to do that, to ask what we are trying to understand and why. Then we might review what we are currently doing and apply a simple test to it: Does the use to which it is put justify the cost of producing it?

If it were possible to back out of the hole we’re currently in, I’d suggest a different approach to monitoring, evaluation and research, which would begin by distinguishing clearly between the three kinds of activity.

Monitoring

The foundation of knowledge about cultural life must be factual and at least where public spending is concerned, there is a great deal that can be known simply through the process of making grants and monitoring the resulting activity. It should be possible, especially now that the grant-making process is computerised, to design a system that can produce reliable data about the nature and extent of activity supported, say, by Arts Council England.

Accountability in grant making should be achieved mainly through monitoring, not evaluation. If the activity was assessed as having public value at application stage, and monitoring shows it to have been completed as expected, then it should follow that its value was delivered in terms of policy and spending decisions.

Refining the application and reporting requirements so that they are factual, consistent over time and used in across the sector, would enable ACE and independent researchers to analyse the resulting data from different perspectives and produce reports that highlight patterns or inconsistencies.

Evaluation

With monitoring processes that are capable both of demonstrating the public value of grants and providing big data that can be investigated for trends, the evaluation process can be freed from the need to justify spending decisions. It can then become a truly open-minded inquiry into how and why results have been produced from which things can be learned and changes to practice made.

In this context, the Arts Council’s recent focus on self-evaluation is very welcome. Evaluation is a critical part of creative work which artists and arts organisations should be doing naturally and in order to improve their work. It will be very difficult to change a culture that sees evaluation principally as a tool of self-justification. But with time, training support and a greater use of monitoring in assessments, some progress could be made.

At the same time, self-evaluation needs to be reported in a form that can be shared internally and with peers. Again, the collection and cataloguing of this material would provide a huge body of data that could be investigated by researchers.

Research

With more reliable data produced by reformed monitoring and evaluation activity, a body such as the Arts Council would be in a position to commission a range of external and independent studies of cultural activity. Some of that might test the self-evaluation process by looking at a representative sample. Some of it might use the data in order to answer specific questions about sub sectors or kinds of practice. This approach would allow the regular publication of good quality thematic research reports that can contribute to debate within the sector and at the Arts Council and thus lead to changes in policy or practice.

In conclusion

This is no more than a sketch of some ways in which research into cultural value might change to become more influential than it has been to date. This may not be the best approach and, to be honest, even if it were I’m doubtful whether it is achievable given the  entanglement of judgements about the value of individuals and of practices with financial decisions. Better research can give us better knowledge but in the end, politics will determine the decisions we make.

Post-script

I’ve been writing about cultural value in one way or another for a long time: some of my previous posts on the subject can be found here, though I’m aware that there’s a certain amount of saying the same things again and again:

6 thoughts on “Some thoughts on monitoring, evaluating and researching culture

  1. Hi Francois,

    Very interesting post – thanks for sharing. But you don’t really deal with the contradictions you describe. You present a well-established philosophical argument that arts are inherently experienced subjectively, and that objective evaluative measures are inherently flawed when attempting to square the circle of individuals’ qualitative evaluations. The recommendation that we should therefore act politically, and use rhetorical and emotive persuasion rather than empirical demonstration seems like a sensible solution to the problem of relative values as you pose it – but the contradictions there are clear – why should anyone who disagrees with your evaluations agree with your arguments? You say we should ‘stop digging and look around’ – which seems like a good idea. But where to look? Your recommendations seem to describe of the status quo of evaluative strategies rather than specifying any particular direction or approach to the problem.

    So given your knowledge of participatory arts practice, what specific methods and approaches to evaluation seem to work? And which strategies might be generalizable enough to work across different contexts with different configurations of participants, materials and resources for evaluation?

    Cheers,

    Saul.

    1. Hi Saul,

      Thanks for your response to this. I appreciate you taking the time, particularly in the context of your own work. First, let me stress that these are notes for a 10 minute intervention, so necessarily constrained. There’s evidently a much bigger piece of work to do here, though I suspect that others will have to do it now.

      You ask why anyone who disagrees with my evaluations should agree with my arguments. Surely that is the essence of democratic debate? It rests on the (admittedly optimistic) belief that we are ready to be persuaded when we engage in it. If we are not persuadable, however confident of our positions we are to start with, the entire democratic project falls. In my experience people do change their minds, and so do I, but evidence (evaluations) is not the main reason. Experience and argument are much more powerful in shifting our positions (and the second is deeply political, though you would be forgiven for not thinking so in the state of contemporary political discourse). People ask for evidence only at the point when they are thinking they might change their mind.

      You are right that my closing proposals are just a sketch of arguments that I have made over many years to bodies like Arts Council England about untangling monitoring, evaluation and research. I’ve had absolutely no success in persuading them of my arguments, despite the experience of everyone who actually has to operate the existing systems. This may not change in my lifetime but I believe it will because existing approaches are burdensome, intrusive and ineffective. They are also poor value for money. Sooner or later, new approaches will have to be put in place.

      As regards participatory art, that is a more precise though very important question. I may be able to say something about it in the book I’m working on about community art, but for now, the best I can do is point you towards this paper, if you don’t already know it.

      François

  2. Part of the challenge is that we in the arts have (if I can put it very provocatively) a vampiric tendency to attach ourselves to other higher status area of public good such as health, education, environment, social and criminal justice. We do this because we believe the arts, our art practice, has the capacity to make a difference, and of course it does. But in doing so we learn the values and systems of these other areas. In health we become obsessed by ‘evidence’ and the gold standard of randomised control trials, little understanding the tortuous constraints on practice that such tests require (as evidenced by the Matisse study into art therapy). What we seem to fail to do in too many cases is to persuade these other sectors of the value of our values – probably as you say because we are so obsessed with ‘whether’ rather than ‘why’ and ‘how’. Part of this would be for instance why in the arts we don’t want to (or value) art which is delivered identically all over the country (where in health we do want for instance cancer screening to be absolutely consistent everywhere). So our understanding of quality in the arts is differently structured from that in health. Even in mainstream theatre where a cast may deliver a production, the audience experience it as unique on the night they attend (if its good). Given the increasing concern with the ‘patient as person’ perhaps we could have something to offer?

    1. Thanks Chris. The difficulty of working with other professions without taking on their concepts is really important. Here, for example is Adrian Jackson from Cardboard Citizens, talking about working in that ambiguous territory:

      ‘Perched between the arts and the social sector, one of our continual tensions has been not adopting wholesale the language and customs of the social sector, while at the same time having great respect for why people in that sector have arrived at certain ways of doing things.’ (https://arestlessart.com/2017/02/06/true-to-the-art-cardboard-citizens/)

      I agree too with your wider point about art and health. The arts have something unique to offer health, or they have nothing that can’t be done better with existing methods. So we need to find ways of articulating that better – and, in my experience, the people in the health sector who are interested in hearing are interested precisely because the arts offer can different – tailored and personal, as you say. I’ve added a link at the bottom of the page to something I wrote for an arts and health website in Ireland that touches on that.

  3. I think you are right about the importance of argument and debate, but what makes this more powerful is its connection to some kind of proof or fact. We can never be absolutely sure of anything, but we can make an argument about its likelihood to be true based on our presentation of some forms of evidence. This is the case in the natural sciences too, with Climate Change being a classic example of our time. The way that scientists can demonstrate a correlation between climate temperature, carbon dioxide and the burning of fossil fuels is fair enough for me because a) the scientists that show it seem to be sensible b) there are lots of them saying the same thing c) the logic of their arguments makes sense and d) it is more or less predictable if the same variables are entered. However, there are still plenty of people who don’t believe it to be true and I can’t say it is definitely true myself, if I am honest, because I am not an expert and I have not conducted any of the experiments myself. I am relying on mediation and my own sense of the way that the scientists present the evidence.

    It’s true that concepts such as the ‘value of culture’ are rather big for proof or evaluation and in its entirety it should be a matter for political debate, but there are some elements which could be said to be component parts (that is a matter of discussion I agree) which you *could* measure and as long as it is clear what you are measuring then you could use that as a part of the process of argument. As you say, the why and the how are important here and maybe that is why this project about cultural value is a good one because it challenges people to state why they believe certain things to be true and how they have come to believe them to be true.

    They are very interesting and timely points you make though and especially worth considering in an age of Trump style politics. So, Trump and Co got drawn into – or started more accurately – a really silly debate about how many people were at the inauguration. It seemed pretty clear and provable that there were more people at Obama’s inauguration than at Trump’s so it was rather a stupid line to keep arguing that it wasn’t true. But what they could have done is debate the terms – why is having more people at the inauguration a ‘good thing’? – they surely missed the obvious points which are to say that Trump’s supporters don’t live near Washington, are not as wealthy, less likely to travel, don’t engage in that sort of thing etc etc (proof actually that Trump’s advisors are pretty stupid and naïve). So rather than arguing about the matter of the numbers they could have made a case for saying that this was why Trump was different.

    1. Thanks Jonathan. You’re right, of course, that evidence is important, and I hope that these notes don’t suggest otherwise. It’s just a matter of what kind of evidence and what we expect it to achieve. I do think that probability – which is a standard concept in health care and outcomes – is the most fruitful direction for better scientific data about the arts, but I haven’t been persuasive enough to get anyone to want to work with me on that. Again, lots of work for others to do there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s