Innovation in research provides benefits to society and the economy… but how do we measure it?

Scientists at UK universities are among the world’s most innovative. In the 2017 Global Innovation Index, the UK ranked second globally for the quality of its university research and development, and sixth globally for university business interactions. And it is this innovation that is helping to develop new, world-changing technologies.

At The Institute of Cancer Research, we are passionate believers in the importance of innovation – designing and developing novel techniques and cancer treatments that work in completely new ways. It is only by being innovative that we will address the major areas of unmet need in current cancer diagnosis and treatment.

This innovative approach is behind our success in the discovery and development of so many new types of cancer treatment, as well as the advances in cancer biology that have underpinned many more – including the fundamental discovery that it is DNA damage that causes cancer, the identification of the link between lung cancer and cigarette smoking, and the discovery of our world beating prostate cancer drug, abiraterone. 

It’s also why we are often at or near the top of international tables that rank academic innovation – but how useful are these tables? 

Measuring innovation

Those of us who work in knowledge exchange and commercialisation – bringing together researchers, companies, the public and patients to develop academic knowledge into societal benefit – agree that it is important to be innovative.

And if it is important to be innovative then it must also be important for institutions to be able to measure their level of innovation to ensure they are doing a good job – but that’s easier said than done. Innovation is very hard to measure.

One measure that we may look at is how a university works with industry. Interaction with industry is an important step in translating research discoveries into those world-changing products and services. So it would be useful to know how effectively our scientists work with commercial partners.

But the measures used routinely to measure university industry interaction all seem to be flawed.

One popular way of scoring innovation, for example, is to count the number of patents filed by or granted to a university and its partners. Some of the most influential international university rankings, including the Reuters 100 Worldwide and European lists and U-Multirank, measure patents applied for as part of their methodology.

But counting patents is fundamentally flawed as a measure of innovation.

Filing patents is relatively easy, can be done for projects that are not particularly innovative, does not necessarily signal that an invention is useful and can be a drain on finances.

And at the ICR, we sometimes deliberately avoid filing patents, because we think patient benefit would be better served by publishing data or disseminating it to practitioners so that it is available for all to use.

This flexibility in our approach to patenting is a fundamentally important part of our business engagement strategy. It helps us to ensure that the new treatments that come from our discoveries are delivered to doctors and patients as quickly as possible.

Patent citations

Perhaps analysing the number of papers from a research institute that are cited in patent applications – a measure in which the ICR has been ranked joint top in the world – could be more useful than looking at raw numbers.

But this too has its flaws. High scores on this measure could be down to the same paper being cited many times, or through old papers being cited rather than measuring current innovation.

Could other types of commercial interaction help to measure innovation?

Agreements with companies

At the ICR, we monitor a range of different interactions with companies. In the last academic year, we entered into several hundred agreements with companies – everything from complex research collaborations and clinical trial agreements to material transfer agreements (MTAs) and confidential disclosure agreements (CDAs).

The difficulty in using the numbers of these types of agreements as a measure of innovation comes in part from their variety.

Some of the agreements will undoubtedly lead to industry partners developing new products and treatments that reach the clinic, and generate income for the ICR that will be invested back into our research.

But knowing the number of agreements or the number of different companies with which agreements were concluded does not show how meaningful these interactions were.

Discussions under CDA do not necessarily lead to further interaction; programmes for which CDAs are needed do not necessarily lead to new treatments or technologies. And MTAs could entail the sharing of lots of different material, or just sending out the same material to lots of people.

Similarly clinical trial agreements between academic, hospital and industry partners can involve innovative drugs.

But they may also involve testing existing drugs in slightly different patient populations and even when there are novel drugs involved, the innovative research may have taken place some years before a trial begins, and may involve organisations other than the ones running the trial.

Qualitative measures

Could more qualitative measures be more effective in scoring innovation?

The most thorough and widely-respected measure of research quality in UK academia is the Research Excellence Framework (REF) – which includes a measure of impact through evaluating case studies submitted by research institutes.

The ICR has topped the last two REFs outright, and also came out top in impact in 2014, when impact evaluation was first added to the scoring.

Effective, qualitative measures are powerful – but a problem here is that they are difficult, requiring someone to read them all and subjectively rank them.

They are also difficult to compare between institutions. How do you compare research that leads to innovative new cancer treatments with aeronautical engineering that makes Formula 1 cars faster?

Our new quarterly newsletter, Connections, showcases our work and commercial partnerships, shares learning, and identifies ways for staff, students and industry partners to get involved.

Sign up

Income from collaboration with industry

Income from collaboration with industry or from royalties paid on successful products is also sometimes used as a proxy for innovation – the idea being that the more successful an industry partnership or product, the more income will be generated.

The ICR generated the most invention income of any UK academic institution in the past academic year – including much larger Oxbridge and the Russell Group universities – and has ranked first on size-adjusted income for several years.

But income received from inventions generally measures what a research institute was doing several years ago, and not what it is doing now. It’s also a function of revenue-sharing negotiations made at a time when the future commercial potential of a research project would have been very difficult to predict.

It is influenced by the area of research – science that aligns with more profitable industries, such as the pharmaceutical industry, will tend to generate more income. And it may not measure sustained innovation but rather a single blockbuster product.

It can also be influenced by the efforts of others to get something to the market.

Benefit, not profit

And again – both at the ICR and generally across academia – first and foremost, we do not make decisions based on how much income we expect to generate from our discoveries. All of our commercial decisions and agreements are made with a view to turning our research findings into patient benefit as quickly as possible.

That can mean publishing data when it would be more lucrative to keep it in the private domain.

It can mean choosing an industry partner based on whether we believe they have the skills and expertise to bring a product to market, not on how much cash they offer. It can mean deliberately restricting the amount of income in some circumstances to ensure that a treatment is available at an affordable price.

And it means actively avoiding commercial agreements that, while potentially lucrative, could slow down the translation of discoveries into new cancer drugs or other treatments.

So how can we measure innovation?

Perhaps the best way of measuring innovation is to use a mixture of all of these different criteria. At the ICR, we believe that the fact we come high up in a range of different league tables and ranking systems is a good indication of how innovative we are.

But no-one has found a perfect system yet. And the best academic science will always be carried out with the main priority being not profit, but a determination to make discoveries that lead to completely new understandings of the world, and which could benefit society in new and sometimes unexpected ways.

comments powered by Disqus