Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

1

Conceptual Challenges of Evaluating the Impacts of a Sizable Technology Programme Portfolio: The

Promotion of Industrial Collective Research (IGF)

Verena Groß, Bernhard Lageman, Michael Rothgang (RWI Essen)

Session C: Portfolio Evaluation

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

2

Content

 A Portfolio Evaluation

B About IGF

C Some Questions Concerning the IGF Program

Design

D Evaluating IGF: the Challenge

E Evaluation Design

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

3

A Portfolio Evaluation

Portfolio evaluation aims at assessing rationality, programme-specific institutional solutions, effects and efficiency of complex programmes

Two central dimensions:– Analyzing the structures, procedures, effects of

different components of the programme– Comparative analysis of the programme with

respect to similar programmes

Methodological solutions?– Evaluation research offers a lot of relevant

quantitative and qualitative solutions– Evaluation studies do not lead always to convincing

results– Portfolio evaluation remains a very challenging task

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

4

B About IGF

Old programme:

• IGF – state-funded collective industrial research: a technology programme introduced in the early 50th of the German Ministry of Economics

• The German Federation of Industrial Research Associations "Otto von Guericke" (AIF) is commissioned with the programme execution

B.1 Old programme - modern features

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

5

B About IGF

Modern features:

• development of industry-wide research networks projects

• should mainly promote SMEs• precompetitive research and results available for every

firm

• few ex ante restrictions on the topics of the research projects

• project ideas are developed bottom up

B.1 Old programme - modern features

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

6

B About IGF

• total budget since the early fifties 2 billion Euro (2005 prices)

• nearly 15,000 collective research projects

• actual annual budget ca. 90 million €, which is to be increased up to 120 million € in 2010

• 600 to 700 new collective research projects per year

• average project cost about 138,000 €, project costs reaching from 50,000 to 350,000 €

• normal project period is two years

B.2 Costs and projects

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

7

B About IGF

• 103 research associations (in 1954 on 17) in most sectors and technology fields that network:

- enterprises (~ 50,000 SMEs) - ~ 700 research institutes

Examples: Association for Cooperative Research for Hot Dip

Galvanizing Research Association of the German Food

Industry Society for Chemical Engineering and

Biotechnology

B.3 Industrial Research Associations, Institutes and Enterprises

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

8

B About IGF

Industrial Research

Associations

EnterprisesResearch Institutes

Phase 2: Discussion of competing research ideas and elaboration of proposals within research asscociations Phase 3: Presentation and official submission of proposals to AIF Phase 4: Evaluation of proposals and funding recommendation to the Ministery Phase 5: Final decision about funding

by the Ministery

B.4 Project Genesis

Phase 1: Identifying research problems and development

of research ideas

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

9

C Two Questions Concerning IGF Programme DesignC.1 Question 1: „Bottom-up“ vs. „Top-down Approach“ (I)

A) The traditional programme “philosophy” and institutional construction

• Bottom-up: the IGF decide themselves about relevant technological fields and research projects according to the principle of self-organization

B) Alternative approach: • Top-down: the “relevant” technological fields could

be defined by expert bodies working on behalf of the Ministry

What is the optimal solution: A or B or a mixture of both?

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

10

C Two Questions Concerning IGF Programme DesignC.2 Question 1: „Bottom-up“ vs. „Top-down Approach“ (II)

Clear strengths of the bottom-up approach

• Good chances for identifying real needs of SMEs in branches where industrial research associations are active

• The university institutes are bound to look for active and passive SME support for their research

• Incentives for promoting transfer activities are immanent to IGF structures

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

11

C Two Questions Concerning IGF Programme DesignC.3 Question 1: „Bottom-up“ vs. „Top-down Approach“ (III)

“Bottom-up” in practice: project portfolio• Sectoral project distribution: large part of project

funding for “old” industries (estimated 40% over the last ten years)

• Strong weight of traditional technologies• Up to the 90th no high-tech-projects, since 2002 high-

tech-projects make about 10 to 15 % of the total budget

“Bottom-up” in practice: institutional structures• The challenge of combining well functioning

institutional structures with flexibility

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

12

C Two Questions Concerning IGF Programme DesignC.4 Question 1: „Bottom-up“ vs. „Top-down Approach“ (IV)

New developments in IGF:

• New research associations were established in the 90s, specialized in high-tech and cross-sectional technological fields

• A portion (presently about 15 %) of financial means for high-tech projects is reserved

• The traditional key of distributing funds to the research associations is going to be abolished

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

13

C Two Questions Concerning IGF Programme Design

The logic behind IGF:• Limited financial, material and personal resource

base of SMEs leads them to “underinvestment” in R&D

• IGF as compensation for “structural weaknesses” of SMEs

– by finding solutions for technical problems which arise in SME production processes

– by inspiring SMEs to engage in R&D activities themselves

– by creating innovation and research networks between SMEs

C.5 Question 2: Target Group SMEs (I)

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

14

C Two Questions Concerning IGF Programme Design

SME participation revisited:• The IGF definition of “SMEs” is rather broad (turnover <

125 Mio. €, < 500 employees) compared to EU definition

• An increasing part of dependent SMEs

• Securing an adequate share of SMEs in IGF bodies and project advisory boards seems to be a problem

• The question whether IGF projects strengthen the innovative capacity of SMEs has to be convincingly answered

• The basic concept is to questioned: What is the target group? - Isolated, single SMEs or SMEs in industrial innovation systems?

C.6 Question 2: Target Group SMEs (II)

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

15

C Two Questions Concerning IGF Programme DesignC.7 Conclusion

Evaluation task:Do the new regulations improve IGF performance by

• strengthening the part of modern highly developed technologies in the project portfolio?

• better enabling SMEs to take actively part in the IGF system?

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

16

D Evaluation IGF: the challenge

D.1 Identifying economic project results

Complications• Project results are by definition (and in practice!) a

public good, that means usable by everybody – Proof of concrete implementation activities of IGF

project results in single firms only is of limited value for evaluation

– No firm which uses IGF results has to render an account of this

– Results become part of the general body of technological knowledge, they may not be identifiable as IGF results by users

• A time lag between closing date of IGF projects and practical uses reaching from 3 to 10 years

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

17

D Evaluation IGF: the challenge

D.2 Identifying the “relevance” of projects for SMEs

Complications• The idea that projects are “relevant” because they

have been carried out under the IGF regulations is to be questioned

• Bottom-up has to be weighted against the factual selectivity of the system

– articulation of SME needs depends on the presence and power of research associations and

– the engagement of research institutes which are specialized in the respective technological field

• Comparing economic relevance of different innovations

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

18

E Our Evaluation Design

E.1 Levels of Analysis and Components

Micro Level: • Investigating genesis, realization and application of

research results on the project levelMeso Level:• Analyzing the activities of IGF actors and bodies:

– industrial research associations– the role of AiF and its central bodies– research institutes which participate in IGF projects– IGF projects in the framework of sectoral innovation systems

Macro Level:• Comparing IGF with other technology programmes

oriented towards SMEs• Analysing the IGF project portfolio in the framework of the

new and old programme design

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

19

E Our Evaluation Design

E.2 Instruments (I)

Interviews: • Expert interviews with:

– Representatives of (all) industrial research associations (over 4 years)

– Persons in charge of single projects, selected by chance– Independent experts evaluating project proposals and results– Representatives of central IGF bodies

Surveys:• Different enterprise surveys:

– General survey of industrial firms with respect to their knowledge on and estimation of IGF

– Project oriented surveys with respect to estimated relevance of the projects and the use of project results

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

20

E Our Evaluation Design

E.3 Instruments (II)

Case studies of the role of IGF in sectoral innovation systems: • The selected sectors represent:

– traditional (but developed) technologies like food and furniture industries

– far developed technologies but well established industrial products (like mechanical engineering industries and the automotive sector)

– high-technology fields or high-end cross-sectoral technologies

Quantitative analysis:• Econometric Analysis of IGF programme data and the results of

our surveys• Comparing the participation of SMEs in different technology

programmes, as far as relevant data are accessible

Rhe

inis

ch W

estf

älis

ches

Ins

titut

für

Wirt

scha

ftsf

orsc

hung

21

Thank you for your attention!