Step 9.3

Review performance against expectations.

Primary Findings

Secondary Findings

Primary findings

Barriers

Impact studies often rely on outcomes generated during the grant reporting period although the actual benefits cannot fully manifest in that short a time.  
Case study findings
(View full citation)

Devices used in clinical care are particularly hard to improve incrementally based on field results because processes must be standardized, as must training. Variations in training and implementation skew user feedback. 
Postmarket questionnaire results
(View full citation)

Outcomes of R&D partnerships are complicated and often produce indirect, hard to measure effects.
Project data analysis
(View full citation)

Results from product platforming efforts are proven to last throughout the product life cycle in comparison to non-platformed products.
Case study analysis
(View full citation)

Carriers

Platformed products perform better in the market.
Case study analysis
(View full citation)

Healthcare providers including surgical staff (advanced users) are a group capable of making significant contributions as co-developers or innovators. 
Postmarket questionnaire results
(View full citation)

Methods

Comparing patents after a partnership is a means of retroactively measuring cognitive dissonance or proximity among partners.
Case study analysis
(View full citation)

Level 1 — Single and Double-loop learning or as we have termed, Non-Duetero, is characterized as passive, routine based systems learning that supports current behavior, limits feedback, and reinforces a short-term orientation. Level 2 — Duetero learning is characterized as active, holistic learning, where continuous feedback is sought, and the process reinforces a long-term orientation.
Authors secondary research.
(View full citation)

Must be aware that examining product effectiveness as compared to productivity outcomes or marketplace performance will correlate higher with NPD success and cross functional integration
Meta-analysis. β = .185, p < .01
(View full citation)

Utilize novel estimation procedure for testing order entry models. Procedure is based on the generalized method of moments.
Survey. This model was developed and applied by the study's authors to overcome the limitations of regression-based methods.
(View full citation)

When measuring the success of NPD, use objective measures, such as sales or profits, instead of subjective measures
Meta-analysis. β = -.381, p < .01 (see Table 2 –H9)
(View full citation)

Measures

Assessment to identify the current and future new product development performance of a firm.
Survey.
(View full citation)

New product success — Dimensions include degree to which a new product’s profits exceeded or fell short of expectations; degree to which sales exceeded or fell short of expectations; degree to which a new product provided an entry point into an existing market for a new line of products; degree to which a new product created a market that was new to the industry.
Survey. Reported in Tables 1, 3 and 4 in the text; (see supporting literature in the next section).
(View full citation)

To measure the performance of individual business units, ten performance metrics were boiled down into two measures of performance: profitability of the business's total new product efforts; and impact of the total new product effort on the business.
Survey of 161 business units.
(View full citation)

Use the following metrics to measure the performance of new product development projects; earned revenue versus forecasted revenue, operating profits,customer satisfaction, and time to market.
Authors' research experience. Between 65% — 80% of companies use these metrics to measure performance.
(View full citation)

Tips

Engage in practices such as, formal audits, newsletters, reports or seminars, which help retain knowledge developed from prior new product development projects.
The hypothesis of companies promoting knowledge retention will have higher levels of intertemporal integration was found to be true. (B = .27, p < .05)
(View full citation)

Many factors can influence the utility of post project reviews. For example, if they are held too late there is a risk that key learning points will have been forgotten. Generally they should be held 6 months after launch. Similarly, the way the discussions are moderated will influence the generation and sharing of knowledge, an outside moderator is recommended. Any knowledge that the NPD generates should ideally be widely shared within the organization.
Case Study results.
(View full citation)

Results indicate that the changes and problems that occurred during the execution of the project contribute much more to learning than meticulous planning, which is based on past knowledge and proven solutions that have been devised to meet previously defined needs.
Survey of 475 research and development projects
(View full citation)

Sort, categorize and give meaning to knowledge in the company's current strategic context.
The hypothesis that companies that engage in activities that enable the interpretation of knowledge in it's current strategic context will have higher levels of intertemporal integration was found to be true. (B = .40, p < .001)
(View full citation)

The results indicate that metaphors and stories form an important part of post project review discussions and are used — consciously or subconsciously — to stimulate or to summarize key points. They also appear to enable the transfer of knowledge on complex technological points by making the discussions understandable to those without detailed knowledge.
Case Study — Researcher's analysis of interviews, documents and observations.
(View full citation)

Secondary findings

Barriers

Superstitious learning occurs when positive results are interpreted as learning outcomes in spite of little or no association. Success learning involves expectations or assumptions that what worked best in the past will work for the future. Competency traps occur when the organization refuses to adopt superior technology despite its availability. These negative impacts are avoidable if organizations implement learning within a transparent framework where employees are emotionally intelligent enough to recognize, avoid and manage such pitfalls
Source: Scott-Ladd, Chan (2004). In: Liepe, Z. & Sakalas, A. (2008)

Carriers

The existence of an overarching relationship (historical or longstanding) between a researcher and relevant decision makers (knowledge users), where research utilization is only one activity in the larger, ongoing relationship, can be important for the utilization of research findings.
Source: Goering (2003); Mohrman (2001); Canadian Health Services Research Foundation (2002); Golden-Biddle (2003). In: Ginsburg, L.R., Lewis, S., Zackheim, L. & Casebeer, A. (2207)

Methods

A valuable way to capture knowledge generated during the course of a new product development (NPD) project is to hold a post project review (PPR). This is ‘‘a formal review of the project which examines the lessons which may be learned and used to the benefit of future projects’’ (Lane, 2000).
Source: Lane (2000). In: Koners, U., & Goffin, K. (2007)

An approach from psychology — repertory grid technique, appears to access tacit knowledge. The particular advantage of repertory grid technique is that it forces the respondent to think deeply and probes their tacit knowledge.
Source: Goffin (2002); Reed (2000). In: Koners, U., & Goffin, K. (2007)

Deutero learning is the most advanced style of learning and seeks to gain insight into the "learning process" itself, all while addressing the problems or opportunities at hand. This learning style also embraces an atmosphere where failures are fully tolerated, if not encouraged.
Source: Allee (1997), Argyris & Schon (1996), Bateson (1973) & McKee (1992). In: Liepe, Z. & Sakalas, A. (2008)

Measures

Operational effectiveness and strategic flexibility can be used to measure new product development performance.
Source: Brown & Eisenhardt, 1995. In: Bernasco, W., Weerd-Nederhof, P.C., Tillema, H. & Boer, H. (1999)

To assess the performance of product design, the following measures are often used: quality, time, competency, and costs directly related to profit.
Source: Ulrich & Eppinger, 2000; Wheelwright & Clark, 1992. In: Meybodi, M.Z. (2003)

Tips

By definition, tacit knowledge cannot be clearly expressed, documented, or observed and so it is necessary to use special data collection and analysis techniques. Metaphors and stories have been recognized in the literature as indicators of the generation and exchange of tacit knowledge.
Source: Cook and Brown (1999); Nonaka (1994). In: Koners, U., & Goffin, K. (2007)

Harter et al. (2000) argue that NPD quality, lead time and cost can all be simultaneously improved by reducing defects, waste, and rework in NPD project activities. They provide empirical evidence indicating that process maturity may contribute to greater project performance.
Source: Harter et al. (2000). In: Swink, M., Talluri, S., & Pandejpong, T. (2006)

One of the major reasons that concurrent engineering has gained broad acceptance is its impact on time. CE will get the manufacturing and marketing departments involved early in the development process. This will enable these departments to influence the design and obtain a more cost effective and high quality products. CE takes advantage of the latest advances in computer and information technologies to develop libraries with comprehensive accumulations of product and process designs. Therefore, knowledge gained during the development of one product is captured and then passed on to subsequent product developments as templates.
Source: Salomone (1995), Kamrani and Salhieh (2002). In: Kamrani, A., & Vijayan, A. (2006)