Step KTA 3.F

Evaluate outcomes — may have to develop new outcome measures.

Primary findings

Secondary findings

Primary findings

Barriers

The amount of social science and humanities groups participating in knowledge transfer is proportionate to engineering and the physical sciences; however the societal impact of knowledge produced by the arts, humanities and social science academics is more difficult to quantify.
Non-experimental study
(View full citation)

Adopting new technologies is presumed to be a logical decision; however individual users determine for themselves the costs and benefits of investing in new tech, including their own natural resistance to change. If the return on investment doesn’t support the change, not adopting new tech is the rational choice for the user.
Case study findings
(View full citation)

Formal coordination measures can hinder innovation. For example, if headquarters dictates goals to subsidiaries with higher technological expertise.  
Case study findings
(View full citation)

In a complex environment, determining precisely why a particular knowledge translation process failed may be challenging. Applying a systems approach may help to elucidate contributing factors.
Application of Graham’s Knowledge-to-Action Process model in occupational therapy.
(View full citation)

Carriers

Publically acknowledging collaborative efforts motivates ambitious researchers to embrace collaboration.
Non-experimental study 
(View full citation

Including collaborative efforts among outcome tracking as part of a greater accountability system motivates researchers to work together. 
Non-experimental study 
(View full citation)

Social sciences and humanities contribute to society by promoting self-reflection along with critical and conceptual thinking – activities that sustain ideas and legitimize societal needs.
Non-experimental study
(View full citation)

Examining economic, cultural, policy, and innovative progress works to quantify research collaborations that are not commercial or technology based.
Non-experimental study
(View full citation)

Universities are embedded in the regional innovation system as influencers promoting technological innovation and economic development in their regions. 
Non-experimental study
(View full citation)

Innovations have three components: an innovative opportunity  consists of the following three components:(a)an economic value for someone; (b) mobilization of resources; and (c)the ability to appropriate at least some part of the economic value of the innovation, by the actor pursuing the opportunity.
Case study findings
(View full citation)

A clear understanding of the diffusion process can deepen and broaden the scope of an evaluation and help to reveal factors that contributed to the adoption of an innovation (intervention).
Literature review and synthesis.
(View full citation)

Methods

Action Cycle within the KTA Model — the sixth step is to determine the impact on the target system that results from knowledge use. It is the only way to determine whether the efforts to promote knowledge use were successful and worthwhile.
Summary of the Knowledge To Action Model and its application to knowledge translation.
(View full citation)

Identify a problem that needs addressing; Identify the need for change; Identify change agents (i.e., the appropriate actors to bring about the change); Identify target audience; Assess barriers to using the knowledge; Review evidence and literature or develop or adapt innovation; Select and tailor interventions to promote the use of the knowledge; Link to appropriate individuals or groups who have vested interests in the project; Implement; Evaluate; Develop a plan to evaluate use of the knowledge; Pilot test; Evaluate process; Evaluate outcomes; Maintain change or sustain ongoing knowledge use.
Results from a knowledge synthesis
(View full citation)

Measures

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: How are the actors in the social system interconnected?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: In what ways do the appliers of the new knowledge differ from the resistors (or non-appliers)?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Were potential knowledge users given enough time to process their decision to apply the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: What contextual factors contributed to application of the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: What were the attributes of the process that facilitated or impeded application of the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Which communication channels were most effective at different times in the process, or with different categories of potential knowledge users?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Who are the opinion leaders among the intended audience, and what role do they play in applying the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a research project for knowledge translation success, one useful researcher-stakeholder knowledge application (passive or active) indicator would be, knowledge being used to help create and support interventions.
Literature review.
(View full citation)

When evaluating a research project for knowledge translation success, useful researcher-stakeholder interaction indicators would include showing evidence of: communication channels, processes, and contexts between knowledge translation actors; working relationships among stakeholders; shared vocabulary among stakeholders; knowledge being relevant to and understood by the target audience; a linking or brokerage role being taken among stakeholders; and members of the target audience being engaged as co-researchers.
Literature review.
(View full citation)

When evaluating a research project for knowledge translation success, useful researcher-stakeholder knowledge application (passive or active) indicators would be: knowledge being used to inform decision making, in relation to individuals or in relation to policy and practice within systems, institutions, and regions; research products being used to inform policy or agenda; changes in behavior, awareness, communication, or interaction patterns evident among stakeholders.
Literature review.
(View full citation)

When evaluating the success of a knowledge-for-action process, one of the factors that should be considered is the fidelity or integrity of the knowledge at its point of application (in contrast to its original research-based form).
Literature review and application of knowledge processes to evaluation.
(View full citation)

When using the Knowledge to Action Process model to effect an evidence-based change (e.g., in a clinical practice), it is important to monitor how the knowledge is being used, and whether the desired practice changes are being adopted. One way to assess this is to re-administer the survey that was used in the problem identification phase.
Applying the Graham Knowledge to Action Process model in stroke rehabilitation.
(View full citation)

Tips

One of the factors that is associated with productive knowledge management (knowledge translation) is that performance assessment is linked with innovation. Knowledge performance is assessed based upon the degree to which the new knowledge contributes to the delivery of the expected outputs and outcomes( e.g., value to the knowledge user, best practice, value for money, etc.).
Literature review and conceptual framework development.
(View full citation)

Engage in practices such as, formal audits, newsletters, reports or seminars, which help retain knowledge developed from prior new product development projects.
The hypothesis of companies promoting knowledge retention will have higher levels of intertemporal integration was found to be true. (B = .27, p < .05)
(View full citation)

Sort, categorize and give meaning to knowledge in the company's current strategic context.
The hypothesis that companies that engage in activities that enable the interpretation of knowledge in it's current strategic context will have higher levels of intertemporal integration was found to be true. (B = .40, p < .001)
(View full citation)

Secondary findings

Carriers

Evaluators can use diffusion theory to guide the formulation of research questions.
Source: Guba (1967); Bozeman (1988). In: Ashley, S.R. (2009)

Ottawa Model of Research Use (OMRU) — There are six essential elements in the design of a strategy to introduce research findings into practice: 1) the practice environment; 2) the potential adopters of the evidence; 3) the evidence-based innovation; 4) research transfer strategies; 5) the evidence adoption; 6) health-related and other outcomes. The systematic assessment, monitoring and evaluation of the state of each of the six elements is required before, during and after any efforts to transfer the research findings.
Source: Logan, J et al (1999). In: Estabrooks, C.; Thompson, D.S., Lovely, J.J.E., & Hofmeyer, A. (2006)

Participation of research project-based policy entrepreneurs in public policy networks can strengthen the linkages and exchanges between researchers and policy makers.They provide awareness of research and initiatives and facilitate social learning.
Source: Reinicke (1999, 2000); Kingdon (1984). In: Williams, A., Holden, B., Krebs, P., Muhajarine, N., Waygood, K.,Randall, J. & Spence, C. (2008)

Measures

Evaluators can use diffusion theory to guide the design of measurement instruments.
Source: Guba (1967); Bozeman (1988). In: Ashley, S.R. (2009)

Tips

Effective, accurate and timely knowledge management which involves the recording, retrieving, and reviewing of detailed information on past product development projects will affect performance of new product development projects.
Source: Lynn, Reily, and Akgun (2000); Lynn, Simpson, and Souder (1997); Lynn, Skov, and Abel (1999). In: Sherman, J. D., Berkowitz, D., & Souder, W. E. (2005)

When designing a knowledge translation communication strategy, researchers should consider how they will monitor progress, obtain feedback, and evaluate success.
Source: Yuan (2010); CRD (1994, 2009); Harmsworth (2001); Herie (2002); Lavis (2003); Canadian Health Services Research Foundation (2004); European Commission (2004); Carpenter (2005); Zarinpoush (2007). In: Wilson, P.M., Petticrew, M., Calnan, M. W. & Nazareth, I. (2010)