Step KTA 7.F

Evaluate outcomes. May have to develop new outcome measures.

Primary Findings

Secondary Findings

Primary findings

Barriers

Universities commercialize relatively few inventions especially in cases where there is a lack of adequately trained staff or technology transfer services.
Case-based research
(View full citation)

Outcomes from academic research are subtle, widespread and therefore hard to quantify. It is difficult to measure academic contributions to industry.
Case-based research
(View full citation)

In a complex environment, determining precisely why a particular knowledge translation process failed may be challenging. Applying a systems approach may help to elucidate contributing factors.
Application of Graham’s Knowledge-to-Action Process model in occupational therapy.
(View full citation)

Carriers

A clear understanding of the diffusion process can deepen and broaden the scope of an evaluation and help to reveal factors that contributed to the adoption of an innovation (intervention).
Literature review and synthesis.
(View full citation)

Measures

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: How are the actors in the social system interconnected?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: In what ways do the appliers of the new knowledge differ from the resistors (or non-appliers)?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Were potential knowledge users given enough time to process their decision to apply the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: What contextual factors contributed to application of the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: What were the attributes of the process that facilitated or impeded application of the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Which communication channels were most effective at different times in the process, or with different categories of potential knowledge users?
Literature review and synthesis.
(View full citation)

When evaluating a knowledge translation process, one of the questions an evaluator can asks is: Who are the opinion leaders among the intended audience, and what role do they play in applying the new knowledge?
Literature review and synthesis.
(View full citation)

When evaluating a research project for knowledge translation success, one useful researcher-stakeholder knowledge application (passive or active) indicator would be, knowledge being used to help create and support interventions.
Literature review.
(View full citation)

When evaluating a research project for knowledge translation success, useful researcher-stakeholder interaction indicators would include showing evidence of: communication channels, processes, and contexts between knowledge translation actors; working relationships among stakeholders; shared vocabulary among stakeholders; knowledge being relevant to and understood by the target audience; a linking or brokerage role being taken among stakeholders; and members of the target audience being engaged as co-researchers.
Literature review.
(View full citation)

When evaluating a research project for knowledge translation success, useful researcher-stakeholder knowledge application (passive or active) indicators would be: knowledge being used to inform decision making, in relation to individuals or in relation to policy and practice within systems, institutions, and regions; research products being used to inform policy or agenda; changes in behavior, awareness, communication, or interaction patterns evident among stakeholders.
Literature review.
(View full citation)

When evaluating the success of a knowledge-for-action process, one of the factors that should be considered is the fidelity or integrity of the knowledge at its point of application (in contrast to its original research-based form).
Literature review and application of knowledge processes to evaluation.
(View full citation)

When evaluating the success of a knowledge-for-action process, one of the factors that should be considered is the reasonableness of the time it takes for the knowledge to be acted upon (or applied) by stakeholders.
Literature review and application of knowledge processes to evaluation.
(View full citation)

When using the Knowledge to Action Process model to effect an evidence-based change (e.g., in a clinical practice), it is important to monitor how the knowledge is being used, and whether the desired practice changes are being adopted. One way to assess this is to re-administer the survey that was used in the problem identification phase.
Applying the Graham Knowledge to Action Process model in stroke rehabilitation.
(View full citation)

Methods

Comparing patents after a partnership is a means of retroactively measuring cognitive dissonance or proximity among partners.
Case study analysis
(View full citation)

Tips

One of the factors that is associated with productive knowledge management (knowledge translation) is that performance assessment is linked with innovation. Knowledge performance is assessed based upon the degree to which the new knowledge contributes to the delivery of the expected outputs and outcomes( e.g., value to the knowledge user, best practice, value for money, etc.).
Literature review and conceptual framework development.
(View full citation)

Secondary findings

Carriers

Evaluators can use diffusion theory to guide the formulation of research questions.
Source: Guba (1967); Bozeman (1988). In: Ashley, S.R. (2009)

Participation of research project-based policy entrepreneurs in public policy networks can strengthen the linkages and exchanges between researchers and policy makers.They provide awareness of research and initiatives and facilitate social learning.
Source: Reinicke (1999, 2000); Kingdon (1984). In: Williams, A., Holden, B., Krebs, P., Muhajarine, N., Waygood, K.,Randall, J. & Spence, C. (2008)

Measures

Evaluators can use diffusion theory to guide the design of measurement instruments.
Source: Guba (1967); Bozeman (1988). In: Ashley, S.R. (2009)

Tips

When designing a knowledge translation communication strategy, researchers should consider how they will monitor progress, obtain feedback, and evaluate success.
Source: Yuan (2010); CRD (1994, 2009); Harmsworth (2001); Herie (2002); Lavis (2003); Canadian Health Services Research Foundation (2004); European Commission (2004); Carpenter (2005); Zarinpoush (2007). In: Wilson, P.M., Petticrew, M., Calnan, M. W. & Nazareth, I. (2010)