Strengthening Workforce Development Programs: Evaluation of the ReTool Erie Initiative

September 16th, 2010 | General »

This research is a comprehensive process and outcome evaluation of ReTool Erie, a program partially funded through the American Recovery and Reinvestment Act (ARRA).

Researchers at KeyStone Research Corporation (KSRC), Joyce A Miller, Ph.D. and Tania Bogatova, MBA, ABD, are pleased to announce they will serve as external evaluators for the ReTool Erie initiative. This research is a comprehensive process and outcome evaluation of ReTool Erie, a program partially funded through the American Recovery and Reinvestment Act (ARRA). ReTool Erie (http://www.retoolerie.org), a partnership of The Erie Community Foundation (http://www.eriecommunityfoundation.org), The Nonprofit Partnership (http://www.thenonprofitpartnership.org), and the Regional Center for Workforce Excellence (http://www.nwpawib.org) is a new project aimed at strengthening local nonprofits focused on workforce development. ReTool Erie provides training, one-on-one technical assistance, and competitive capacity building grants to help local nonprofits successfully walk the path to transformation. ReTool Erie is set up to ignite that transformation.

KSRC’s responsibilities include these components of the overall evaluation:

  • Evaluation of Technical Assistance and Contracted Services (including partner organization satisfaction with the TA and contracted services and their impact of on partner organization outcomes)
  • Assessment of Leadership Development
  • Evaluation of Workforce Development Program Workshop Series

The evaluative data will have a number of implications for policy and practice. The leadership development and technical assistance impact data assess the achievement of outcomes for the partner organizations. The results of the data analysis will show if the initiative’s goals were achieved, which will provide evidence (either positive or negative) as to the effectiveness of the capacity building efforts for organizations offering workforce development programs.

Also, the results of the data analysis on satisfaction with the technical assistance provided through this initiative will provide additional evidence that can be used to explain the outcomes achieved and point to areas of the initiative that can be improved. The following description for each part identifies an overarching (general) research question as well as the corresponding specific research question(s) that guide the collection and analysis of evaluative data.

Evaluation of Technical Assistance and Contracted Services

  1. Satisfaction with Technical Assistance and Contracted Services

General Question: How satisfied are partner organizations with the one-to-one technical assistance (TA) they received from their ReTool Erie Project Officer and any of the contracted consultants they used per work plans for their grant awards?

Specific Questions:

1. What is the level of satisfaction of partner organizations with the one-to-one technical assistance and the contracted services with respect to the following areas?
  • Overall experience
  • Overall usefulness of the one-on-one technical assistance and/or contracted services
  • Usefulness of specific components of the technical assistance and/or contracted services (e.g., phone calls, materials, face-to-face, referrals to training, referrals to community resources, and email/correspondence, etc.
  • ReTool Erie project officer and/or contracted consultant’s performance with respect to:
  • Expertise in area of service provided
  • Attentiveness to the organizational needs
  • Ability to communicate clearly with the personnel
  • Appropriateness of the technical assistance with respect to the organizational needs
  • Accessibility of the consultant to partner organization
  • Timeliness in the delivery of technical assistance
  • Quality of the written materials
  • Ability to establish a positive working relationship based on trust and honesty
  • Building your capacity to apply the knowledge and skill within your organization
2. What are the strengths of the Project Officer and/or contracted consultant? 3. In what ways can the Project Officer and/or contracted consultant improve?

Indicators/Instrumentation

Technical Assistance Assessment Tool (for ReTool Erie’s project officers) and Contracted Consultants Assessment Tool developed by KSRC.

Methodology

The Technical Assistance Assessment Tool and the Contracted Consultants Assessment Tool will be sent to the partner organization to complete at the end of the technical assistance and consultant services received, either via mail or through a web-based survey.

  1. Impact of Technical Assistance and Contracted Services on Partner Organizations’ Outcomes

General: Is the one-on-one technical assistance correlated with partner organizations’ outcomes?

Specific:

  1. What changes have the partner organizations made as a result of their work with a ReTool Project Officer and/or consultant?
  2. Has there been a significant improvement within the partner organizations that received technical assistance and contracted services with respect to the following areas?
  • Amount of revenue
  • Diversity of revenue sources
  • Number of individuals served and the service outcomes attained (success rate as indicated in the Performance Progress Report of SCF)
  • Number and type of organizational partnerships and collaborations
  • Leadership outcomes (see Part IV of evaluation)
  • Evaluation of effectiveness

Indicators/Instrumentation

ReTool Erie Project Officers will collect the following pre and post administrative data (prior to when the organization begins the work associated with ReTool Erie and once they complete the project): amount of revenue received, number and type of revenue sources, number of individuals served and the service outcomes achieved, number and type of partnerships and collaborations. The Leadership Practices Inventory (LPI) will be used to assess leadership outcomes (per Part IV of this evaluation plan)

Methodology

The administrative data gathered by ReTool Erie will be exported to a data file for KSRC to analyze and prepare a report at the end of the project (October 2011.).

  1. Improvement of Knowledge, Skill, and Ability Regarding Best Practices for Workforce Development Programs

General: Has the level of knowledge, skill, and ability of partner organizations related to implementing best practices in workforce development programs improved as a result of workshop training delivered through the ReTool Erie initiative?

Specific:

1. To what extent have the partner organizations’ ReTool Erie Teams increased their knowledge and understanding of best practices in workforce development programs?

2. To what extent have the partner organizations’ ReTool Erie Teams applied their learning to the delivery of services to program participants according to best practices in workforce development programs?

Indicators/Instrumentation

Best Practices for Workforce Development Programs Assessment Tool will be developed for each individual workshop in the ReTool Erie training series. KSRC researchers will develop these individual assessment tools based on the content and learning objectives of workshop series.

Methodology

The Best Practices for Workforce Development Programs Assessment Tool will be completed, both pre and post, by the participants of the training services on workforce development. The tool will be administered prior to each workshop (through the ReTool Erie office, either via a web-based survey or by other means) and at the conclusion of a workshop, administered by the workshop trainer. The KSRC researchers will analyze these data and prepare a report after all organizations have completed the workforce development training series, due around October 2011.

Assessment of Leadership Development

General: Has there been an improvement in leadership knowledge and ability as a result of participating in the Strengthening Erie’s Workforce Initiative?

Specific:

  1. To what extent has the executive leadership shown improvements in the areas of:
  • Inspiring a shared vision
  • Enabling others to act
  • Challenging the process
  • Modeling the way
  • Encouraging the heart
  • Building relationships and partnerships

Indicators/Instrumentation

Leadership Practices Inventory (LPI) (Kouzes and Posner, 2004) will be used to assess the first five areas of executive leadership. The Assessment of Organizational Functioning instrument (Community Collaborations, part I, questions 43-45) will be used to assess the area of “building relationships and partnerships.”

Methodology

The KSRC researchers will administer a pre and post LPI via a mailed survey. This tool is a 360-assessment, which will include ratings from the following: self (leader being assessed); 7-10 other observers (if appropriate for the organization) capable of rating the leader, which may include but not be limited to: organizational manager(s), direct report(s), co-worker(s), and others, such as board members and/or individuals in partnering organizations. The Assessment of Organizational Functioning will be administered by TNP/ECF Project Officers. The KSRC researchers will analyze these data and prepare an aggregate report across all leaders assessed at the end of year two.

Reference

Kouzes, J.M. and Posner, B.Z. (2004). Leadership Practices Inventory. San Francisco, CA: Pfeiffer


Share |