Introduction
When to Use This Handbook
Project Evaluation is Important to the Library of Michigan
Project Evaluation is Important to Your Library
The Stakeholder Approach to Project Evaluation
SEVEN STEPS of Project Evaluation
Wrapping Up
Appendices A, C, D, E, & F [PDF 65K]
Appendix B [PDF 33K]

Introduction

This handbook was developed in 1994 by the Library of Michigan with the encouragement of the U.S. Department of Education. The handbook was intended to be a practical, easy to use, and relevant tool of library staff to evaluate needs. It is being updated to reflect the change in federal legislation from Library Services and Construction Act, to Library Services and Technology Act (LSTA) P.L. 104-208, enacted on September 30, 1996. LSTA is administered by the Institute of Museum and Library Services.

The handbook will guide you in planning and conducting evaluations of your subgrant programs. Following comments on the use of the handbook and the stakeholder approach to evaluation, a step-by-step method is used to present the information. Seven steps for project evaluation take the reader from start to finish. These steps are explained in detail later in this publication

Top of Page

When to Use This Handbook

It is important to understand the context in which this handbook is meant to be used. Evaluation is only one component of project planning and the LSTA application process. This handbook covers only the development of your project evaluation, it is not meant to be an overall guide to preparing an LSTA subgrant application. Prior to applying the seven-step method presented here, you should do all of the following:

1. Recognize unmet community needs which library services and/or programs may meet;
2. Read all documents in the current LSTA application package;
3. Match the needs you have identified with LSTA funding areas for potential project development; and
4. Attend a Library of Michigan subgrant writing workshop to improve your overall understanding of the application form and guidelines.

In order to receive the most benefit from this handbook, we suggest that you have a project in mind, complete with documentation of the needs it will address, as well as the project goal and objectives. At this point you are ready to apply the stakeholder approach as you develop your project's plan for evaluation.

Top of Page

Project Evaluation is Important to the Library of Michigan

The Library of Michigan is responsible for the administration of the Library Services and Technology Act in Michigan. The subgrant program awards federal funds to support a variety of library services. The Library of Michigan offers subgrant writing workshops to familiarize libraries with program guidelines, funding areas and application requirements. The workshops make applying for federal subgrants more manageable for applicants and the Library of Michigan alike.

Former State Librarian James W. Fry said, "Evaluation is the watchword for the 90's when using public funds." This applies to the potential subgrant recipient as well as to the Library of Michigan. The Library of Michigan seeks well-planned project proposals supported by meaningful evaluations. Evaluation of subgrant-funded projects is important to the Library of Michigan for many reasons. The results of project evaluation are used for:

• Identifying model projects to serve as examples for other libraries
• Completing reporting requirements of the Institute of Museum and Library Services
• Ensuring development of a well thought out project by the applicant
• Documenting to Congress the continued need for LSTA funding

Upon becoming a subgrant recipient, you are responsible for the expenditure of federal dollars. Likewise, the Library of Michigan is accountable to the federal government for the effective use of Michigan's allotment of LSTA funds. We share responsibility for the thoughtful expenditure of federal funds with you and because of that, the Library of Michigan is fully committed to strong, purposeful evaluations of every LSTA-funded project in Michigan.

Top of Page

Project Evaluation is Important to Your Library

In addition to meeting the requirements of the Library of Michigan, you should evaluate programs to plan effectively, to improve programs, and to increase program impact. A program or project is a set of activities, services, and materials organized to achieve a specific goal, supported by project objectives. Examples of LSTA-funded projects include an event sponsored by a library or an outreach service of the library or of the library cooperative.

One purpose of this handbook is to assist library staff in learning the simple steps that can provide a meaningful evaluation. Some library staff have been reluctant to evaluate programs. Common reasons for this reluctance include seeing no direct benefits for the library, believing there is a lack of knowledge or time to conduct a credible evaluation, or finally, fearing that evaluation results could show failure rather than success.

Evaluation of a project or a program can produce information that is useful to all interested parties. You may want to adjust a program in mid-stream to make it more effective, or decide whether to repeat a special event a second year, or approach a funding source to expand a given service. In any of these cases, information on the impact of your projectevaluation datawill be of great help. Effective evaluation requires additional investments of time and resources; however, the benefits of evaluation are worth these extra efforts.

Top of Page

The Stakeholder Approach to Project Evaluation

The Library of Michigan encourages an evaluation process that involves a representative group of those people who manage, receive services, or otherwise have an interest in a project. This is a stakeholder approach involvement of those people who have a stake in the evaluation findings and the success of the project. Library staff, library users and community leaders are all potential stakeholders. The identification and use of stakeholders is the cornerstone of the evaluation technique presented in this handbook.

The involvement of stakeholders in your project's evaluation will have many benefits. Overall, the varied points of view and expertise these individuals will bring to the project are likely to be very helpful. While the grant administrator has responsibility and authority for all aspects of the project, stakeholders can be an excellent resource whose advice may prove invaluable in planning evaluation.

Other benefits of the stakeholder approach include the added legitimacy that involving a broader group of individuals brings. This same group, extending beyond the library and into the community, may also bring added community support before, during and following the project's life.

The Library of Michigan strives to make the subgrant application process understandable to applicants. We also want to help you to plan strong projects and to write effective proposals. The better organized the plans are for a project, the more likely it is that the project will be a success. Project evaluation is an integral part of the Library of Michigan LSTA application form. Evaluation should also be an essential part of all project planning. In reviewing this handbook, notice that the planning process for a project and planning for the project's evaluation should occur simultaneously. Conducting evaluation as a continuous process during the life of a project will help to insure the success of your project.

A final word about evaluation before you begin to read this handbookthere is no reason to restrict evaluation to LSTA subgrant projects. While this handbook has been developed within the context of the Library of Michigan's LSTA subgrant program, the stakeholder approach to evaluation can be applied to any program or project your library is undertaking. The desire to offer useful and successful services to their users is common to all librarians. Evaluation can serve as the key ingredient in your recipe for success by providing information for project fine tuning as well as the documentation needed to seek additional funds. Feel free to use the ideas presented here in many areas of your library's operations.

Top of Page

SEVEN STEPS of Project Evaluation

The seven steps for project evaluation that take the reader from start to finish are:

1. Develop a Preliminary Project Plan
2. Identify the Project Stakeholders
3. Hold a Meeting of Project Stakeholders
4. Redefine the Project Proposal
5. Collect and Analyze Evaluation Data
6. Summarize Evaluation Data for Stakeholders
7. Prepare Final Evaluation Report

Step One: Develop a Preliminary Project Plan

As indicated earlier in this handbook, your first step must be to examine your program and service needs in comparison with current LSTA funding areas. After identifying the areas eligible for LSTA funding, begin thinking about each individual project's goal and objectives. Long before you begin to complete an application form, think through your project carefully, beginning to identify what you will do to achieve each objective and who will be involved in the project.

Back to SEVEN STEPS

Step Two: Identify the Project Stakeholders

The cornerstone of project evaluation design is the participation of the stakeholders; those people who have a stake in the evaluation findings. Library staff, library users, library board members, and other community leaders are all potential stakeholders in a program or project. These individuals may be involved in the project's implementation, may be in decision-making positions for future project funding, or may be potential recipients of project services.

Evaluations should be designed to be both useful and used. Before you begin, ask yourself these questions:

• How can I design the evaluation?
• How should I collect and analyze the data?
• How should the findings be reported, so that my stakeholders can plan, implement, and increase the impact of our program?

Answering thse questions requires addressing the needs of stakeholders and being responsive to the ways in which information is used within your organization.

Early and continuous involvement by representatives of the various stakeholder groups will increase the likelihood that your evaluation findings will also be used by the stakeholders. Their participation means that they will have a say in the information to be collected; they will feel a sense of ownership in both the evaluation and in the program or project itself.

Back to SEVEN STEPS

Step Three: Hold a Meeting of Project Stakeholders

A successful meeting with stakeholders will require that these three sub-steps are completed. You will need to:

1. Prepare an agenda for the meeting
2. Review the Project Evaluation Worksheet prior to the meeting
3. Hold the meeting, following the agenda items

A. Prepare an agenda for the meeting

Invite representatives of each of your stakeholder constituencies to a meeting. At the time these invitations are issued, you may want to seek input from these individuals regarding additional stakeholder constituencies for your consideration. At this meeting, plan to discuss the overall purpose of the proposed project or program, the tentative outline of project activities, and the purpose of the evaluation. A tentative agenda, ready to be individualized with your library name, meeting place and time for this first meeting, is included in Appendix A. It is helpful to mail copies of the agenda to participants in advance of the meeting.

B. Review the Project Evaluation Worksheet prior to the meeting

A Project Evaluation Worksheet (Appendix B contains a completed sample worksheet) is designed to help you ask and answer all the questions necessary to create a project evaluation at this first meeting. To ready yourself for the meeting, you should develop the basic details of the proposed project, be prepared to share this information with stakeholders, and have carefully reviewed the Project Evaluation Worksheet. As the subgrant administrator, you should lead your stakeholders through a review of the worksheet, and have specific ideas on the desired outcomes of these efforts. Resist the temptation to skip ahead in the questions. The sequence of these questions is important. Take time when you start designing your evaluation to answer each one as fully as possible. The completed Project Evaluation Worksheet may be included as a component of your LSTA subgrant application.

C. Hold the Meeting, Following the Agenda Items

1. Introduce Meeting Participants

2. Review Purpose of Meeting

• Identify Source of Subgrant Funds
• Discuss Community Need in Relationship to Subgrant Funding Categories
• Present Overview of Project Concept
• Describe Stakeholder Evaluation Process
• Summarize Efforts to Date

3. Describe the Proposed Project

• Goal and Objectives
• Activities
• Budget
• Evaluation

4. Discuss and Complete Project Evaluation Worksheet

5. Determine On-Going Role for Meeting Participants

6. Schedule Next Meeting Date

7. Adjournment

Agenda Item 1. Introduce Meeting Participants

Introduce each of the participants, including the agency or constituency which they represent, and their relationship to the proposed project.

Back to agenda list

Agenda Item 2. Review Purpose of Meeting

The better prepared you are to present the project plan and the overall purpose of the meeting, the more productive the group's efforts will be. Be clear about what the project or program will try to achieve, as well as the reason you are meeting with the group. This agenda item allows you to cover five important points while describing the purpose of the meeting: identifying the source of the grant funds, discussing community needs in relationship to grant funding categories and your library's mission, presenting the broad project concept, describing the stakeholder evaluation process, and summarizing your efforts to date.

Back to agenda list

Agenda Item 3. Describe the Proposed Project

In order to help your group to get a firm understanding of the proposed project, outline the goal and project objectives, the activities you expect will take place to achieve each objective, and the budget. You may want to re-emphasize the importance of the group's work by ending with the potential for project evaluation. The adage, "If you don't know where you're going, how will you know when you get there?" applies here. Understanding your overall project goal and knowing the objectives and their desired outcomes is essential for both you and your group of stakeholders.

Back to agenda list

Agenda Item 4. Discuss and Complete Project Evaluation Worksheet

The Project Evaluation Worksheet which will be completed during this meeting is a key planning document for project administration. It helps to set the foundation blocks for what the project will be and how its success will be assessed. The worksheet (See Appendix B for a completed sample) identifies five questions which must be answered in developing your project evaluation:

1. What questions will the evaluation answer?
2. What sources of data will be used to answer the questions?
3. What methods will be used to collect the data?
4. In what forms and to whom will the evaluation be reported?
5. How will the evaluation results be used?

Determine what questions will be used to guide the measure of project success before identifying what data you will collect. Avoid the temptation of determining the data that can be easily gathered for the project and designing evaluation questions to match. The key step is identifying the questions that the evaluation should answer in order to determine the impact of the project. Your project evaluation should ultimately seek to answer the question, "So what; what difference did this project make?" This is a role for the stakeholders. Ask them what they want to know and why this information is important.

For example, evaluation of a project to develop a special library collection for older adults might address the following questions:

• What were the characteristics of the people who used the collection?
• Was the collection used by older adults in the community?
• In what ways did the new collection have an impact on the library?
• In what ways did the collection have an impact on the local communities?
• What do older adults think of the library and the new collection?
• Is the cost of the collection worth the benefits that it provides?

Both objectives and the methodology or activities in your project proposal may be relatively easy to measure. For example, evaluation data can be gathered to give you counts of items purchased, numbers of individuals in attendance at a program, and so forth. When looking more broadly at your project or program goal, you and the stakeholder group must identify questions whose answers will document real success or project impact, but may be more difficult to measure. The questions you seek to answer should go well beyond counts of materials purchased or numbers of individuals in attendance at an event.

For example, if the goal of your project is to enhance the image of the library as an information provider in the community, potential indicators of achievement of this goal include:

• Has library funding improved?
• Have general community attitudes toward the public library changed?
• What is the reputation of the library among community leaders?
• What are the library staff's attitudes toward the library's services?
• In what ways is the library used by the local community?

Let the evaluation questions determine the best methods and procedures to use in data collection. Be sure to identify your project's evaluation questions before selecting any collection methods or data sources. Avoid reversing this order by attempting to determine how you will collect the data before developing the questions. If you are asking, "How many people in the target population were affected by this program?" then records of program usage and demographics of participants should be reviewed. If you are asking, "How does this program affect library users?" then a survey of all users or interviews of small groups of users are possible strategies.

People, library statistics, reports, and program records are all potential data sources to answer your evaluation questions. Frequently many sources will help you answer a question. By identifying the best data sources for answering each question, you will also be able to select the most effective data collection methods.

For example, to answer the question, "How has our new library facility improved the quality and quantity of service to library users?," data could be collected from the following sources:

• Circulation records
• Comments by library users
• Applications for library cards
• Community groups that use library facilities
• Head count of library users
• Comments by public officials
• Comments by outside consultants
• Comments by library staff

While the stakeholder group may identify multiple sources, you should select a limited number of reliable sources for use in answering any single question. Choices should be made based on the sources which will provide you with the best data. In deciding, you will also want to take into consideration available staff time, reliability of data, financial resources, timeliness of information, and deadlines for reporting the findings.

Useful evaluations can be conducted using any one of a wide variety of data collection methods. The Data Collection Methods table in Appendix C lists seven general categories of methods that can be used to evaluate library programs or projects. Your choice depends on the question that you are asking and the sources from which you want to collect the data. You can use the table to determine which methods are best to use in your situation. Many excellent resources on evaluation methods are available. A bibliography that includes a few of these sources is provided at the end of this handbook in Appendix D.

The last two questions to be answered on the Project Evaluation Worksheet relate to matters following the data collection phase of your work. As the stakeholders discuss their interests and responsibilities in the proposed project, help them to identify the most useful formats in which evaluation information should be reported, how the information will be used, and the potential outcomes of their deliberations after project evaluation. Appendix B provides a sample of possible formats for reporting data and ideas on the possible use of evaluation results. Stakeholders may not be aware of the potential for project fine tuning, extension, and/or expansion that this information may provide.

Back to agenda list

Agenda Item 5. Determine On-Going Role for Meeting Participants

Your final agenda item, other than setting another meeting date, should be identifying the ongoing role, if any, for the stakeholder group. Appendix E lists many other roles for members of this group. At the very least, invite a smaller core group to serve as an ongoing advisory committee for the project. This group can assist in pilot testing evaluation methods, conducting some phases of actual project evaluation, and providing advice on mid-project fine tuning.

Back to agenda list

Agenda Item 6. Schedule Next Meeting Date

You may actually be setting dates for two meetings. While your smaller core group may need to meet again in the near future to discuss the development of evaluation instruments or related matters, the entire stakeholder group should plan a meeting at the conclusion of the project at which time you share evaluation results and the draft of final project reports.

Back to agenda list

Back to SEVEN STEPS

Step Four: Redefine the Project Proposal

As we have already said, your project plan is likely to benefit from the varied experiences and points of view of the stakeholder group. It is probable that following the completion of the stakeholder meeting, new ideas will have surfaced, inspired by the proposed project. As grant administrator, you have the ultimate responsibility of evaluating all recommendations and advice, and consolidating it into a well-written project proposal which has the strongest likelihood of meeting the project goal.

After completing the review and revision of all aspects of the proposal, you will seek the endorsement and approval of your library board, supported by appropriate signatures, and submit the completed application to the Library of Michigan for consideration.

Back to SEVEN STEPS

Step Five: Collect and Analyze Evaluation Data

Take advantage of the period prior to the announcement of subgrant awards by developing and testing data collection instruments or investigating data sources. When appropriate, new methods should be tested before being used to actually collect the data for evaluation of the project. This pilot test is particularly important for surveys but it is a critical part of the design of many methods. Even if you are using existing records, try out your procedures on a small portion of the records first to see if you can locate and interpret the data that you want. Following the pilot test, review the experience with others in the stakeholder group before proceeding. Make any necessary changes in the instruments and procedures before moving to the stage of actual data collection.

Whichever methods you use to collect data, respect the privacy and the rights of the individual. You may ask for permission when using data that could be considered confidential, and if you promise confidentiality then make sure that effective procedures are in place to guarantee it.

Back to SEVEN STEPS

Step Six: Summarize Evaluation Data for Stakeholders

Following the completion of all data collection, select and organize the data according to the evaluation questions. Collate the data from each source and method, such as questionnaire surveys and individual interviews, under each question.

The procedures for analyzing data differ depending on whether the information you collect is quantitative or qualitative. In general, quantitative data are used to measure the extent of something this is reported numerically, for example the number or percentage of people who gave each answer on a questionnaire; or the number of new library cards issued, the increase in interlibrary loan requests processed, or the number of database searches conducted.

Qualitative data is gathered through open-ended answers to interviews, questionnaires and narrative observations of events, and can be categorized to answer the evaluation questions. It is based on values, not numerical data. For example, seniors' comments about a new collection might be categorized as one of the following: "helpful with retirement," "entertaining reading," or "does not have what was wanted." In doing data analysis, you must take the quantitative and qualitative information you have collected and determine its significance in both numerical and narrative forms. The analysis should be presented in a way that answers the project's evaluation questions. Consider reviewing the resources listed in the bibliography of this handbook for assistance in selecting the appropriate data analysis method.

To aid your stakeholder group in interpreting the project data, you will need to summarize the information simply, accurately, and clearly. Identify key themes that are suggested by the findings. Providing charts and graphs to express data from lists and counts will also help stakeholders understand the information.

Back to SEVEN STEPS

Step Seven: Prepare Final Evaluation Report

The stakeholders should be involved in the interpretation of the findings and in reacting to preliminary and final reports. Find out what individual stakeholders believe are the implications of the data. Their involvement will add credence to your overall findings and will create a sense of stakeholder ownership in the results of the evaluation and in the program. Stakeholder involvement at this point can also bring added visibility to the program when emphasized with a press release or other publicity techniques. The original evaluation questions developed by this group at the beginning of the project can serve as the guide for this discussion. The stakeholder group's final report may then be included in the report sent to the Library of Michigan and to other appropriate audiences.

Back to SEVEN STEPS

Top of Page

Wrapping Up

You may want to consider meeting with the stakeholders again to plan future actions for implementation of their recommendations. These might include continuing certain activities, changing others, or working to create even greater public awareness of the program or service. Continue to evaluate an ongoing project or program with help from the stakeholder group.

If the stakeholder evaluation design process has worked well, the library staff, users, and people from the community will be interested in the findings. They will use the findings as evidence for continuing, improving, expanding, or discontinuing the project. The thorough evaluation you have done will help to assure that an appropriate response is made, and the reputation of the library for sound management will be enhanced.

Top of Page