Proven economic outcomes are an effective way to gain the attention of public stewards.
Evaluation as a learning tool
Evaluation is a learning tool that funders and grantees use to measure an initiative’s impact on a community. It is a way for people to learn what works and what doesn’t. NEI, learning as it evolved its overall economic development strategy, also evolved its evaluation and impact approach over the length of its investments.
How to establish a solid evaluation plan
Hire an external evaluator
An evaluation consultant will develop an evaluation process and strategy that…
- engages the initiative’s stakeholders to ensure that all parties are aligned in terms of outcomes;
- immerses its evaluators with the funding team to connect the evaluation framework and specific measurable objectives with the design of the overall initiative;
- narrows the scope to what is most critical to measure;
- determines the data collection methods;
- builds a platform to receive the reporting; and
- creates a communications plan for sharing results and for stakeholder reporting.
Assess grantee readiness
Many grant-funded neighborhood organizations (business support organizations) have not done rigorous program evaluation in the past. When funders and communities introduce milestones, metrics, and financial reports for evaluation, the process can seem daunting.Â
Allowing time to transition business support organization grantees to a more detailed, quantitative, and responsive approach to reporting is necessary. Multiple meetings with each grantee will ready them for the transition; for some it will be relatively painless, but for others, it will be a significant struggle.
Measure what matters
Performance metrics, data and analysis allow for well-informed funding decisions to drive continuous improvement and long-term impact in any given community. The core purposes of an evaluation are to make judgments about a funded program, to improve its effectiveness, and/or to inform programming decisions.
Use data for decision making
Ultimately, integrating performance measurements into daily operations will allow funders access to solid data in order to drive decision making, illustrate progress and establish a culture of continuous learning that will lead to amplified social and economic impact.
What is evaluation?
Evaluation is a systematic set of processes to analyze a program’s performance. It involves collecting and analyzing data about program activities, characteristics, and outcomes. Evaluation can fall within two broad categories: formative and summative.
Learn more
- Formative evaluations are conducted during program development and implementation; they are useful for directing how to best achieve the goals or improve the program.
- Summative evaluations are completed once a program is well-established, and they illustrate the extent to which a program is achieving its goals.
The information below, provided by Growth Capital Network, NEI’s chief evaluator, explains when to use formative or summative evaluation methods.
Formative Evaluation Types
- Needs Assessment
- Determines what populations need the program, the size of the need, and what activities best meet that need. A needs assessment can determine what populations are not served by the current programs and offer insight into what features the new programs should imbed to meet these needs.
- Process or Implementation Evaluation
- Reviews the implementation process of the program and establishes whether the program is operating as planned. This can be performed as a continuous or one-time assessment. The resulting data are used to improve the program.
Summative Evaluation Types
- Outcome Evaluation
- Reviews to what extent a program is achieving its desired outcomes (short- and medium-term transformation in participants) that are a direct result of the program.
- Impact Evaluation
- Determines the broad, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the individual, their business, or community.
Why evaluate
Demonstrate funding investments make a difference
No matter the differences among funding sources, they all share the need to know what works and, especially, what works well. Grants that make a difference bring more value to their communities. They do so by:
- positioning efforts,
- informing investment,
- promoting stronger relationships, and
- helping funders reach their goals.
To know what is working, foundations must evaluate their grants to
- assess the quality or impact of funded programs,
- plan and implement new programs,
- make future grant decisions, and
- demonstrate accountability to the public trust.
Some funders use evaluation to determine if their efforts are aligned across their grant programs, initiatives, values, and mission. Effective focus and alignment create an opportunity for more strategic impact, as the synergy of efforts and dollars work together serve the overall mission.
Create understanding that leads to informed investments
Promote collaboration
Strengthen grantee outcomes
NEInsight  Client data fields
These are the items NEI asked its business support organizations to collect data on the business owners they served; these were also used for initiatives where NEI directly provided awards of dollars and practical assistance to small businesses.
Learn more
Company Name
Number of Founders
Founder1 First Name
Founder1 Last Name
Founder1 Age Range
Founder1 Email
Founder1 Phone
Founder1 Resident of Detroit, Hamtramck, Highland Park
Founder1 Minority Owned
Founder1 Immigrant Owned
Founder1 Woman Owned
Founder2 First Name
Founder2 Last Name
Founder2 Age Range
Founder2 Email
Founder2 Phone
Founder2 Resident of Detroit, Hamtramck, Highland Park
Founder2 Minority Owned
Founder2 Immigrant Owned
Founder2 Woman Owned
Year Company Started**
Year Became Client**
Current Employee Count**
Estimated Number of Employees Residents of Detroit, Hamtramck, Highland Park
Current Annual Revenue ($value only)
Total capital (value only)
Primary Funding Source
Street Address
City
State
Zip
Website URL
Industry Category**
Notes
Evaluation tips
Connect metrics to program activities
It’s important to connect the metrics to program activity. Developing a logic model for each program helps to articulate how activity inputs translate into outcomes.
Look beyond basic trends
Beyond basic year-over-year business trend summaries, business dynamics (churn) and job quality (living wages, benefits provided, skill development) are other metrics to consider when reviewing the small business landscapeÂ
NEInsight  Statistical indicators NEI used to measure progress
Learn more
Program
- # Of one-on-one company interactions
- # Clients Served
- # Company Clients
- # Company Clients – Minorities
- # Company Clients – Women
- # Company Clients – Immigrants
- # Company Clients – Detroit based companies
- # Companies Clients – with secondary degree
- # Online Users
- # Online Users – Large Corporations
Company Growth Metrics
- # Companies Formed
- # Companies Formed – Minority Owned
- # Companies Formed – Immigrant Owned
- # Companies Formed – Detroit based
- # Companies Formed – Woman Owned
- # Companies Formed – Women > 45 years old
- # Companies Formed – Formed by women with college degrees
- Research Expenditures per Invention (IDF)
- Portfolio – # of new patents
- Portfolio Licensing Income
Industry growth – put the companies into buckets
- # Portfolio – Aerospace
- # Portfolio – Automotive/Manufacturing
- # Portfolio – Construction
- # Portfolio – Creative Arts
- # Portfolio – Design/Technical Services
- # Portfolio – Education
- # Portfolio – Energy/Alternative Energy
- # Portfolio – Entertainment/Hospitality
- # Portfolio – Facilities Mgmt.
- # Portfolio – Food
Investment metrics
- # Companies Receiving investment
- Total Investment Capital Received by Clients
- Total Investment Capital to Minority Owners Total Investment Capital to Women Owners Total Investment Capital to Immigrant Owners
- Total Capital Investment – Public Dollars
- Total Capital Investment – Private Dollars
Space utilization
- # Tenants
- # Tenants – Physical Space
- # Tenants – Mailbox
- # Tenants – Virtual
- Square Feet Available
- Square Feet Utilized
- Square Feet Available – Testing facilities
- Square feet under management
- Square Feet Utilized – Testing facilities
- Square feet developed
Measure economic activity
Understanding how to measure economic inclusion can be challenging. Methodologies that capture relevant data about program impacts related to racial, ethnic or gender diversity — and related to the distress of the places where investments are made — should be used.
Have baseline economic data – from the beginning
This tip speaks for itself and bears repeating. Without a baseline, it’s hard to be sure that progress is happening. It may take discipline on the part of the funders to think about this at the very beginning, but it is well worth the time and investment.
Use baseline data to see the micro-movements
Measure economic topography
Using sets of indicators (leading, coincident and lagging) across multiple perspectives allows for a deep understanding of the economic topography of the ecosystem being created, augmented, or scaled. At the baseline, these indicators might just look like a list of numbers. But as time passes, the opportunity to see whether the infusion of capital into a particular section of the city is spurring activity or making an impact will be far more nuanced and interesting.
Leading indicators point toward future events
Leading indicators give a sense of the financial vibrancy of the community. These include important local data such as average weekly hours, initial jobless claims, building permits for new private housing units, manufacturers’ new orders – and national data such as the S&P 500, money supply (M2), interest rate spread, and consumer expectations.
Learn more
Relevant coincident indicators include local real earnings and unemployment, as well as Gross Domestic Product (GDP) on a national level.
Lagging indicators confirm a pattern that is already in progress
Key lagging indicators include the average duration of unemployment, outstanding commercial and industrial loans, change in the Consumer Price Index (CPI), change in labor costs, the ratio of outstanding consumer credit to personal income, and the average prime rate charged by banks.
Understand that circumstances vary by community
Choosing statistical measures
Some national statistics may seem too broad or local data too narrow to be useful, but it all plays its part in illustrating the opportunities or challenges within a particular neighborhood.
See an illustration of common barriers
And a community with low wages, depressed housing prices, and chronically high unemployment cannot provide the same volume of friends and family risk capital to a burgeoning business or collateral for a small business loan. See an illustration of that here.
The data and trends provide the context and the contrast for the evaluation and create an opportunity to measure primary impact from a program, as well as potential secondary and tertiary effects.
Find data sources to depict regions and cities, small business ecosystems
The broad indicators in which to place local data come from a variety of places. A comprehensive list of sources are noted in the Why evaluate: A dispatch from Growth Capital Network, NEI’s evaluation partner, but some of the most valuable sources are:
- US Census Bureau
- US Bureau of Economic Analysis
- US Bureau of Labor Statistics
Sources for more narrowly focused data are Burning Glass (labor forces employment) and the University of Michigan (Consumer Sentiment Index). State and city government sites are useful for local stats as are the plethora of reports out in the ecosystem.
Resources for evaluation
Free ebooks by Leap of Reason to keep up with the latest thinking in outcomes evaluation, performance management, and evaluation:
- Leap of Reason
- Working Hard and Working Well
For small nonprofits: Small But Mighty (Leap of Reason’s Performance Imperative)
See more
W.K. Kellogg Foundation – Evaluation Toolkit *
The Step-by-Step Guide to Evaluation is designed to guide programs embarking on evaluation.
United Way of America – Measuring Program Outcomes: A Practical Approach *
United Way has developed a step-by-step evaluation manual for health, human service, and youth and family serving agencies.
State of Evaluation: Evaluation Capacity and Practice in the Nonprofit Sector
How nonprofits engage in evaluation, published by Innovation Network.
The State of Evaluation in Colorado’s Nonprofit Sector
Downloadable PDF from The Colorado Trust, CO Nonprofits, Community Resource Center.
How do we build the capacity of nonprofits to evaluate, learn, and improve?
Downloadable PDF on how grantmakers can support nonprofits’ efforts to learn, improve and, ultimately, expand their impact; published by GEO (Grantmakers for Effective Organizations)
DIVE DEEPER: Find a longer list of resources in the Why evaluate: A dispatch from Growth Capital Network, NEI’s evaluation partner