SEE Action Releases Guidance Report on Evaluation of Residential Behavior-Based Energy Efficiency Programs

A new report released by the State and Local Energy Efficiency Action Network (SEE Action) prepared by researchers at Lawrence Berkeley National Laboratory provides guidance and recommendations on methodologies that can be used for rigorously estimating energy savings from residential behavior-based efficiency programs. The report was designed for regulators, evaluation professionals, program administrators, and other energy efficiency program stakeholders.

Residential behavior-based energy efficiency programs have been identified as a major potential source of new energy savings and are increasingly being implemented by energy efficiency program administrators nationwide to help meet energy saving targets and requirements that now exist in many U.S. states. These programs utilize strategies intended to affect consumer energy use behaviors in order to achieve energy and/or peak demand savings. They typically use one or more elements to achieve their goals, including customer outreach, energy usage feedback, competition, rewards, benchmarking, or feedback elements. Such programs may focus on changes to consumers’ habitual behaviors (e.g., turning off lights), one-time behaviors (e.g., changing thermostat settings), or purchasing behaviors (e.g., buying energy efficient appliances)

However, the widespread adoption of these programs faces obstacles, including questions about whether observed energy savings are valid and attributable to the behavior program, whether the savings persist over time, and whether the results shown for one program can be applied to another program.

“We need rigorous, objective evaluation methods for these programs,” says Malcolm Woolf, Director of the Maryland Energy Administration and a leader of SEE Action for issues related to evaluation, measurement and verification (EM+V), “Strong standards ensure that program administrators, policy makers and regulators can be confident that the savings estimates claimed by these programs are valid.”

The guidance document identifies evaluation issues and recommends methods that ensure the validity of energy saving estimates.

“We recommend using a scientific experimental design method called a randomized controlled trial,” says Annika Todd, the report’s lead author. “This method is the gold standard for producing valid energy savings estimates that are robust and unbiased.”

Randomized controlled trials randomly assign households into two groups: one that receives the behavior-based program (the “treatment group”) and one that does not receive the program (the “control group”). This method allows the evaluators to determine whether any energy saved by the treatment group was due to the program, as opposed to other factors.

The report discusses methods for ensuring that the estimated savings impacts are valid and robust. Top-tier recommendations include:

• Evaluation design: randomized controlled trials
• Avoiding potential conflicts of interest: employ an independent third-party evaluator for program evaluation, assignment of households to treatment and control groups, and data selection and analysis
• Accounting for potential double counting of savings: use rigorous methods to account for energy savings that may be claimed by multiple programs
• Ensuring precision in estimates: 5% statistical significance
• Applying impact estimates to different populations in future years: a control group that is representative of all of the different participating populations should be maintained for every year in which program energy estimates are being used to claim savings
• In the future, it is possible that a calibrated analytic model may be created that is able to rigorously and validly predict program savings estimates

This guidance document is especially designed for state utility regulators, who are in a position to approve utility behavior-based energy efficiency programs. “Speaking as a regulator, we need independent, objective methods that we can turn to, and this provides exactly that,” says Phyllis Reha, Commissioner for the Minnesota Public Utilities Commission and a leader of SEE Action on issues related to customer behavior and energy efficiency.

State and Local Energy Efficiency Action Network. 2012. Evaluation, Measurement, and Verification of Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations. Prepared by A. Todd, E. Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National Laboratory. behavioranalytics.lbl.gov.

Download the report from the SEE Action website here.

The report and a companion powerpoint presentation with an overview of the report findings may also be downloaded from Lawrence Berkeley National Lab’s (LBNL) Behavior Analytics website here or from LBNL’s energy efficiency publications page here.

This report was prepared by Lawrence Berkeley National Laboratory for the State and Local Energy Efficiency Action Network, a state and local effort facilitated by the US DOE and US EPA that helps state and local governments,, utilities, and others stakeholders take energy efficiency to scale and achieve all cost-effective energy efficiency by 2020. Find out more about See Action here.