In 2016, NRCan’s Office of Energy Efficiency (OEE) established a Social Innovation "UnLab" (SIU) to work with partners in and out of government to advance energy efficiency policy innovation and experimentation.
On that front, this year, we’re participating in an initiative called Experimentation Works (EW) led by the Treasury Board Secretariat, which aims to advance experimentation and improve policy and service outcomes for Canadians. One of our EW projects is a home energy label experiment using Carrot Rewards, which demonstrates how the OEE is adopting new tools and agile and inclusive approaches to scale energy efficiency impacts.
Applying experimentation to policy and service delivery: an inclusive approach
NRCan’s EnerGuide label aims to help homeowners become aware of their home’s energy performance and encourage energy efficiency actions, like retrofitting their homes. The OEE has been collaborating with partners to directly engage EnerGuide users, including homeowners, to understand their experiences and needs to inform where we can innovate to improve EnerGuide’s reach, uptake and impact. As a result, a question emerged: does the EnerGuide label effectively convey energy efficiency information to homeowners? Through this experiment, the OEE will learn if different label designs influence homeowner understanding to inform EnerGuide’s continuous improvement and the transition to digital solutions.
We’ve also been working with Carrot Insights. Their healthy living and wellness app, Carrot Rewards, leverages Artificial Intelligence (AI) and smartphone technology to engage over 800 000 Canadians. Carrot users are rewarded with loyalty points of their choosing (e.g., Scene, Aeroplan, RBC Rewards) when they complete offers that demonstrate learning and action.
When we heard about EW ramping up in 2017-18, we were well positioned to say, “sign us up!”
Building the team: mobilizing talent in an agile way
Our EW experiment required testing different graphics to see how well they conveyed energy efficiency information to homeowners to inform EnerGuide design and delivery. We’d been putting experimental approaches into practice, but a randomized control trial (RCT) was new to us. We needed to tap into new skills!
"I’m most proud of the way we’re working – with others, in the open, trying new things, testing and iterating as we go."
- John Kenney, Manager
We engaged home energy efficiency subject matter experts in our Housing division and NRCan’s Experimentation and Analytics Unit, who have expertise in applying behavioural insights and RCTs in policy and service contexts, to work with us on the project. Together, we refined our research question and selected different label designs to test, and worked with a graphic designer to create the graphics.
We also needed a platform and plenty of users for our experiment so it was a logical step to see what was possible with Carrot Rewards. We brought the idea to the Carrot team and they were, as always, game to play.
Designing the experiment
Our Carrot offers run for a few days only in order to reach and engage enough Canadians. How much is enough? Typically, determining what is “enough” depends on budget and time. For a randomized control trial, those things still matter, but the size of the sample determines the power (i.e. reliability) of the results. We calculated that we needed 30 000 users to complete the experiment for enough statistical power to detect differences between our graphic treatments.
Randomly assigning an offer to users was easy for Carrot - they routinely do this as part of the app experience. Check!
The trickier task, due to a technical constraint, was figuring out how to randomize the different label combinations within the offer. We worked out a solution by creating an independent offer for each combination: 36 offers randomly assigned to 30 000 users with common questions but different graphics. This created more work, but was feasible.
Iterating to get it right
Early testing on the Carrot platform showed that the graphics we created were not easily readable on small smartphone screens. There wasn’t enough real estate on the screen for our original designs! The medium truly is the message – our graphics had to be readable on the app for the integrity of the experiment and to meet user experience standards.
Working with our graphic designer, we reduced the image size, altered the fonts, and adjusted colour contrast to improve on-screen readability. We looked at different ways to display the dual images to optimize the visual experience. We finally settled on vertically stacking the simplified image pairs, giving us the largest, clearest image possible. Pre-launch testing will help us gauge if we have hit the mark on the visual quality of this design.
Learning as we go and in the open
As we equip ourselves with new tools and methods (e.g. randomized controls trials) and adopt agile and inclusive ways of working, we are learning a lot. The EW cohort has been blogging about our experiences, opening up our learning events to other public servants, and supporting each other throughout the process.
Our energy label experiment will launch on Carrot Rewards this month and we will have preliminary results soon after that. All EW experiments will openly report findings and lessoned learned this winter.