By Jamie Mandel and Gavin McCormick. Originally posted on RMI Outlet.

Carbon emissions are arguably the most important thing for our society to learn how to manage in the coming years. The largest single source of U.S. carbon emissions is our electricity system. And yet, we do not measure emissions from our electricity use correctly, meaning we cannot manage our emissions effectively.

But now, thanks to a new technology that accurately measures moment-to-moment carbon emissions on our electricity system, we can unlock a whole host of new opportunities to manage emissions creatively and with less effort. With new software that automatically tracks the actual emissions impacts associated with specific actions on the electricity system, both in real time and ahead of time, we can now use our appliances at times when our electricity is the cleanest.

End-use flexibility

Many uses of electricity have inherent flexibility—that is, the timing can be changed by small or large amounts without impacting the quality of the service that device is providing. As Rocky Mountain Institute explored in The Economics of Demand Flexibility, harnessing this flexibility can save consumers and companies money while lowering grid costs.

The same is true of carbon emissions—harnessing the flexibility of end-use devices can make them run, on average, 15 percent cleaner than a “dumb” device, at no cost or quality impacts for the end-user.

Millions of people and thousands of corporations try every day to manage their carbon emissions. Unfortunately, much of this effort occurs without measuring these emissions correctly. Personal and corporate efforts to manage carbon emissions from electricity typically happen in one of two ways:

1) Wthout any measurement, by focusing on efforts that are generally associated with reduced emissions. For example, many corporations invest in things like efficiency, solar PV, and grid-sourced clean energy, but do not attempt to quantify the emissions savings associated with specific investments.

2) With coarse measurement of average emissions intensity, primarily by using eGrid historical data to estimate averages for electricity-related emissions. For example, a corporation might deliberately site a data center at a location on the grid that is, on average, cleaner than other options and claim some associated carbon emissions savings.

Thanks to new technology, it is now possible instead to know the actual emissions impacts associated with specific actions at a specific place on the electricity system, in real time—and even ahead of time through predictive algorithms. More importantly, it is now possible to assess future decisions based on marginal—rather than average—emissions factors, which, according to most economists, is the correct way to properly understand emissions impacts.

The emissions hidden in the margins

The difference between average and marginal emissions factors can be very large, and quite important. An average factor refers to the amount of emissions generated over a given time, divided by the amount of energy produced in that time. For example, the U.S. Pacific Northwest gets most of its electricity from hydropower, a low-emissions energy resource, and thus its average emissions factor is very low.

A marginal emissions factor refers to rate at which emissions would change with a small change to electricity load. Continuing the simplified Pacific Northwest example, imagine a time when hydropower is providing 75 percent of the region’s power and gas-fired power plants are providing the remaining 25 percent. This means that the average emissions factor of power in the Pacific Northwest would be very clean, at 25 percent the emissions intensity of natural gas (approximately 210 lbs. CO2 per megawatt-hour (MWh)). So at first glance, a great way to reduce a company’s or a person’s carbon footprint would be to move to the Pacific Northwest, where the electricity is very clean.

Yet in many cases, natural gas is the marginal resource, meaning that if a new kilowatt-hour of electricity is needed at a certain time, it will be provided by natural gas. So a company or an individual moving to the Pacific Northwest would increase carbon emissions at a rate equal to 100 percent of natural gas (840 lbs. CO2 per MWh)—a very big difference! Thinking in marginal rather than average carbon emissions can dramatically affect a company’s or a person’s choice of optimal environmental impact.

Estimating emissions impacts based on average emissions factors can have these types of effects on a recurring basis, across the U.S. This is because the portfolio of generators dispatching energy into the grid changes every five to 15 minutes, changing the marginal resource. For example, Midwest utilities mostly burn coal at night; if you own an electric vehicle there, you would have lower CO2 emissions if you deliberately charged it during the day. On the other hand, California’s electricity market has more efficient gas plants on the margin at night than during the day, so you should charge your electric vehicle (EV) in the evening to minimize your CO2 emissions. And with an Internet-connected EV charger, you can cut emissions even further with micro-timing. For example, you can time the EV charging to shut down when less-efficient peaking plants briefly kick on (say when the wind subsides or a cloud passes over), and turn it back on five minutes later when the wind returns or the cloud moves on and the marginal generator is cleaner.

Things you can do when you account for carbon emissions correctly

Accounting for carbon emissions correctly unlocks a whole host of new emissions-management opportunities. You can:

• Change the timing of energy use: As noted above, this is likely the easiest and largest source of untapped savings since it doesn’t take any additional investments or impact end-use quality. It simply takes advantage of emerging controls technology associated with Internet-connected devices. Thermostats, EV chargers, commercial-scale HVAC units, and demand-response programs, for example, can all run with lower emissions rates with just a few lines of computer code.
• Change the location of both electricity use and generation: By locating energy generating (or consuming) facilities in a particular place, you can affect which power plants they take offline or use. Each MWh produced by a wind farm in West Virginia reduces twice as much carbon as a MWh from an identical farm in California. Yet today, a corporate buyer of renewable energy would view these two wind projects identically. To help buyers start to understand these effects, RMI’s Business Renewables Center has begun to track emissions intensity—the differences in emissions reduction from different renewables projects.
• Value different energy conservation measures based on their actual carbon reduction potential: Today, all so-called negawatts, or avoided energy generation, are valued equally. But negawatts should have different values, as the Ohio and Texas examples show. A negawatt that displaces a kilowatt of coal power does more good to the planet than a negawatt that displaces a kilowatt of natural gas, wind, or hydropower. Valuing on-site energy conservation measures and distributed generation based on actual emissions reduction potential can support better decision-making.
• Price carbon markets correctly: If we do move to a world in which carbon emissions have a price, it would be great if it has the right price. Currently, renewable energy credits (RECs) and carbon offsets are priced, but only on average carbon impacts. Differentiating highly carbon-reducing RECs will lead to more cost-effective emissions reduction.

RMI and WattTime are working together to measure carbon emissions correctly and reduce them cost-effectively.

WattTime is a California-based nonprofit that has developed software to accurately forecast carbon emissions on the margin, in real time. This data can be used to control the timing of device charging, apply carbon emissions data to the models that renewable energy developers use to site projects, provide strategic advice to corporations on how to most cost-effectively reduce emissions, and provide more accurate reporting and verification of emissions.

RMI is using this new technological tool to unlock new markets for carbon reduction, and to maximize the value of these reductions. This technology can be used to improve the profitability of distributed energy resource companies and retail energy providers by lowering customer acquisition costs, accelerating corporate sustainability efforts, and improving the way that carbon emissions are measured and, ultimately, priced.

For example, 240 EV customers nationwide are charging their EVs with cleaner energy than their neighbors. Thousands of thermostat customers in Chicago are learning that cooling their houses with fewer carbon emissions is as easy as pushing a button. By using WattTime, millions of independent devices can be seamlessly checking the emissions content of the grid and making small decisions about the timing of electricity use to lower carbon emissions.

A key founding principle at RMI is that people don’t want raw kilowatt-hours. They want hot showers, cold beer, and illumination. Similarly, the planet doesn’t care how many kilowatt-hours we reduce. It cares how much we reduce CO2 emissions. So why not start measuring them directly? Together, we will help people and companies easily reduce their carbon emissions to help create a world that’s thriving, verdant, and secure, for all, for ever.

What’s the problem?

WattTime analyzes power grid data in real time from dozens of open data sources. Because the data is used to optimize the behavior of smart devices in real time, it is very important that we always have the most accurate up-to-date data. This means that we have had to tackle a classic engineering problem: building a highly reliable system out of less reliable components.

There are two main sources of unreliability in our data ingestion system. First, any of our incoming data sources can go down for a period of time without warning, creating a gap in our data record. Second, our cluster of worker servers may not run the data scraping jobs for any number of reasons, deepening the potential gaps. You can see how we’re in a tricky predicament of always needing the most accurate data and yet having a number of reasons for something to go wrong.

As the saying goes, if you can’t measure it, you can’t manage it. Today’s post explains how we built a monitoring and alerting system to detect gaps in our data ingestion pipeline using Librato.

Designing the Solution

Here are the primary characteristics we wanted in a monitoring and alerting system:

1. The solution needs to run 24/7. We want to provide our end-users with cleanest available energy, and whether the current energy supply is clean can change every five minutes. If our data is not up-to-date, then the energy supply can change without our knowledge, and we might miss an opportunity to give our users a choice to save carbon.

1. The solution needs to run in a way that's isolated/decoupled from how the data scraping tasks normally get run. We don’t want our monitoring system to be dependent on the system it’s supposed to be monitoring! This means that we’d either have to spin up our own separate monitoring service (and maybe a monitor for the monitor…) or use a reliable third-party SaaS tool.

1. The solution needs to have a way to identify any new gaps in our data as soon as possible, so we can triage and fix the problem before it gets worse. If we’re using a third-party tool, that means we need to give it a way to hook into our data pipeline.

1. The solution needs to have a way to send us alert messages when a problem occurs. We like the workflow of Slack, but email would be ok as a fallback.

1. The solution should make it easy to configure the frequency and thresholds for triggering alerts. Overly noisy alerting systems get ignored, so we thought it was a good idea that any system we implemented didn’t bombarded us with notifications, just sending the important actionable ones.

The WattTime API is currently running on Heroku, a popular cloud platform-as-a-service provider. Heroku has a great ecosystem of high-quality third-party “add-ons” that we can trust to have good uptime (satisfying criterion #1), even if something bad happens on our end (satisfying criterion #2). We decided to start by surveying Heroku’s add-ons to see which ones would help us satisfy our other design criteria: data pipeline integration (#3), Slack integration (#4), and configurability (#5).

After some doc hunting, we established that many add-ons had Slack integration, so that didn’t narrow our solution space very far. Instead, we decided to make our choice primarily based on the mechanism of integrating with our data pipeline. There was a wide range of options here: some add-ons would collect data from us if we printed it to our logs, others would collect data if we raised an exception, etc. Exception-based add-ons would be a great fit for detecting failed requests during data ingestion, but they wouldn’t help us monitor failures in our worker cluster overall. A sufficiently configurable log-based add-on, on the other hand, would be able to send alerts either if a problematic value appeared in the logs, or if the log stream stopped getting updated altogether. If such an add-on existed, it would allow us to meet all five design criteria.

And it does: Librato! Librato is a service for visualizing and creating alerts based on metrics. The Heroku add-on comes preconfigured to read metrics from Heroku log streams. In fact, we were already using Librato to visualize dyno performance metrics that Heroku printed to our log stream—but we hadn’t tried configuring any alerts. Reading Librato’s docs on alerts, we discovered that it supports both kinds of alert triggers we wanted. This made it a great choice for a monitoring service that we can configure to watch our data and alert us if something unexpected happens.

Step 1: Logging data quality

Librato is organized around the concept of a “metric.” A Librato metric can be any kind of time series data: something that can be graphed with a time stamp on the x axis and a number on the y axis. When the metric hasn’t been reporting or hasn’t logged a new datapoint within a certain period of time, visualizations like this can help us clearly see what is going on.

We chose “lag time” as the metric to track. We define lag time as the age of the most recently ingested data point of a particular type. Lag time serves as a good metric because it helps us get to the heart of the problem. If our goal is to have the most accurate up-to-date data, we’d like to know when our gaps start and for how long they occurred for.

To implement data quality logging, every time we collect data, we figure out how old the newest data point is, and print that number of minutes to the logs in Librato’s specific format. Here’s a screenshot of what the output looks like in our Papertrail logs:

Step 2: Setting up alerts

We have several ways to detect if something has gone wrong with our data collecting. If a new datapoint has not been added within an acceptable amount of time, then we may want to get alerted and see if there is anything we can do on our end. To do this, we set “condition type” to “goes above” when creating an alert:

We can also check if our metric stops reporting entirely. In that case, we set “condition type” to “stops reporting”. Both alerts serve to inform us as quickly as possible if something were to go wrong.

Step 3: Receiving alerts

To set up Slack integration, Librato requires you retrieving the Webhook URL for your Slack channel. In the Slack desktop app, go to the top left and click on Apps & Integration. This will take you to a directory of different apps that can integrate with Slack. Search for "Incoming WebHooks."

From there, go to "add configurations" and you should be able to find the webhook URL.

Once you have the URL, you can go to Librato’s Integrations tab and click “add configuration” on the Slack tab. Paste in the URL and give it a title you’ll be familiar with.

Then whenever an alert is triggered, you’ll receive a Slack message showing you why the alert was triggered and other relevant information. 

What we're thinking about next

While it’s really nice for our system to alert us whenever anything goes wrong, we thought that maybe it would be even better practice if our system was self-healing. As it stands, our system sends us an alert, and from there a living breathing human being has to take time out of what they’re doing and investigate. So for our next step, we will create a system to patch up data holes as they happen. In this way we can be more sure that we have the most accurate up-to-date data at any time.

Do you manage a large commercial or industrial facility? Perhaps a university, apartment complex, or hotel? Today we're pleased to share that, thanks to WattTime's newest partnership with Building Clouds, you can now cut your emissions through WattTime-enabled technology in nearly any equipment.

Facility managers everywhere have learned that not all equipment manufacturers play nice with each other and that interoperability is a major problem. What we love about Building Clouds Strati-Fi(TM) controllers is that they allow for remarkably quick and cost effective monitoring and control of most equipment types in commercial or industrial scale HVAC equipment. So whatever your building currently runs on, you can now WattTime-enable it by giving Building Clouds a call.

We've been working with Building Clouds for over a year to pilot this technology and work out all the kinks. Our first project began with Sutardja Hall, which became the first building in the world to optimize its HVAC package's energy loads in real time to cut carbon emissions. Building Clouds President Bob Wallace met with us on the rooftop of Sutardja Dai Hall and demonstrated the technology installation, which took place in under an hour. We were able to collect data almost immediately and automatically started cutting carbon less than a week later.

The implementation and results of this went over so well with UC Berkeley that we recently agreed upon a second project for the Residential and Student Services Building (RSSB). This deal included connecting the Strati-Fi(TM) controllers to WattTime-enable two sixty-ton air handlers. Both projects have successfully achieved absolutely zero increase in the building's energy bill or on occupant comfort. In fact, as a mischievous test, we did not announce the project until one month after installation, during which no building occupants were even aware of any change.

After over a year of successful pilots, we're delighted to share that WattTime and Building Clouds are scaling up our partnership to release our technology to the broader market. If you're looking for cost-effective building automation solutions that also cut emissions, please take a look at buildingclouds.com or WattTime's own Shop page.

We are thrilled to announce that WattTime has received a substantial grant from the Great Lakes Protection Fund to lead a coalition of nonprofits and companies in reducing mercury pollution from coal plants. The project will be a collaboration between WattTime, Rocky Mountain Institute, National Wildlife Federation, Delta Institute, Energy Emissions Intelligence, and several corporate partners.

In 2008, a Federal court struck down the national Clean Air Mercury Rule that required coal-fired power plants to limit dangerous mercury emissions. With repeated attempts to replace the rule continuing to face uncertain political futures, badly-needed efforts to return to safe mercury levels in the Great Lakes have stalled.

Right now if you're using the power grid near the Great Lakes, you're dumping mercury in the water.

But a core WattTime value is choice. Whether it's mercury, carbon dioxide, or any other pollutant, we believe nobody should be allowed to make you pollute without your consent.

So in collaboration with this powerhouse team of leading names in environmental activism and technology, WattTime will be developing and deploying technology to make it possible for people in the area to tune their smart devices to “just say no” to drawing power from the dirtiest mercury-spewing coal plants. You can read more about the project here.

Would your smart home or smart building technology company like to showcase your eco-friendly credentials and join our pilot? There's still time to get involved: contact us to learn more.

WattTime yesterday joined the University of California Carbon Neutrality Summit [link] as one of two featured startups for the Entrepreneurs forum [video]. At the conference, Governor Jerry Brown and UC President Janet Napolitano both spoke about the urgent need for more innovative climate change solutions.

Many speakers focused on Napolitano’s vow to turn the ten UC campuses into “living laboratories” to generate solutions that can be adopted on state, national and global levels.

“Climate change impacts issues as varied as disease management, food security, the preservation of water resources, the stability of fragile governments, and transportation infrastructure,” Napolitano said. “Addressing these challenges, and reducing our carbon footprint, is a moral imperative.” [source]

To address these challenges, the UC Climate Solutions Group presented 10 scalable solutions to move the world towards carbon neutrality. The group, comprised of 50 experts from 10 UC campuses and national laboratories, stressed the moral implications of climate change in the executive summary of their report: "Bending the Curve: Ten scalable solutions for carbon neutrality and climate stability.”

“15 percent of us contribute 60 percent of the pollution. We’re leaving behind a planet of uncertain future for our children, grandchildren and generations unborn,” said Veerabhadran Ramanathan, chair of the UC Climate Solutions Group.

As part of the Entrepreneurs forum, WattTime Executive Director Gavin McCormick spoke in particular of the importance of the UC’s “living laboratories” concept. Panelists agreed that the concept has been invaluable for helping ideas take off by allowing university facilities to be used to test innovative new ideas early on. McCormick remarked that early adoption of new WattTime technologies at UC Merced and UC Berkeley was a crucial factor in WattTime’s ability to calibrate our new carbon saving technology [link] to the daily operating needs of users.

“This is a call to action. We put all of our best minds in California on this — a very formidable force. Nothing less than that is required,” said Brown.

I'm a long-time Harper's Magazine reader. My two favorite features are the insanely hard cryptic crossword, and the Harper's Index. So for your data-digesting pleasure, check out this mini Harper's Index of some of WattTime's favorite facts about building a smart, clean grid!

(more…)