Methodology + Validation

WattTime has built a marginal emissions model based on empirical techniques published in the peer-reviewed academic literature.

A 3D icon shows a trio of data visualizations in charts and graphs.

What matters most? Authentic impact.

WattTime’s purpose is to help others solve climate change as fast as possible. We produce data that enable all the good work that others are doing to go farther and faster than it otherwise would. The foundation of making more impactful decisions is to measure the truly causal impacts of those decisions. Our data are not meant to simply be included in a report once a year, they are used by organizations that want to maximize the authentic impact per dollar they invest. If every project or initiative is made twice as effective by using better data, that could mean we achieve the goal twice as fast.

Our approach to methodology + validation

For an emissions model to achieve its desired purpose, accuracy alone is not enough. Different types of emissions rates apply to different questions being asked, so first and foremost, the appropriate rate should be applied for a particular use case. The data being used should apply to the relevant location and to the time period during which an intervention will be deployed.

WattTime publishes new marginal emissions data every 5 minutes, including a forecast of the next 72 hours, because stale data results in lower effectiveness. We advise our partners about the best available data for each individual use case, whether or not the data are our own. We invest heavily in not only improving our modeling techniques but also validating that the use of our models produces real emissions reductions. We collaborate with other electricity grid and emissions scientists, grid operators, and technology organizations to advance the science of marginal emissions. In places around the world that are lacking in publicly available emissions data, we figure out ways to measure it ourselves.

To learn more about our approach to methodology & validation and our goals for future research, see our paper from 2022.

WattTime has formed a working group to advance the science of marginal emissions validation, called VERACI-T. Learn more here.

Methodology

WattTime research produces data that answer the question, “What is the change in emissions caused by a change in electricity use (or generation) at a particular time and place?” Academic research refers to this type of data as marginal emissions rates.

How does WattTime calculate marginal emissions?

WattTime staff over the past decade have extended the basic techniques in peer-reviewed published literature on estimating marginal electricity emissions, including this peer-reviewed literature co-authored by our founder Gavin McCormick. This technique is a specific instance of a widely accepted class of marginal emissions analysis models used in numerous peer-reviewed articles. A selection of these papers are included below for reference.

As our research has produced many iterative advancements, we have kept a focus on the fundamental reliability of causal, empirical modeling, while also making them more useful for active emissions reductions and controlling devices in real-time.

Please contact us for a more detailed step-by-step description of our latest modeling methodology, which we can make available upon request.

Using an empirical model

The fundamental approach of marginal emissions models in the literature is very similar: they start with raw electricity generation and emissions data available from the US EPA at an hourly frequency by every major fossil fuel-fired power plant in the United States (via continuous emissions monitoring system, CEMS). A similar approach applies in other countries where data are available. Each then applies regression-based modeling to ask, every time a rise or fall in electricity demand occurs in a given place and time, which power plants actually increase or decrease their output in response? This allows a modeler to compare how marginal emissions rates vary by time and place. For example, by running such regressions separately for day and night periods, such models can compare the marginal emissions caused by using electricity during the day or during the night.

WattTime believes strongly in such empirical modeling techniques because they are driven almost purely by the data and require almost no assumptions. This is part of a growing trend in economics and social sciences towards increasingly questioning the validity of assumption-driven models, and replacing them instead with almost purely data-driven causal models, a trend known as the “credibility revolution”. The most notable proponent of this approach, Esther Duflo of MIT, won the 2019 Nobel prize in economics for this innovation.

WattTime staff have run numerous tests of how much Esther Duflo’s fundamental insight of questioning assumption-driven models applies to the energy sector and shown that most basic assumptions often fail to hold up in the real world. These and other insights have deepened the WattTime team’s conviction that empirical models are essential for accurate marginal emissions analysis.

Improving on the published literature

WattTime has continuously improved our modeling techniques rooted in literature while keeping the fundamental reliability of causal, empirical modeling as a foundation. The established empirical approach above works well for historical datasets. We’ve also made them more useful for applications that use the data to reduce emissions, such as controlling devices in real time.

For example, it is one thing to say emissions are cleaner during the day than at night, but quite another to measure slight variations every five minutes. So for years, WattTime’s models have leveraged real-time power grid data from individual grid operators as well as the Energy Information Administration to detect very fine-grained changes in power grid behavior that are highly predictive of patterns when marginal emissions are higher or lower. Furthermore, because CEMS only measures fossil fuel plants, standard models cannot detect the increasingly common moments when non-fossil (e.g. solar) plants are marginal. So, WattTime integrated additional datasets that measure moments of renewable energy curtailment, allowing us to build predictive models for when curtailment is occurring.

No model is perfect, and WattTime is continuously researching new modeling techniques and methods for validating them. Our goal is to constantly improve the efficacy and feature set of such models to enable ever-greater emissions reductions.

Input data sources

Our empirical modeling is highly dependent on the availability of relevant data, which has so far been exclusively from publicly available sources. Where possible, our models are derived from historical time series data including:
  • Power plant emissions and generation data: For example, US EPA’s Continuous Emissions Monitoring System (CEMS) reports hourly data from the CAMPD database.
  • Demand, interchange, and generation by fuel type: e.g., from the US Energy Information Administration (EIA) or European Network of Transmission System Operators for Electricity (ENTSO-E).
  • Energy prices (LMPs) & curtailment volumes (observed and forecasted): reported by grid operators such as RTO/ISOs.
  • Global power plant locations and fuel types: e.g., from data gathered and aggregated by Climate TRACE. We use established methodologies for estimating the carbon intensity of power plants based on known factors when ground truth data are unavailable.
  • Weather data: obtained from Apple WeatherKit and sometimes others.

Model quality + hierarchy

Thus far, our priority has been to produce models of the highest quality while expanding our coverage into regions that have adequate publicly available data to produce these high-quality models. We’ve recently begun running out of places to expand that also have complete and public datasets.

In many electricity grids around the world, not all of the data above are made publicly available by grid operators or governments. Nevertheless, there is still great opportunity for emissions reduction interventions in places without public data, and technology companies that are capable and eager to deploy those interventions in those locations.

For regions that lack complete public data, WattTime has developed alternative methods of estimation to make the most of the data that are available. Model accuracy suffers as we rely on less endogenously reported and directly measured information, however, we aim to produce a marginal emissions signal that is still effective at producing meaningful emissions reductions from load flexibility and/or renewables siting. The result is a hierarchy of models that allows us to continue expanding the reach of our impact to every corner of the globe.
  1. Binned Regression: This causal model creates a tree of regressions conditioned on grid operating conditions. These models include factors such as interchange with neighboring balancing authorities, emissions associated with charging pumped hydro storage, and renewable curtailment where applicable. This is considered WattTime's highest-quality methodology for the MOER signal type.
  2. Average Interchange: In the grid regions of British Columbia (BCHYDRO) and Quebec (HQ), abundant hydropower resources mean that the internal marginal fuel source is nearly always reservoir-based hydropower. Both of these regions are significant net-exporters to other grid regions; BCHYDRO exports to AESO and the US Northwest, while HQ exports to IESO and NYISO. Because of this interchange, and the fact that additional demand from interchange cannot indefinitely cause additional hydropower generation (as reservoirs are limited in their capacities) marginal consumption in either BCHYDRO or HQ restricts the possible export of hydropower generation. To represent this causal chain, our MOER model for these regions is representative of a blend of the MOERs in regions to which they export.
  3. Proxy Regression: This model is used in regions where we have access to some real-time and forecasted information about the status of the grid (such as demand or energy prices), but are lacking specific ground truth time series data such as emissions or generation by fuel type required to train a binned regression model. To estimate a MOER without these data, we use machine learning clustering algorithms to find grids (present or past) with similar characteristics to the grid of interest. Once we find these cluster ‘matches' to be used as proxies, we regress their demand against generation from each fossil fuel type for a number of quantile bins. We then use location-specific power plant data (obtained through Climate TRACE) for power plants within the grid of interest to estimate carbon intensities by fuel type. The MOER model is then based on the relationship between marginal demand (reported in real time), and economical dispatch assuming a similar supply curve as the proxy regions. This model produces an inherently less accurate MOER than a binned regression model trained on endogenous data, however, it still produces a signal useful for both carbon-aware load shifting and renewables siting. This is considered a medium-quality signal, and we will look to upgrade these models to higher-quality binned regression methodologies as data transparency increases.

Forecasts

The efficacy of avoiding carbon emissions by load shifting can be greatly improved with real-time estimates of the emissions impact of those shifts, instead of using historical averages. Even better performance is achieved when using a forecast to rank all of the intervals within the flexibility horizon and scheduling consumption during the lowest impact intervals. WattTime has been publishing forecasts of marginal emissions rates since 2019.

How does your forecast model work?
We currently produce forecasts of CO2 MOER and Health Damage with a horizon of 72 hours. WattTime uses an ensemble of ML models trained on the historical MOER signal. There are multiple features in that model, including ARIMA-style features and exogenous features. And we directly forecast each time horizon.

How accurate is your forecast?
We forecast performance primarily using an efficacy metric, in addition to expected rank correlation and mean absolute error. We measure efficacy by comparing the emissions reduction performance of load-shifting using (a) our forecast and (b) perfect information (knowing the emissions rates in advance). This performance efficacy metric helps us gauge how well we are capturing the carbon reduction opportunity when load is scheduled based on the forecast. This is substantially a function of predicting the rank order of MOER in upcoming time intervals (predicting the timing of the peaks and valleys is the most important aspect of the forecast for this use case).

Forecast efficacy varies by grid region and is heavily dependent on the generating resource mix and unpredictability of marginal emissions rate. Wind-dominated regions (i.e. the great plains of the US) tend to be harder to predict than solar-dominated regions (i.e. California). Most grid regions have a forecast efficacy of 50-90% carbon reduction compared to a perfect forecast. The MAE for most regions falls between 1% to 9%.

Validation

Marginal emissions are not directly measurable, so validation is very important to ensure that our research progress translates to more effective real-world carbon reductions. Without ground truth, it can be challenging to determine which models are closer to the truth and quantify their accuracy.

WattTime welcomes peer review of our methods and has continued to publish literature on our methods. In 2017, RMI conducted further validation of WattTime’s methods and their application to automated emissions reduction. Many of our partners that build technology features that rely on our data have performed their own evaluations of our data; some of those partners have formed coalitions to advocate for the use of marginal emissions in GHG accounting standards.

WattTime doesn’t only validate our own models, we’ve started collaborating with other organizations to evaluate their own emissions datasets and to share best practices. The working group that WattTime convened is called VERACI-T (Validating Emissions Rates for Accurate Consequential Impact Taskforce). By learning from other organizations modeling marginal emissions and other electric grid phenomena, we hope to more rapidly advance the techniques for validating marginal emissions models and improve the accuracy of the models, in order to drive greater impact.

Academic references

The method for determining marginal emissions rates builds on the techniques in these academic papers, including one written by our founder, Gavin McCormick.

Featured resources

Energy Institute at Haas logo

Location, Location, Location: The Variable Value of Renewable Energy and Demand-Side Efficiency Resources

by Duncan S. Callaway, Meredith Fowlie, and Gavin McCormick
READ MORE
Environmental Science and Technology logo

Marginal Emissions Factors for the U.S. Electricity System

by Kyle Siler-Evans, Ines Lima Azevedo, and M. Granger Morgan
READ MORE

All resources

Environmental Science & Technology

"How well do emission factors approximate emission changes from electricity system models?" (2022)

Electric Power Systems Research

"Using geographic load shifting to reduce carbon emissions" (2022)

Environmental Science & Technology

"Current and future estimates of marginal emission factors for Indian power generation" (2022)

PNAS

"Why marginal CO2 emissions are not decreasing for US electricity: Estimates and implications for climate policy" (2022)

EYP Mission Critical Facilities

"Demand response opportunities for data center embedded generation and energy storage systems" (2021)

The Electricity Journal

"Using marginal emission rates to optimize investment in carbon dioxide displacement technologies" (2021)

Microsoft

"Made to measure: sustainability commitment progress and updates" (2021)

Association for Computing Machinery

"A guide to reducing carbon emissions through data center geographical load shifting" (2021)

Advanced Science

"Green algorithms: quantifying the carbon footprint of computation" (2021)

SSRN: Social Science Research Network

"(Not so) clean peak energy standards" (2020)

Proceedings of the 8th International Conference on Smart Cities and Green ICT Systems

"On the use of average versus marginal emission factors" (2019)

Environmental Science & Technology

"Marginal emission factors considering renewables: a case study of the U.S. Midcontinent Independent System Operator (MISO) System" (2017)

RMI

"WattTime Validation and Technology Primer" (2017)

Journal of Cleaner Production

"Consideration of marginal electricity in real-time minimization of distributed data centre emissions" (2016)

Environmental Science & Technology

"Comparative assessment of models and methods to calculate grid electricity emissions" (2016)

Energy Institute at Haas

"Location, Location, Location: The Variable Value of Renewable Energy and Demand-Side Efficiency Resources" (2015)

Environmental Science & Technology

"Bulk energy storage increases United States electricity system emissions" (2015)

Journal of Economic Behavior & Organization

"Spatial and Temporal Heterogeneity of Marginal Emissions: Implications for Electric Cars and Other Electricity-Shifting Policies" (2014)

Applied Energy

"Evaluation of a rapid LMP-based approach for calculating marginal unit emissions" (2013)

Environmental Science and Technology

"Marginal Emissions Factors for the U.S. Electricity System" (2012)

Handbook of CO2 in Power Systems

"Locational carbon footprint of the power industry: implications for operations, planning and policy making" (2012)

Energy Policy

"Estimating marginal CO2 emissions rates for national electricity systems" (2010)

European Commission

"Quantification of the effects on greenhouse gas emissions of policies and measures" (2009)

Applications in research

Many other researchers have used marginal emissions data produced by WattTime in their research. Below are those we’re aware of. Please reach out to us if you’d like to use WattTime data in your research.

Featured resources

UC Berkeley logo

Automated Demand Response Refrigerator Project

by Jessica Tran et al.
READ MORE
Applied Energy Journal logo

Site demonstration and performance evaluation of MPC for a large chiller plant with TES for renewable energy integration and grid decarbonization

by Donghun Kim et al.
READ MORE

All resources

Green Software Foundation

"Measurement for the Software Carbon Intensity (SCI) specification" (2023)

IEEE Transactions on Power Systems

"Carbon-aware computing for datacenters" (2023)

Sustainable Energy Technologies and Assessments

"Do ‘green’ data centres really have zero CO2 emissions?" (2022)

Applied Energy

"Site demonstration and performance evaluation of MPC for a large chiller plant with TES for renewable energy integration and grid decarbonization" (2022)

Arxiv

"Measuring the Carbon Intensity of AI in Cloud Instances" (2022)

Crypto Climate Accord

"Guidance for accounting and reporting electricity use and carbon emissions from cryptocurrency" (2021)

University of California, Berkeley

"Automated Demand Response Refrigerator Project" (2015)

University of California, Berkeley

"Optimal Refrigeration Control For Soda Vending Machines" (2015)

Journal of Economic Behavior & Organization

"Spatial and temporal heterogeneity of marginal emissions: Implications for electric cars and other electricity-shifting policies" (2014)