What’s the problem?

WattTime analyzes power grid data in real time from dozens of open data sources. Because the data is used to optimize the behavior of smart devices in real time, it is very important that we always have the most accurate up-to-date data. This means that we have had to tackle a classic engineering problem: building a highly reliable system out of less reliable components.

There are two main sources of unreliability in our data ingestion system. First, any of our incoming data sources can go down for a period of time without warning, creating a gap in our data record. Second, our cluster of worker servers may not run the data scraping jobs for any number of reasons, deepening the potential gaps. You can see how we’re in a tricky predicament of always needing the most accurate data and yet having a number of reasons for something to go wrong.

As the saying goes, if you can’t measure it, you can’t manage it. Today’s post explains how we built a monitoring and alerting system to detect gaps in our data ingestion pipeline using Librato.

Designing the Solution

Here are the primary characteristics we wanted in a monitoring and alerting system:

1. The solution needs to run 24/7. We want to provide our end-users with cleanest available energy, and whether the current energy supply is clean can change every five minutes. If our data is not up-to-date, then the energy supply can change without our knowledge, and we might miss an opportunity to give our users a choice to save carbon.

1. The solution needs to run in a way that's isolated/decoupled from how the data scraping tasks normally get run. We don’t want our monitoring system to be dependent on the system it’s supposed to be monitoring! This means that we’d either have to spin up our own separate monitoring service (and maybe a monitor for the monitor…) or use a reliable third-party SaaS tool.

1. The solution needs to have a way to identify any new gaps in our data as soon as possible, so we can triage and fix the problem before it gets worse. If we’re using a third-party tool, that means we need to give it a way to hook into our data pipeline.

1. The solution needs to have a way to send us alert messages when a problem occurs. We like the workflow of Slack, but email would be ok as a fallback.

1. The solution should make it easy to configure the frequency and thresholds for triggering alerts. Overly noisy alerting systems get ignored, so we thought it was a good idea that any system we implemented didn’t bombarded us with notifications, just sending the important actionable ones.

The WattTime API is currently running on Heroku, a popular cloud platform-as-a-service provider. Heroku has a great ecosystem of high-quality third-party “add-ons” that we can trust to have good uptime (satisfying criterion #1), even if something bad happens on our end (satisfying criterion #2). We decided to start by surveying Heroku’s add-ons to see which ones would help us satisfy our other design criteria: data pipeline integration (#3), Slack integration (#4), and configurability (#5).

After some doc hunting, we established that many add-ons had Slack integration, so that didn’t narrow our solution space very far. Instead, we decided to make our choice primarily based on the mechanism of integrating with our data pipeline. There was a wide range of options here: some add-ons would collect data from us if we printed it to our logs, others would collect data if we raised an exception, etc. Exception-based add-ons would be a great fit for detecting failed requests during data ingestion, but they wouldn’t help us monitor failures in our worker cluster overall. A sufficiently configurable log-based add-on, on the other hand, would be able to send alerts either if a problematic value appeared in the logs, or if the log stream stopped getting updated altogether. If such an add-on existed, it would allow us to meet all five design criteria.

And it does: Librato! Librato is a service for visualizing and creating alerts based on metrics. The Heroku add-on comes preconfigured to read metrics from Heroku log streams. In fact, we were already using Librato to visualize dyno performance metrics that Heroku printed to our log stream—but we hadn’t tried configuring any alerts. Reading Librato’s docs on alerts, we discovered that it supports both kinds of alert triggers we wanted. This made it a great choice for a monitoring service that we can configure to watch our data and alert us if something unexpected happens.

Step 1: Logging data quality

Librato is organized around the concept of a “metric.” A Librato metric can be any kind of time series data: something that can be graphed with a time stamp on the x axis and a number on the y axis. When the metric hasn’t been reporting or hasn’t logged a new datapoint within a certain period of time, visualizations like this can help us clearly see what is going on.

We chose “lag time” as the metric to track. We define lag time as the age of the most recently ingested data point of a particular type. Lag time serves as a good metric because it helps us get to the heart of the problem. If our goal is to have the most accurate up-to-date data, we’d like to know when our gaps start and for how long they occurred for.

To implement data quality logging, every time we collect data, we figure out how old the newest data point is, and print that number of minutes to the logs in Librato’s specific format. Here’s a screenshot of what the output looks like in our Papertrail logs:

Step 2: Setting up alerts

We have several ways to detect if something has gone wrong with our data collecting. If a new datapoint has not been added within an acceptable amount of time, then we may want to get alerted and see if there is anything we can do on our end. To do this, we set “condition type” to “goes above” when creating an alert:

We can also check if our metric stops reporting entirely. In that case, we set “condition type” to “stops reporting”. Both alerts serve to inform us as quickly as possible if something were to go wrong.

Step 3: Receiving alerts

To set up Slack integration, Librato requires you retrieving the Webhook URL for your Slack channel. In the Slack desktop app, go to the top left and click on Apps & Integration. This will take you to a directory of different apps that can integrate with Slack. Search for "Incoming WebHooks."

From there, go to "add configurations" and you should be able to find the webhook URL.

Once you have the URL, you can go to Librato’s Integrations tab and click “add configuration” on the Slack tab. Paste in the URL and give it a title you’ll be familiar with.

Then whenever an alert is triggered, you’ll receive a Slack message showing you why the alert was triggered and other relevant information. 

What we're thinking about next

While it’s really nice for our system to alert us whenever anything goes wrong, we thought that maybe it would be even better practice if our system was self-healing. As it stands, our system sends us an alert, and from there a living breathing human being has to take time out of what they’re doing and investigate. So for our next step, we will create a system to patch up data holes as they happen. In this way we can be more sure that we have the most accurate up-to-date data at any time.

Do you manage a large commercial or industrial facility? Perhaps a university, apartment complex, or hotel? Today we're pleased to share that, thanks to WattTime's newest partnership with Building Clouds, you can now cut your emissions through WattTime-enabled technology in nearly any equipment.

Facility managers everywhere have learned that not all equipment manufacturers play nice with each other and that interoperability is a major problem. What we love about Building Clouds Strati-Fi(TM) controllers is that they allow for remarkably quick and cost effective monitoring and control of most equipment types in commercial or industrial scale HVAC equipment. So whatever your building currently runs on, you can now WattTime-enable it by giving Building Clouds a call.

We've been working with Building Clouds for over a year to pilot this technology and work out all the kinks. Our first project began with Sutardja Hall, which became the first building in the world to optimize its HVAC package's energy loads in real time to cut carbon emissions. Building Clouds President Bob Wallace met with us on the rooftop of Sutardja Dai Hall and demonstrated the technology installation, which took place in under an hour. We were able to collect data almost immediately and automatically started cutting carbon less than a week later.

The implementation and results of this went over so well with UC Berkeley that we recently agreed upon a second project for the Residential and Student Services Building (RSSB). This deal included connecting the Strati-Fi(TM) controllers to WattTime-enable two sixty-ton air handlers. Both projects have successfully achieved absolutely zero increase in the building's energy bill or on occupant comfort. In fact, as a mischievous test, we did not announce the project until one month after installation, during which no building occupants were even aware of any change.

After over a year of successful pilots, we're delighted to share that WattTime and Building Clouds are scaling up our partnership to release our technology to the broader market. If you're looking for cost-effective building automation solutions that also cut emissions, please take a look at buildingclouds.com or WattTime's own Shop page.

We are thrilled to announce that WattTime has received a substantial grant from the Great Lakes Protection Fund to lead a coalition of nonprofits and companies in reducing mercury pollution from coal plants. The project will be a collaboration between WattTime, Rocky Mountain Institute, National Wildlife Federation, Delta Institute, Energy Emissions Intelligence, and several corporate partners.

In 2008, a Federal court struck down the national Clean Air Mercury Rule that required coal-fired power plants to limit dangerous mercury emissions. With repeated attempts to replace the rule continuing to face uncertain political futures, badly-needed efforts to return to safe mercury levels in the Great Lakes have stalled.

Right now if you're using the power grid near the Great Lakes, you're dumping mercury in the water.

But a core WattTime value is choice. Whether it's mercury, carbon dioxide, or any other pollutant, we believe nobody should be allowed to make you pollute without your consent.

So in collaboration with this powerhouse team of leading names in environmental activism and technology, WattTime will be developing and deploying technology to make it possible for people in the area to tune their smart devices to “just say no” to drawing power from the dirtiest mercury-spewing coal plants. You can read more about the project here.

Would your smart home or smart building technology company like to showcase your eco-friendly credentials and join our pilot? There's still time to get involved: contact us to learn more.

WattTime yesterday joined the University of California Carbon Neutrality Summit [link] as one of two featured startups for the Entrepreneurs forum [video]. At the conference, Governor Jerry Brown and UC President Janet Napolitano both spoke about the urgent need for more innovative climate change solutions.

Many speakers focused on Napolitano’s vow to turn the ten UC campuses into “living laboratories” to generate solutions that can be adopted on state, national and global levels.

“Climate change impacts issues as varied as disease management, food security, the preservation of water resources, the stability of fragile governments, and transportation infrastructure,” Napolitano said. “Addressing these challenges, and reducing our carbon footprint, is a moral imperative.” [source]

To address these challenges, the UC Climate Solutions Group presented 10 scalable solutions to move the world towards carbon neutrality. The group, comprised of 50 experts from 10 UC campuses and national laboratories, stressed the moral implications of climate change in the executive summary of their report: "Bending the Curve: Ten scalable solutions for carbon neutrality and climate stability.”

“15 percent of us contribute 60 percent of the pollution. We’re leaving behind a planet of uncertain future for our children, grandchildren and generations unborn,” said Veerabhadran Ramanathan, chair of the UC Climate Solutions Group.

As part of the Entrepreneurs forum, WattTime Executive Director Gavin McCormick spoke in particular of the importance of the UC’s “living laboratories” concept. Panelists agreed that the concept has been invaluable for helping ideas take off by allowing university facilities to be used to test innovative new ideas early on. McCormick remarked that early adoption of new WattTime technologies at UC Merced and UC Berkeley was a crucial factor in WattTime’s ability to calibrate our new carbon saving technology [link] to the daily operating needs of users.

“This is a call to action. We put all of our best minds in California on this — a very formidable force. Nothing less than that is required,” said Brown.

I'm a long-time Harper's Magazine reader. My two favorite features are the insanely hard cryptic crossword, and the Harper's Index. So for your data-digesting pleasure, check out this mini Harper's Index of some of WattTime's favorite facts about building a smart, clean grid!

(more…)

For the past 2 weeks, WattTime has been gearing up for two exciting Demo Days. In this blog post, we highlight our team’s experiences presenting at both BlackRock and Google.org, sponsors of Fast Forward’s 2015 summer accelerator.

BlackRock is the world’s largest investment firm, with $225 billion in mandates that explicitly address social, ethical or environmental considerations (link). BlackRock is also known for their philanthropy, supporting 20,000 young people all over the world for higher education and 1,942 nonprofit organizations through their programs (link).

We put a lot of preparation into this presentation, and are very thankful to the Fast Forward’s team, who helped us so much with advice, connections, and resources along the way.

Our cofounders Gavin McCormick and Dr. Anna Schneider had a blast practicing their one-liners, conducting interviews and finally presenting in front of BlackRock employees. When asked, “how would you summarize your experience at BlackRock’s Demo Day?”, Gavin says:

"One thing that was very fun was getting to pitch WattTime to a very finance-savvy crowd. Our technology can frankly be pretty complicated! But the folks at BlackRock immediately got it, and even started talking about linear optimization models and the connection to Enron. One even referred to us as the "Enron for good", which might be my new favorite tag line."

The center for Google’s philanthropy initiatives, Google.org proclaims “technology for social impact”, a strong initiative to use technology to make the world a better place. Each year, Google.org donates $100,000,000 in grants, 80,000 volunteer hours and $1 billion worth in products to nonprofit organizations doing good in their communities.

One of the biggest achievements was on the morning of Google’s Demo Day, Cofounder and CTO Dr. Anna Schneider and our analysis fellow Sohum had a breakthrough with WattTime’s Impact API. After years of R&D, we’ve finally expanded our real-time power grid analysis from just the largest power grids to every single grid in the continental United States. Get a sneak preview of the map here!

As part of the pitch event on Wednesday night, Google.org announced that they would be giving each of the 9 Fast Forward teams $30,000 AND matching donations from all other gifts that night. What a generous contribution!

When asked how he felt about this announcement, Gavin says:

“This is an amazing offer from Google.org and we are so grateful for their generosity. It was the cherry on top because we received a call from an anonymous donor that same day, also matching donations given that night. Which means, October 7 was an incredibly important night for us, with donations matched triple!”

After a smooth stream of presentations, we asked our team how they felt about the night and WattTime’s progress.

Dr. Anna Schneider, Cofounder and CTO: "Watching everyone pitch on stage, I felt so proud to be included in this cohort of wonderful humans. I've seen every single team reach milestones, clarify their vision, and gain confidence in ways that matter to them."

"Mentors from everywhere from Facebook to BlackRock to TechSoup came up to tell me how excited they were about our progress since we'd met a few short weeks ago. It feels like our team's hard work is making a real difference!"

Sho Kawano, Business Development: "The Google Demo Day gave me renewed enthusiasm for the potential for tech-nonprofits to enact change on the scale that we yearn for in the nonprofit sector."

Theresa Zhang, Communications: “It was amazing and inspiring to see so much support and enthusiasm for tech nonprofits like us. All in all, an amazing night!”

Breaking the nonprofit lifecycle into the typical "early" and "late" stages elides a vital stage—the "exit" stage. Is your org intending to exist forever, or is it intending to lead a non-eternal life?

Path away from profit

Pose this question to the founder of any for-profit startup in the venture pipeline, and they'll know their answer without hesitation. Either they plan to get acquired or they plan to IPO. Everyone gets rich, everyone is happy. Pretty simple, right?

At WattTime, thinking carefully about exit strategy was a key driver for us to forgo the for-profit model. We talked to a number of VCs and angel investors in the early months, and the more we reflected on their feedback, the more it became clear that our vision was incompatible with a for-profit exit.

To target IPO, investors want to see high barriers to entry for competitors. While we think it's important to protect our IP for now, we feel it would be impossible (and highly counterproductive!) to shut everyone else out in the long run.

On the other hand, to target acquisition by any of the big players in our field, we would have had to make our peace with ultimately being locked into that single corporate entity. With the right acquirer, we would create scalable impact by serving internal partners—but the ultimate scope of our impact would be limited in the likely scenario that we were blocked from forming meaningful external partnerships.

To put it bluntly: pursuing a for-profit exit might scale our business, but it would actually be a pretty crappy way to scale our idea.

So, what's our endgame instead?

For a nonprofit, there are many more exit strategies that are open to you, and often much less clarity about which strategy is the best fit. This SSIR piece from the Global Development Incubator makes a strong case for thinking as carefully about exit strategy as for-profits do, and lays out some options to get you started:

Plotting an Endgame: Six Options. Excerpted directly from "What's your endgame?", SSIR, 2015.

WattTime's intended strategy is a mix of these. In fact, if we divide our work into "research," "education," and "assistance" programs, each program falls into a different exit strategy.

• Our research program is a classic example of the "open source" path. Environmental demand response is a "breakthrough idea," and we are well equipped to conduct and disseminate the foundational research to place the field on firm scientific footing and promote it towards maximum impact.
• Our education program follows the "replication" path by operating pilot projects that lay the groundwork for commercial adoption, with the ultimate goal of evolving into a certification program that keeps for-profit players in check while providing them with highly-desirable branding opportunities.
• Our assistance program follows the "commercial adoption" path by helping for-profits incorporate environmental demand response into their existing products, from electric vehicles to commercial HVAC. In many cases, this looks like providing a software API that links our data and algorithms to their consumer-facing products.

Although the article uses the phrase "pathologically cooperative" to describe the open source model, we live that value across our programs, and probably most actively in the assistance program. Interestingly, this means that our bias toward cooperation is strongest in the area of our work that would be most strongly competitive if we were a for-profit!

In the very latest stages, we hope to follow a variant of the "government adoption" path. Government can "acquire" most of our activities by folding them up into one of the existing Department of Energy national labs, which already have deep expertise in scientific research, pre-commercial pilot projects, commercial assistance, and certification programs. (In fact, we considered moving under the umbrella of a national lab early on, but hoped we could create impact faster by staying small and nimble in our early stages.) Government has a weaker track record for maintaining delightful software that companies are happy to build their products on top of, but it's not unprecedented, and efforts like the US Digital Service are paving the way to improvement on this front.

There's one key role of government that's missing from the article but present in our long-term strategy: regulation. If legislatures, DOE, or public utility commissions mandate corporations to participate in our commercial adoption program, then that's a huge step toward mission achievement for us. Like I mentioned above, many companies are perfectly capable of copying our activities once they have an incentive to do so, and regulation can be an effective incentive. So government regulation could be the most impactful outcome of our work, while also putting us out of business for good. I suspect this is true of many other nonprofits, and I'm surprised it wasn't explicitly included in the SSIR piece.

Putting it into practice

Once you've clarified your nonprofit endgame, there's of course the little matter of making it useful.

Knowing WattTime's exit plan has given us a great answer to that perennial question, "why aren't you a for-profit?" Acting like a scalable Silicon Valley for-profit would be inconsistent with our values, in large part because we think it would lead to a smaller scale in the long run.

Another benefit is giving us a framework for thinking about how scalable we should make different arms of our business. For instance, remembering that our ultimate goal is to enable others to replicate our model helps us resist over-investing our limited resources in any one pilot project. But the core R&D—that should stay in known hands (like our own) to build a trusted brand for the eventual certification program.

Of course, it's not always easy to be growing something you care about to the depths of your being, yet planning its demise at the same time. Keeping a mental distance between the organization and its vision helps with this. So too does remembering that your mission is to actually create real change—and if that change means that your org winds down, you and your colleagues can free up your energy to tackle the next big problem.

The take-home

• Are you leading a baby for-profit? Think carefully about the implications of pursuing acquisition or IPO, and make sure that the benefits outweigh any compromises you'll have to make before or after your exit.
• Are you leading a baby non-profit? Read this article and build one or more endgames into your theory of change from the beginning.
• WattTime's endgame is a mix of a few nonprofit exit strategies, but "sustained service" is not one of them. If all goes well, our impact will continue far beyond the lifetime of the organization.
• Knowing our exit strategy has helped us stand firm on our values, and will help us allocate resources moving forward.