How accurate are Station A's electricity consumption and load estimates?

Today, our load estimates are accurate enough to be useful for sorting buildings and for understanding the value of particular clean energy resource at a given location.

Note: we have made significant improvements to our platform and building assessment since performing the accuracy analysis laid out below. These improvements include identifying the 3D size of a building, accounting for the building height. Along with these improvements, we are actively accumulating more actual load data to provide a more meaningful update to the analysis below. Please feel free to reach out to us directly to discuss.

Context

We estimate electricity consumption and peak load for all ~1,000,000 buildings in our index in order to help our users evaluate the clean energy options in seconds, without requiring any upfront data.

In order to evaluate our accuracy, we compare these estimates to real 2015 load data for hundreds of customers, primarily in the PJM territory. While we estimate an hourly 8760 time series for each building, we judge our accuracy based on two more high-level statistics:

  1. Total annual usage
  2. Peak load (the maximum hourly reading for the year)

We estimate absolute percentage error, i.e. if the real peak load for a building is 100 kW, an estimate of 80 kW or 120 kW is recorded as a 20% error. For a group of buildings, we show the median error.

We also estimate bias, a measure of whether we tend to overestimate or underestimate over an aggregate sample, and a good metric to gauge how our estimates will look for groups of buildings. For a group of buildings, to calculate bias:

  1. We calculate the sum of all the predicted values minus the sum of all the actual values to get the total difference
  2. We divide the total different by the sum of all the actual values to get our bias on a percentage basis

To give an example, if a group of buildings actually uses 10 million kWhs of electricity in a year, and we estimate that they collectively use 10.5 million kWhs, then our estimates have a +5% bias. Conversely, if we estimate that they use 9.5 million kWhs, then we have a -5% bias.

We should caveat all of these results by the fact that our test set is not a random sample. It is drawn from actual data that we have accumulated from our various users. As a result, it is non-random in several ways:

  • It is all from the Northeastern US
  • The buildings tend to be larger and more industrial than our building data at large
  • All data is from the year 2015, which could have had non-representative weather

Please keep all of these caveats in mind as we read the results. We are actively accumulating more actual load data from our users and will use this for benchmarking and improving our load estimates going forward.

Accuracy Today

Generally speaking, on average, we tend to underestimate electricity usage. We consider this to be conservative and better than overestimating, since greater electricity usage generally implies better project economics.

When evaluating accuracy, we break down the test set by the NAICS business classification of the occupant. Looking at these categories on their own shows that estimates are meaningfully more and less accurate for different types of buildings.

We typically meet the threshold our users have characterized as accurate enough (greater than 50% accuracy) to compare buildings and share results with the buyer for three large categories of buildings in the test set:

 

Category Test Set (n) AU Error PL Error AU Bias
Home Centers 157 16% 34% -14%
Schools 60 28% 28% -12%
Hospitals 8 48% 46% -53%

 

  • AU: annual usage
  • PL: peak load
  • Home Centers: home improvement and big box stores such as Lowes, Home Depot, and Kohls.

There are two factors that may account for our good estimates for these types of buildings:

  1. A critical step in our load estimation is mapping buildings to one of our internal industry type models. These three categories of buildings are easy for us to map. Home centers are clearly "stand-alone retail," hospitals are clearly "hospital," and elementary and secondary schools clearly map to either "primary school" or "secondary school."

  2. These building types tend to be home to single-story buildings, or at least buildings with a few stories. We currently assume all buildings are single story, leading us to consistently underestimate the load of buildings with tall buildings on-site, because we are not able to get number of story information about buildings at this time.*

    *Please note, Station A now accounts for the 3D size and shape of a building and no longer assumes all buildings are single story. While we have not re-run the accuracy analysis yet, this product improvement significantly increases the overall accuracy of our load estimates and the figures represented above.

Value Today

Today, our load estimates are accurate enough to be useful for sorting and parsing buildings and for running simulations to understand the value of particular clean energy resources at a given location. Load estimates are more accurate for groups of buildings, though we persistently underestimate; the accuracy for a group of buildings converges to the bias as the number of buildings grows (-34% of peak load, and -46% of annual usage).

Short-term Roadmap of Improvements

Improving load estimation is one of the top three priorities on our analytics roadmap. As mentioned in the note above, we have already improved them significantly by implementing a change to account for multi-story buildings by estimating stories for all buildings. Other improvements on our roadmap include:

  • Improved building classification: classifying mixed-use building to multiple types, with a percentage allocation for each type, and including more industry types
  • Collecting real data and training: collecting more real data, allowing us to experiment with statistical models rather than physics-based models

By implementing this roadmap, we believe we can improve our accuracy significantly and quickly.

We would also like to improve our ability to evaluate our accuracy over time. As we collect more data, and data from a broader, more representative sample of buildings, we can become more confident in the metrics shown above.

Eventually, we would like to move from focusing purely on load estimation accuracy to focusing on the accuracy of the savings estimates that we derive from our load estimates. This will allow us to focus on how much inaccuracies impact or don't impact the calculations that truly drive energy consumer sorting. It will also allow us to measure the impact of every characteristic of the load profile - annual usage and peak load, but also the shape and magnitude throughout the year.