Energy model calibration - normalizing the utility bills to month start-end

29 posts / 0 new
Last post

When calibrating an energy model to utility bills the utility bills often don?t align with the month start and end. I have reviewed a couple methods to calendar normalize the utility bills but find them somewhat unsatisfactory.

For example the method I am looking at does the following:
The April gas bill runs from March 25 ? April 24. The algorithm takes the average number of m3 per day from that bill, applies it to the days in April. Then it takes the average number of days from the May bill which runs from April 24 ? May 25 and applies that average to the remaining days in April.

The issue is that the March-April period has much higher HDD than the April-May period and the ?normalized? gas usage is significantly lower than the simulation data for April.

I am wondering if there are any papers or other sources of information as to how others approach this problem.

[cid:image003.png at 01D09C46.E75BA0D0]
Christopher Jones, P.Eng.
Senior Engineer

WSP Canada Inc.
2300 Yonge Street, Suite 2300
Toronto, ON M4P 1E4
T +1 416-644-4226
F +1 416-487-9766
C +1 416-697-0065

www.wspgroup.com

________________________________

You are receiving this communication because you are listed as a current WSP contact. Should you have any questions regarding WSP?s electronic communications policy, please consult our Anti-Spam Commitment www.wspgroup.com/casl. For any concern or if you believe you should not be receiving this message, please forward this message to us at caslcompliance at wspgroup.com so that we can promptly address your request. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information which is privileged, confidential, proprietary or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying or in any way using this message. If you have received this communication in error, please notify the sender, and destroy and delete any copies you may have received.

WSP provides professional land surveying services through the following entities: WSP Surveys (AB) Limited Partnership and WSP Surveys (BC) Limited Partnership

Jones, Christopher's picture
Joined: 2015-06-11
Reputation: 0

Hello Chris,

ASHRAE Guideline 14-2014 has lots to say about this. So does the ASHRAE PMP, including lots and lots of references. In general, there are several things going on that can add noise to the answer.

First, for best results you should use measured weather data that has been recorded for the same period as the utility bills. Unfortunately, although this is easy to say in principle it can be difficult to apply. For example, not all weather sources have measured data for all channels, and worse, don't tell you when they have used an algorithm to "fill in" for missing data or to synthesize a channel that was never there. In addition, if your utility billing period runs over two different years, then you have to split the simulation since most simulations run from Jan to Dec, don't run contiguously for non-calendar periods (i.e., September to August). Weather data files also need to be consistent with the billing periods for non-contiguous periods. Use of average weather files to match actual utility bills is not recommended as this can introduce more error that you're trying to measure.

Second, since you have the simulation, I'd suggest you extract the hourly data from the simulation to match the calendar periods of the measured data. Then both can be "day normalized", or expressed as monthly average daily energy use (i.e., divide by the days in the month), and then weight the average by multiplying back by the number of days....since a month with 32 days should carry more weight than a month with 28 days when regressing...etc.

Third, don't use HDD or CDD to do anything. This is because commercial buildings don't normally behave according to 65 F change points. Instead, use the ASHRAE Inverse Model Toolkit, which comes free with Guideline 14, including the FORTRAN source code (...yes...FORTRAN). The IMT performs (1P, 2P, 3p,4P 5P) change-point linear and variable-based degree day calculations, depending on the what works best. This will do the weather normalization for you and give results that are consistent with Guideline 14. In fact, there's even a tutorial on how to use it in the appendix to Guideline 14 and you can get the RP 1050 final report to read more, including looking at the test data sets.

If you do use someone's software to do this, look for "algorithms by ASHRAE" in the software manual to make sure they are consistent with ASHRAE Guideline 14. That's why ASHRAE publishes the algorithms in source code to encourage software developers to use them providing they include the "powered by ASHRAE" in the manual.

Stay away from polynomials, quadratics, bi-quadratics, etc. as these are hard to reproduce from one person to the next even though the results look good.

Hope this helps.

Jeff

8=! 8=) :=) 8=) ;=) 8=) 8=( 8=) 8=() 8=) 8=| 8=) :=') 8=) 8=?
Jeff S. Haberl, Ph.D.,P.E.inactive,FASHRAE,FIBPSA,......jhaberl at tamu.edu
Professor........................................................................Office Ph: 979-845-6507
Department of Architecture............................................Lab Ph:979-845-6065
Energy Systems Laboratory...........................................FAX: 979-862-2457
Texas A&M University...................................................77843-3581
College Station, Texas, USA, 77843.............................http://esl.tamu.edu
8=/ 8=) :=) 8=) ;=) 8=) 8=() 8=) :=) 8=) 8=! 8=) 8=? 8=) 8=0

Jeff Haberl2's picture
Offline
Joined: 2011-10-02
Reputation: 0

Maybe I'm missing something here, but why can't you just count up the degree days for the
utility period?
I hope you're not working with average or "typical year" degree days, but the degree days
from the same time period.

I also recall that the old Princeton Scorekeeping Method (PRISM) back in the 1980's allows
the user to enter the degree days for that time period, so it's not a new problem.

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"

Joe Huang's picture
Offline
Joined: 2011-09-30
Reputation: 406

Thanks Joe,

I did not mean to suggest using tmy data but did mean that actual weather
data for the billing period should be used.

One of my favorite sources of real time data is weather underground. They
have a lot of smaller weather stations and a great archive.

Cheers,

Mike
On Jun 22, 2015 10:17 PM, "Joe Huang"
wrote:

Michael Tillou's picture
Offline
Joined: 2011-10-02
Reputation: 0

Hi Chris,

FASER and Metrix also calculated degree days in the actual billing periods.

I would be inclined to match the simulation to the billing periods.
Assuming you are using a DOE-2 based tool, I do not know how to change the
dates reported in the PS-E reports. However you could create 12 periods in
the electrical and fuel rates based on the billing dates. Then the ES-E and
ES-F would report the energy as well as cost split by billing period.

Cheers,

BF

bfountain's picture
Offline
Joined: 2011-09-30
Reputation: 201

Hello Joe,

Yes, you can count the degree days and regress against that to show a correlation. However, one will get a better "fit" to the weather data if you regress to the degree day that is calculated for the balance point temperature of the building -- hence the inverse model toolkit or the variable based degree day method.

PRISM actually calculates the degree days to a variety of change points and actually provides a table for each location that you use as a look up. The IMT will actually perform a variable based degree day calculation that agrees well with PRISM. IMT will also provide you with the average daily temperature for the billing period.

When using DOE-2 for actual billing periods, one will have to extract the appropriate hourly variable, sum it to daily and then regroup to align with the billing periods. Here's a chunk of code that will create a dummy plant, display PV-A, PS-A, PS-E and BEPS, and extract the relevant hourly variables to normalize the BEPS to the utility bills:

INPUT PLANT ..

PLANT-REPORT VERIFICATION = (PV-A)

$ PV-A, EQUIPMENT SIZES

SUMMARY = (PS-A,PS-E,BEPS)

$ PS-A, PLANT ENERGY UTILIZATION SUMMARY

$ PS-E, MONTHLY ENERGY END USE SUMMARY

$ BEPS, BUILDING ENERGY PERFORMANCE SUMMARY

HVAC=PLANT-ASSIGNMENT ..

$ EQUIPMENT DESCRIPTION

$ ELECTRIC DOMESTIC WATER HEATER

BOIL-1 =PLANT-EQUIPMENT TYPE=ELEC-DHW-HEATER SIZE=-999 ..

$ ELECTRIC HOT-WATER BOILER

BOIL-2 =PLANT-EQUIPMENT TYPE=ELEC-HW-BOILER SIZE=-999 ..

$ HERMETICALLY SEALED CENT CHILLER

CHIL-1 =PLANT-EQUIPMENT TYPE=HERM-CENT-CHLR SIZE=-999 ..

$ Graphics block for Data Processing ***

RP-3 = SCHEDULE THRU DEC 31 (ALL) (1,24) (1) ..

$ 8 = Total PLANT heating load (Btu/h)

$ 9 = Total PLANT cooling load (Btu/h)

$ 10 = Total PLANT electric load (Btu/h)

BLOCK-3-1 = REPORT-BLOCK

VARIABLE-TYPE = PLANT

VARIABLE-LIST = (8,9,10) ..

BLOCK-3-2 = REPORT-BLOCK

VARIABLE-TYPE = GLOBAL

VARIABLE-LIST = (1) ..

HR-3 = HOURLY-REPORT

REPORT-SCHEDULE = RP-3

REPORT-BLOCK = (BLOCK-3-1,BLOCK-3-2) ..

END ..

COMPUTE PLANT ..

STOP ..

8=! 8=) :=) 8=) ;=) 8=) 8=( 8=) 8=() 8=) 8=| 8=) :=') 8=) 8=?
Jeff S. Haberl, Ph.D.,P.E.inactive,FASHRAE,FIBPSA,......jhaberl at tamu.edu
Professor........................................................................Office Ph: 979-845-6507
Department of Architecture............................................Lab Ph:979-845-6065
Energy Systems Laboratory...........................................FAX: 979-862-2457
Texas A&M University...................................................77843-3581
College Station, Texas, USA, 77843.............................http://esl.tamu.edu
8=/ 8=) :=) 8=) ;=) 8=) 8=() 8=) :=) 8=) 8=! 8=) 8=? 8=) 8=0

Jeff Haberl2's picture
Offline
Joined: 2011-10-02
Reputation: 0

Hello all,

We usually do the following to calibrate model to monthly utility bills:

1) Create or purchase weather file corresponding to pre-retrofit period
for which we have billing data. Lately we've been using WeatherAnalytics
files, which we found to be more cost effective than creating our own (they
charge $40 for an annual file).

2) Run simulation using this weather file instead of TMY.

3) Standard simulation reports (we typically use eQUEST) show usage by
calendar month (e.g. January, February, etc.) which is usually not aligned
with dates of utility bills, as noted in the question that started this
thread. As Brian mentioned in one of the earlier posts, this may be
circumvented by entering the actual meter read dates into eQUEST as shown in
the screenshot below. This will align usages shown in eQUEST's "E*" reports
such as ES-E with the actual utility bills. The approach does not allow
entering more than one read date per month (e.g. we can't capture April 3 -
28 bill). For projects where this limitation is an issue we generate hourly
reports that show consumption by end use for each meter in the project, and
aggregate it into periods that are aligned with utility bills.

4) We then copy simulation outputs (either from ES-E or hourly reports,
depending on the method used) into a standard spreadsheet with utility data.
The spreadsheet is set up to plot side by side monthly utility bills and
simulated usage, and also calculates normalized mean bias error (NMBE) and
variance CV(RMSE).

5) If we did not to where we want to be with NMBE and CV(RMSE) we
adjust and re-run the model, and re-paste results into the same spreadsheet.

In my experience regression analysis using weather as independent variable
(i.e. running model with TMY file and normalizing for difference in weather)
or relying on HDD to allocate usage to billing periods can be very
misleading, mainly because on many projects weather is not the main driver
of consumption. For example energy usage of a school during a given time
period depends much more on vacation schedule than outdoor dry bulb
temperatures.

Thanks,

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

Maria Karpman's picture
Offline
Joined: 2011-09-30
Reputation: 0

All,

This is a fantastic thread, and I am wondering if it could be taken one step further to query if anyone has experience with methods to attempt calibrating models of energy savings attributable to retrofits of multiple systems simultaneously (plant, envelope, HVAC etc. - as most real-world retrofits likely are), going past the 4- or 5-parameter breakpoint regression models to incorporate inverse modeling of specific load types and their space- or time-variable characteristics. This would fit under multivariate methods in the last line of Table 2 in the older version of ASHRAE Guideline 14 that Jeff Haberl has posted on his website, and would attempt to standardize Maria's Step 5 below without (possibly) the need to conduct as much in-depth field verification as might otherwise be required. I've dabbled in this a little bit...without extensive discussions with others...

Example: changing the OA ventilation rate is going to have a specific load profile versus some retrofit that affects the solar gain rate. Of course, much easier in theory to do calibrations of this sort with hourly meter data versus monthly utility bills...

Bill Collinge
Postdoctoral Scholar
University of Pittsburgh
Department of Civil and Environmental Engineering

WOC6 at pitt.edu's picture
Joined: 2014-07-16
Reputation: 0

I'm encouraged to see so many people addressing this topic because it means
you are modeling existing buildings; a lot of work is needed in this
arena. Keep it up!

We, as usual, have a spreadsheet solution. In this case, the spreadsheet
is happy to use billing periods of any length, such as is normal for
day-of-reading variations, but also to combine "estimated" readings into a
period that has an actual reading at each end.
It requires that you tell EnergyPlus to report hourly meter data for each
fuel (e.g., electricity and natural gas). A macro totals that data into
the billing periods for your site, displays the predicted vs actual energy
and calculates an R-squared value for each fuel and the total. An example
is shown below.

[image: Inline image 1]

James V Dirkes II, PE's picture
Joined: 2011-10-02
Reputation: 203

This is a little off-topic, but something I've pondered for some time...

The question is when people are using eQUEST/DOE-2 with historical year weather, what do
you do when it's a leap year? Since DOE-2
always simulates a 365-day year, do you just ignore the missing leap day, but then don't
the Days of Week also get screwed up starting in March?

Since a quarter of the years are leap years, I've never understood why accounting for them
has been considered an insignificant detail.
I mean, if I told you that a quarter of the time your simulation results would be a little
wrong, isn't that a pretty high frequency?

Many eQUEST/DOE-2 users also have the mistaken impression that the fault lies in the DOE-2
weather files, which is not true.
Believe it or not, but the packed DOE-2 weather file format actually contains 384 days (32
days per month), and all the DOE-2 weather files I produce always contains Feb. 29 for the
leap years (as well as other enhancements like greater precision in the data).

So, where does the problem lie? It's in the clock within DOE-2 that always sets February
to be 28 days. In other words, DOE-2 will read the weather file and do the simulation
only through February 28th, even though the weather file contains data through February
32nd (:-)), although everything beyond the 28th would be blank on non-leap years, and
beyond the 29th on leap years.

When I've looked through the DOE-2.1E code, there are even flags setting the leap years
but these are never used. I've thought many times of toying around with the code to see
how difficult it would be to implement leap years, but just haven't gotten around to it.
As far as I can see, the biggest difficulty might might have to do not with the simulation
itself, but with the reporting.

I'd like to know if others think this is something of sufficient importance to merit
further investigation.

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"

Joe Huang's picture
Offline
Joined: 2011-09-30
Reputation: 406

1. Use EnergyPlus :), which allows >365 days. This is also helpful when
the combined two-fuel billing cycle is 13-14 months.
2. Ignore the 1/365 difference. Do you really think it will matter much?

James V Dirkes II, PE's picture
Joined: 2011-10-02
Reputation: 203

Jim and all,

We have incentive programs in NY (Multifamily Performance Program) and NJ (Pay for Performance Program for C&I buildings, P4P) that require developing calibrated models to estimate ECM savings. Both programs rely on spreadsheet-based tools to facilitate model calibration, and require that the proposed ECM package reduces overall energy consumption by at least 15%. The programs have been around for 5+ years, and have hundreds of participating projects. Part of the incentive is awarded based on the projected (i.e. modeled) savings, and the rest (as much as 50% for P4P) based on the actual achieved savings established using Whole Building approach by comparing pre/post utility bills.

My biggest take away from the involvement with these programs was that a calibrated model that meets MBE, CVRMSE, and uncertainty requirements of Guideline 14 may produce grossly incorrect ECM savings. It is not feasible to create a calibrated model that can be reliably used to project savings from any conceivable ECM in a commercial non-research setting because, aside from the modeling effort, it would require a lot of very detailed field work. On the other hand, developing a calibrated simulation to estimate savings from a particular set of measures considered for a given project is a much more manageable task. So the bulk of the recent updates to the technical requirements of our incentive programs were focused on itemizing parameters that should be tweaked to achieve calibration depending on the ECMs included in the project scope. For example, if project involves boiler replacement, efficiency of existing boilers that are being replaced must be measured. (We had a project which modeled existing boiler as 35% efficient because that produced calibrated simulation. Of course such model would very likely exaggerate savings from installing a new boiler.)

Do any of you know references that outline calibration techniques depending on the ECMs being modeled, beyond the general advice included in IPM&VP?

Thanks,

Maria

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

Maria Karpman's picture
Offline
Joined: 2011-09-30
Reputation: 0

I think the cleanest is you just pretend every day is off by one. Ignore
all of the month garbage (yes you'll be off by a day at times). Just think
about it as days 1-365, with the right day of the week assigned. You can
reassign your holidays if you want. You wind up dropping the real 12/31.

But I like the "just use EnergyPlus" option.

Justin Spencer's picture
Offline
Joined: 2011-10-02
Reputation: 1

The responses so far are not what I expected and, in my view, miss the point.
I was not talking about workarounds or ignoring the missing day in DOE-2, which is what I
presume everyone has been doing up until now. I'm frankly tired of that, because adding
the fixes to DOE-2 seems to be quite easy to do.

I also find the responses of "just use EnergyPlus" to be disingenuous and condescending.
It's like trying to fix a scratch on your car, and then somebody comes by and says, "oh,
just go and buy this new better one".

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"

Joe Huang's picture
Offline
Joined: 2011-09-30
Reputation: 406

I'll second that this has turned into a really good thread. My personal
rule of thumb in doing calibration is, "don't mess with things that relate
directly to an ECM." If you have a retrofit ECM going on, you should have
detailed data on the pre-installation conditions associated with that ECM.
If that isn't documented, your model is never going to provide you with
accurate savings results. It pays to think about these things at the
fundamental level. And also apply a reasonableness range to those values.
What am I most unsure about? What's the possible range of this value?
Personally, I twist my dials all in one direction and then use setpoint as
my fine calibration factor at the end. We need to remember that we're using
models as tools to help guide decision-making. When we make decisions in
calibration, we should think about how it will impact the decision being
made -- does it impose a bias (like the 35% assumed pre-retrofit boiler
efficiency for a boiler retrofit project)? These models are just tools for
extrapolating from known energy consumption data to unknown energy
consumption data.

An earlier post commented that hourly data might make calibration easier...
It doesn't make it easier in the sense of less work, but it does make your
model a lot better if you do it right. Calibrating models to real hourly
consumption data tells you so much about what is likely wrong with your
model and so much about how your real building performs. If you have hourly
end use data, or close, then you are really in business. At the aggregate
level, this kind of data can tell you that you didn't really know much to
start with. For one project for Con Edison, we had access to hourly
whole-premise consumption data for several thousand NYC buildings by
building type. We were able to use this data to uncover that our models
vastly underestimated consumption at nights and on weekends. We didn't know
why, but we altered schedules accordingly.

As for the monthly billing data question at the beginnning of this thread,
if you're dealing with a building and climate where monthly variability in
occupancy and weather are driving changes in energy consumption in smooth,
continuous fashion, you can make some other approximations. We've used a
simplified slope method, where you calculate the slope in usage between the
two adjacent billing months, i.e. if period 1 is 1000 kWh/day, period 2 is
1100 kWh/day, and period 3 is 1200 kWh/day, and all are 30 days long, you
wind up with a slope of 3.33 kWh/day. We then assign daily kWh for each
month using this slope and then reslice the data to match our calendar
months. This is important when you are calibrating an aggregate model,
which is a common need for us when we're estimating program-level savings
for an ECM. For example, we're using a building simulation model to
extrapolate measured consumption at a large sample of sites to usage in a
typical year.

Alternatively, for individual buildings, we don't worry about it and just
calibrate to the billing periods instead. We're not generally working in
eQuest, but rather with the hourly output of whatever engine we're using.

One thing I've been wondering about recently is how to avoid
over-calibration. The econometrics/statty/mathy folks I occasionally
consort with talk a lot about overfitting of regressions and the same
problem applies here. Has anybody ever tried reserving part of their
calibration data set to use as a test set? I've gotten some things that
looked like really great calibrations in the past, but I'm wondering how
folks have sought to prove whether they were getting things right or
overfitting. When I teach junior staff about this sort of thing, I always
include something about not getting too cute. I point them to the John Von
Neumann quote:

- *With four parameters I can fit an elephant
, and with five I can make him
wiggle his trunk.*

Justin Spencer's picture
Offline
Joined: 2011-10-02
Reputation: 1

I guess the better, more direct answer, from my perspective is:
No, I don't think this is that important compared to some of the other
modeling problems out there in the world.

Justin Spencer's picture
Offline
Joined: 2011-10-02
Reputation: 1

I hear your frustrations Joe, but after finding many, many issues with
eQuest/Doe-2 that never got fixed, I eventually did "just" switch to
EnergyPlus. I'm not sure when the last time DOE2 got a major update, but
EnergyPlus is getting major updates all the time. There are people
actively supporting this software and every bug and idiosyncrasy I've found
has either been fixed, or is in the cue to be fixed.

In my opinion, this community needs to either:

1. Support a major update to eQuest/DOE-2 that fixes this, and many other
issues. If you are capable and interested in fixing some of these bugs -
maybe try a crowdfunding campaign? People could contribute towards fixing
particular issues or bugs.
2. Recognize the limitations of eQuest/ DOE-2 and use it only when
projects can be appropriately modeled with this software.
3. Switch software packages completely.

One of the challenges in this industry is that people are used to getting
software for free. eQuest, DOE-2, EnergyPlus have all be developed in
large part with public funding. When that funding goes away, support
stops, but people still have the expectation that the software should be
free.

--
Karen

No Username provide's picture
Joined: 2011-09-30
Reputation: 200

Getting upset that eQuest / DOE-2 doesn?t incorporate leap year data is like getting upset that it can?t predict a snow day. Or a power outage. If you are using an energy model for a specific task where missing 1 day in 1,460 is going to affect someone?s decision making process, and/or you?re not willing to multiply February energy consumption by 29/28 for that year, then that is worrisome. No energy modeling program is so accurate that this would make a difference. But that?s just my opinion. I have lots of scratches on my car and don?t care?

-James

James Hansen's picture
Offline
Joined: 2011-09-30
Reputation: 200

Dear Joe,
Absolutely no condescension intended. I apologize for even coming close.
It was supposed to be tongue in cheek; I realize that switching software is
a tough thing and not necessarily a good plan for all.

James V Dirkes II, PE's picture
Joined: 2011-10-02
Reputation: 203

This has been an interesting thread. I raised this question at an ASHRAE conference a few years ago and nobody in that session did anything to account for Leap Day.

Couldn?t the day of the week selected for January 1 be shifted a day so that the last 10 months of the year will match and only 2 months will be off (instead of 2 months matching and 10 months being off a day)?

Keith Swartz, PE
Senior Energy Engineer | Seventhwave | Madison.Chicago.Minneapolis
(formerly Energy Center of Wisconsin)
608.210.7123 | www.seventhwave.org

Keith Swartz2's picture
Offline
Joined: 2015-04-29
Reputation: 0

It's days of the week that matter. You'll never get your calibration to
line up on daily or hourly usage if your days of the week get off. I was
trying to say in my earlier response that your days of the week are not off
by a day if you just let it ride and have your simulation still run through
days 1-365. What's off by a day is your month start and end days and
potentially your holidays, which you can relatively easily set to alternate
dates.

Justin Spencer's picture
Offline
Joined: 2011-10-02
Reputation: 1

Justin,

That's right. The biggest impact of the missing leap day is not the missing day itself,
but the shift in the days of the week after Feb. 29th. Yesterday I made a run for a leap
year (1992) to confirm that DOE-2 doesn't adjust the days of the week for the missing day,
i.e., Mar. 1 is not two days-of-the-week behind Feb. 28.

Now, if you're modeling a building that has distinct workday and weekend schedules, as all
commercial buildings do, the daily load shapes will be out of phase with what they
actually are from March 1 through the end of the year. To me, that alone would be
sufficient reason to fix, especially that the fix is not very difficult.

The main reason this defect has been neglected so far is that most simulations are still
being done for design assessment using "typical year" weather data. However, now that
more simulations are being done using actual year weather data to compare to actual
performance, the defect is no longer so trivial.

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"

Joe Huang's picture
Offline
Joined: 2011-09-30
Reputation: 406

Excellent insights in this thread!
Our "two cents":

- We measure key variables such as larger motor kW and outdoor airflow.
The idea is to eliminate uncertainty for energy aspects that we know are
"big". Because it's easy to obtain, we use the actual weather data. It's
not always a big impact, but eliminates needless uncertainty.
- We don't trust people's memory about almost anything; it's amazing how
different the story can be when told by two different people, both of whom
should be knowledgeable.
- We also don't trust sensor calibration. Not all sensors matter as
much, however; discharge sensors on reheat systems, for example, matter
more than room temp sensors.
- More data is better. Smart meters are very helpful, as is trend data
if the client has taken time to set them up. Not many do :(
- We have not yet tinkered with sensitivity analysis for uncertain
variables such as boiler efficiency or infiltration, but we do run GenOpt
for the best R-squared fit to each (exact) billing period and review /
revise the output for reasonableness.
- On the one hand, it bothers me that calibrated models are "grossly
incorrect" in Maria's experience. My bias is that those models may have
too little measured data. On the other hand, this is a brave new world and
we probably all have a lot to learn.

James V Dirkes II, PE's picture
Joined: 2011-10-02
Reputation: 203

A dissertation, no less. FYI, I think I 'll be able to absorb this
sometime next week, but not sooner.
Thank you for sending it; I don't think I attended your talk because all of
this is sounding new... and I like learning.

James V Dirkes II, PE's picture
Joined: 2011-10-02
Reputation: 203

Jim,

FYI attached is a presentation from 2014 ASRHAE/IBPSA summarizing our findings on accuracy of the calibrated simulations based on projects that went through an incentive program in NJ. (You presented there also ? loved your Sherlock Holmes hat and pipe J.) To clarify the language, ECM savings estimated via calibrated simulation are referred in the presentation as ?projected savings?; ECM savings estimated via ?whole building? approach are referred to as ?realized savings?. Slide 7 summarizes high level results ? while there was a relatively good alignment between the total projected and realized savings for the entire sample, individual projects were all over the map.

I agree with your last point below that the greatest source of error is too little measured data, but given the finite budget that each project has, I hoped that documents such as Guideline 14 would emphasize the link between the minimum scope of measurements and ECMs considered for a given project. For example Guidelines 14 says the following concerning simulation defaults, which in my opinion is impractical:

5.3.3.3.7 Minimizing Default Values. Check and thoroughly

understand all default input variables in the simulation

program, as many of the default values have little resemblance

to the actual building being simulated. The fewer the

number of default values used, the more representative the

simulation will be?but only if the changes are well reasoned.

This also includes inspection of the default performance

curves of the various systems and plant equipment, because

such curves can significantly impact the results of the simulation.

Any program default values that are altered, however,

should be well documented.

There is also a section on adjusting infiltration / ventilation rates to calibrate the model (quoted below), but it should only apply if project does not include ECMs involving air-sealing (which are common in multifamily and school projects) or ventilation controls such as DCV. If project includes infiltration or ventilation related measures, the related inputs cannot be used to calibrate the model even as ?a last resort?.

5.3.3.3.6 Estimating Infiltration Rates. Infiltration

rates are difficult to measure and may be treated as an

unknown that is iteratively solved with the simulation program

once the other major parameters are determined. This

approach is only recommended as a last resort. To solve for

the infiltration rate and/or the ventilation rate iteratively, conduct

a series of simulation runs such that only the infiltration

and/or ventilation rates are changed from one-tenth to as

much as ten times the expected rates. Next, compare the simulation

outputs produced to the measured building data as discussed

below. In addition, supporting evidence should be

used to justify the final choice of variables.

I think calibrated simulation should be treated as an extension of retrofit isolation approach, and site measurements performed in support of simulation must at minimum include (a) parameters that are modified to model ECM savings, and (b) simulation inputs that do not change but drive ECM savings. For lighting fixture replacement ECM, pre-retrofit wattage would be an example of (a), and lighting runtime hours would be an example of (b). On the other hand, if there are no ECMs that reduce base load (e.g. plug load, lighting, etc.), but there are ECMs involving HVAC controls, site measurements should focus on operation of existing controls, and establishing existing lighting wattage is not necessary because model calibrated to capture the overall base load (lighting + plug loads) usage may be good enough in this case. We had to develop guidelines for incentive programs that rely on calibrated simulation to specify minimum scope of measurements and acceptable ?estimated? inputs for each common ECM type. I hope that a similar guidance would be included at some point in one of ASHRAE standards / guidelines. As it stands now, there is a sea of difference between the specificity of simulation requirements in 90.1 Appendix G compared to calibrated simulation of existing buildings. Unless I am missing some key reference?

Maria

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

Maria Karpman's picture
Offline
Joined: 2011-09-30
Reputation: 0

Hello Maria,

Thanks for actually reading Guideline 14-2014. However, I'm sorry to disagree, but Section 5.3.3.3.7 in Guideline 14-2014 seems pretty clear to me (...since I had a hand in writing it). In fact, I can't imagine doing a calibrated simulation without checking the defaults (as a minimum). Unfortunately, understanding the performance curves of the installed equipment can be tricky since few if any systems installed today come with factory-generated performance curves that are plug-and-play with simulation curve inputs. However, I think Guideline 14 is right in shining a light on this issue. Don't forget, one of the purposes of a guideline is to bring issues to the surface so knowledgeable users can decide to do something or justify why they did not do it.

In the same light, the advice in Section 5.3.3.3.6 also seems reasonable (....I think I wrote this too...). If you have actual blower door or duct blaster measurements, then these would be preferred over an empirical adjustment. There are many studies out there that used the "brute force" ACH adjustment for infiltration, which can improve a model's fit after all other variables are considered.

Your suggestion to more closely link the calibrated simulation to the intended ECM is reasonable. In fact, now that Guideline 14 is published, we'll probably be starting in on the "next version" in a year or so and I would invite you to join the Committee.

Jeff

PS: Did you download and try the software that now comes with Guideline 14-2014?...this includes the IMT from RP1050 and Diversity Factor toolkit from RP1093.

8=! 8=) :=) 8=) ;=) 8=) 8=( 8=) 8=() 8=) 8=| 8=) :=') 8=) 8=?
Jeff S. Haberl, Ph.D.,P.E.inactive,FASHRAE,FIBPSA,......jhaberl at tamu.edu
Professor........................................................................Office Ph: 979-845-6507
Department of Architecture............................................Lab Ph:979-845-6065
Energy Systems Laboratory...........................................FAX: 979-862-2457
Texas A&M University...................................................77843-3581
College Station, Texas, USA, 77843.............................http://esl.tamu.edu
8=/ 8=) :=) 8=) ;=) 8=) 8=() 8=) :=) 8=) 8=! 8=) 8=? 8=) 8=0

Jeff Haberl2's picture
Offline
Joined: 2011-10-02
Reputation: 0

Since we have your ear Jeff, I have wondered if Guideline 14 describes any penalty to apply to the goodness of fit metric when allowing unknown parameters to be varied to improve the fit to monthly data? Something like how adjusted R-squared or AIC penalize models with more parameters? It seems that if you achieve a given level of goodness of fit after fiddling with a bunch of parameters then that is not as good as achieving the same level of fit with fewer free parameters. Is there any way to estimate the number of degrees of freedom lost when changing a single numeric model parameter like ACH vs a more complicated one like changing out entire HVAC curves?

Dan

Macumber, Daniel's picture
Joined: 2011-10-02
Reputation: 0

Really! You think I can answer that? That's bordering on the Holy Grail.

Of course, you're right in asking the question, and in my mind that's a good question to ask. Although, in my experience, I have never had all the measured data I need to answer it for a given building. So, I guess it is always an uphill battle.

I think the closest thing I've ever seen to having a 'rational method' that calibrated a simulation to specific, measured data for individual parameters is the PSTAR/BEVA method that Kris Subbarao developed in the 1990s, motivated to some extend by the ETP models that Sondereggar developed in the 1970s. This method worked well for residences because it required the user to go and measure specific things that plugged into certain variables in the simplified model. However, I'm not aware that the method was successfully extended to a general purpose model for commercial buildings.

Nevertheless, we can and do learn from it. For example, the calibration procedure developed by Hsieh and Norford in the 1980s was developed for a commercial building and is at the heart of the guidance in Guideline 14. Namely, measure the electric loads into the building and force the model to use the measured results. Then, see if the accuracy improves or not, and adjust as needed. Amazingly, this single variable has a large impact on improving the accuracy.

Next, check to see if you have the appropriate system and the temperature settings/schedules of the HVAC system to see if it improves things. This was also proposed by Hsieh and Norford as well as Kaplan in his Energy Edge paper.

Finally, use Manke and Hittle's method to exercise the model inputs and see what other variables need tuning.

I personally prefer to use canned graphs to 'see' the calibration as well as statistics to judge the final outcome.

Hope this helps.

Jeff

Jeff S. Haberl, Ph.D., P.E.inactive, FASHRAE, FIBPSA
Department of Architecture
Texas A&M University
College Station, Texas 77845-3581
Office: 979-845-6507, Lab: 979-845-6065
Fax: 970-862-2457, jhaberl at tamu.edu, www.esl.tamu.edu

Jeff Haberl2's picture
Offline
Joined: 2011-10-02
Reputation: 0

Jeff,

Guideline 14 has been my favorite read for some years now J, and was one of the key resources used for developing technical requirements for a number of modeling-based incentive programs for existing buildings here in North East. My comments come from witnessing its application on a mass scale, involving hundreds of projects, over 7-8 years, with modeling done by dozens of different companies. Program participants have to comply with the Simulation Guidelines that reiterate key concepts of Guideline 14 and IPM&VP related to calibrated simulation in a more enforceable language. A few years into running these programs it became clear that, to avoid repeating the same comments again and again when reviewing submitted reports, and to ensure that projected ECM savings (and claimed incentive $) were reasonable, we needed to expand the guidelines to address measure-specific modeling requirements. These requirements cover simulation inputs that should be obtained via measurements for a range of common ECMs, and defaults that must be used if measurements were not performed, to ensure that projected savings were conservative.

You wrote that ??There are many studies out there that used the "brute force" ACH adjustment for infiltration, which can improve a model's fit after all other variables are considered.? Adjusting ACH is indeed very common and often the quickest way to meet MBE & CV RMSE, and many of the programs? participants did just that. Projects that included air-sealing ECM would then estimate savings by reducing the ACH in the calibrated model by a certain (usually generous) percentage. To combat the issue, our program?s guidelines require that projects itemize the scope of air-sealing work, and use a spreadsheet developed based on a table from 2001 ASHRAE Fundamentals to convert it to equivalent ACH reduction. The ASHRAE table listed ELA associated with a wide range of building components in various states of disrepair, e.g. un-caulked wood doorframe and caulked wood door frame. The tool used the data to calculate ELA reduction from air sealing scope (e.g. caulking a doorframe), and then converted it to equivalent ACH reduction, discounting for effects of ventilation. This delta ACH can then be incorporated into simulation to capture ECM savings. We also limited pre-retrofit ACH that can be entered when calibrating the model. For example, if it took ACH 5 to get a good model fit, then report should identify areas of significant air leakage and include related ECMs. Alternatively, if a building appears to be relatively tight, another parameter(s) affecting heating/cooling load must be investigated and used to calibrate the model; these parameters may also be considered for improvement.

You also wrote that ??. I can't imagine doing a calibrated simulation without checking the defaults (as a minimum).? Some 90.1- compliant tools give users much more flexibility in defining systems, and consequently have many more editable defaults, compared to others where more parameters are hard-coded in software?s algorithms. So requiring that all defaults are checked, aside from penalizing users of more flexible tools, will not ensure consistent quality of results across projects. I think a more practical alternative would be to require checking defaults (or using actual measured data) for areas that drive ECM savings, and define these areas for common types of ECMs. For example, if a calibrated model is used to estimate savings from exhaust air energy recovery ECM, then defaults related to design and OA flow and schedule, recovery effectiveness, fan efficiency, pressure drop associated with recovery device, etc. must be edited to match site conditions and proposed new equipment. 90.1 ECB / App G compliance form asks for a ?Number of defaults overridden?, which was also incorporated into early LEED EAp2/c1 templates. This question always puzzled me ? how do you count this for say an eQUEST model? And as submittal reviewer, would you flag projects with too many over-written defaults, or too few over-written defaults? Or do you check for similar number of over-written defaults between baseline and proposed model?

To summarize, I think Guideline 14 is a great resource, but am disappointed with the pace of its development. With one ASHRAE president using ?Sustaining our Future by Rebuilding our Past? as a theme of his presidency, and another ?Modeling a Sustainable World? as a theme of hers, I was hoping for a much more substantial update to the calibrated simulation portion of the guideline given that 12 years have passed since its last release. For example, 2014 version has 5+ pages of normative references and bibliography, including research papers from 1980s. Seems like there was plenty of time to pull actionable findings from these papers into the main body of the guideline? I?d be happy to help with the development and would love an opportunity to join the committee. I actually reached out to some of the authors of the guideline a while back (I am not going to name names J), but did not get much of a response. And then I overlooked public comment opportunity, so have no right to complain J.

Maria

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

Maria Karpman's picture
Offline
Joined: 2011-09-30
Reputation: 0