Maybe I'm missing something here, but why can't you just count up the degree days for the
utility period?
I hope you're not working with average or "typical year" degree days, but the degree days
from the same time period.

I also recall that the old Princeton Scorekeeping Method (PRISM) back in the 1980's allows
the user to enter the degree days for that time period, so it's not a new problem.

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"

Hi Chris,

FASER and Metrix also calculated degree days in the actual billing periods.

I would be inclined to match the simulation to the billing periods.
Assuming you are using a DOE-2 based tool, I do not know how to change the
dates reported in the PS-E reports. However you could create 12 periods in
the electrical and fuel rates based on the billing dates. Then the ES-E and
ES-F would report the energy as well as cost split by billing period.

Cheers,

BF

Hello all,

We usually do the following to calibrate model to monthly utility bills:

1) Create or purchase weather file corresponding to pre-retrofit period
for which we have billing data. Lately we've been using WeatherAnalytics
files, which we found to be more cost effective than creating our own (they
charge $40 for an annual file).

2) Run simulation using this weather file instead of TMY.

3) Standard simulation reports (we typically use eQUEST) show usage by
calendar month (e.g. January, February, etc.) which is usually not aligned
with dates of utility bills, as noted in the question that started this
thread. As Brian mentioned in one of the earlier posts, this may be
circumvented by entering the actual meter read dates into eQUEST as shown in
the screenshot below. This will align usages shown in eQUEST's "E*" reports
such as ES-E with the actual utility bills. The approach does not allow
entering more than one read date per month (e.g. we can't capture April 3 -
28 bill). For projects where this limitation is an issue we generate hourly
reports that show consumption by end use for each meter in the project, and
aggregate it into periods that are aligned with utility bills.

4) We then copy simulation outputs (either from ES-E or hourly reports,
depending on the method used) into a standard spreadsheet with utility data.
The spreadsheet is set up to plot side by side monthly utility bills and
simulated usage, and also calculates normalized mean bias error (NMBE) and
variance CV(RMSE).

5) If we did not to where we want to be with NMBE and CV(RMSE) we
adjust and re-run the model, and re-paste results into the same spreadsheet.

In my experience regression analysis using weather as independent variable
(i.e. running model with TMY file and normalizing for difference in weather)
or relying on HDD to allocate usage to billing periods can be very
misleading, mainly because on many projects weather is not the main driver
of consumption. For example energy usage of a school during a given time
period depends much more on vacation schedule than outdoor dry bulb
temperatures.

Thanks,

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

I'm encouraged to see so many people addressing this topic because it means
you are modeling existing buildings; a lot of work is needed in this
arena. Keep it up!

We, as usual, have a spreadsheet solution. In this case, the spreadsheet
is happy to use billing periods of any length, such as is normal for
day-of-reading variations, but also to combine "estimated" readings into a
period that has an actual reading at each end.
It requires that you tell EnergyPlus to report hourly meter data for each
fuel (e.g., electricity and natural gas). A macro totals that data into
the billing periods for your site, displays the predicted vs actual energy
and calculates an R-squared value for each fuel and the total. An example
is shown below.

[image: Inline image 1]

1. Use EnergyPlus :), which allows >365 days. This is also helpful when
the combined two-fuel billing cycle is 13-14 months.
2. Ignore the 1/365 difference. Do you really think it will matter much?

I think the cleanest is you just pretend every day is off by one. Ignore
all of the month garbage (yes you'll be off by a day at times). Just think
about it as days 1-365, with the right day of the week assigned. You can
reassign your holidays if you want. You wind up dropping the real 12/31.

But I like the "just use EnergyPlus" option.

I'll second that this has turned into a really good thread. My personal
rule of thumb in doing calibration is, "don't mess with things that relate
directly to an ECM." If you have a retrofit ECM going on, you should have
detailed data on the pre-installation conditions associated with that ECM.
If that isn't documented, your model is never going to provide you with
accurate savings results. It pays to think about these things at the
fundamental level. And also apply a reasonableness range to those values.
What am I most unsure about? What's the possible range of this value?
Personally, I twist my dials all in one direction and then use setpoint as
my fine calibration factor at the end. We need to remember that we're using
models as tools to help guide decision-making. When we make decisions in
calibration, we should think about how it will impact the decision being
made -- does it impose a bias (like the 35% assumed pre-retrofit boiler
efficiency for a boiler retrofit project)? These models are just tools for
extrapolating from known energy consumption data to unknown energy
consumption data.

An earlier post commented that hourly data might make calibration easier...
It doesn't make it easier in the sense of less work, but it does make your
model a lot better if you do it right. Calibrating models to real hourly
consumption data tells you so much about what is likely wrong with your
model and so much about how your real building performs. If you have hourly
end use data, or close, then you are really in business. At the aggregate
level, this kind of data can tell you that you didn't really know much to
start with. For one project for Con Edison, we had access to hourly
whole-premise consumption data for several thousand NYC buildings by
building type. We were able to use this data to uncover that our models
vastly underestimated consumption at nights and on weekends. We didn't know
why, but we altered schedules accordingly.

As for the monthly billing data question at the beginnning of this thread,
if you're dealing with a building and climate where monthly variability in
occupancy and weather are driving changes in energy consumption in smooth,
continuous fashion, you can make some other approximations. We've used a
simplified slope method, where you calculate the slope in usage between the
two adjacent billing months, i.e. if period 1 is 1000 kWh/day, period 2 is
1100 kWh/day, and period 3 is 1200 kWh/day, and all are 30 days long, you
wind up with a slope of 3.33 kWh/day. We then assign daily kWh for each
month using this slope and then reslice the data to match our calendar
months. This is important when you are calibrating an aggregate model,
which is a common need for us when we're estimating program-level savings
for an ECM. For example, we're using a building simulation model to
extrapolate measured consumption at a large sample of sites to usage in a
typical year.

Alternatively, for individual buildings, we don't worry about it and just
calibrate to the billing periods instead. We're not generally working in
eQuest, but rather with the hourly output of whatever engine we're using.

One thing I've been wondering about recently is how to avoid
over-calibration. The econometrics/statty/mathy folks I occasionally
consort with talk a lot about overfitting of regressions and the same
problem applies here. Has anybody ever tried reserving part of their
calibration data set to use as a test set? I've gotten some things that
looked like really great calibrations in the past, but I'm wondering how
folks have sought to prove whether they were getting things right or
overfitting. When I teach junior staff about this sort of thing, I always
include something about not getting too cute. I point them to the John Von
Neumann quote:

- *With four parameters I can fit an elephant
, and with five I can make him
wiggle his trunk.*

I hear your frustrations Joe, but after finding many, many issues with
eQuest/Doe-2 that never got fixed, I eventually did "just" switch to
EnergyPlus. I'm not sure when the last time DOE2 got a major update, but
EnergyPlus is getting major updates all the time. There are people
actively supporting this software and every bug and idiosyncrasy I've found
has either been fixed, or is in the cue to be fixed.

In my opinion, this community needs to either:

1. Support a major update to eQuest/DOE-2 that fixes this, and many other
issues. If you are capable and interested in fixing some of these bugs -
maybe try a crowdfunding campaign? People could contribute towards fixing
particular issues or bugs.
2. Recognize the limitations of eQuest/ DOE-2 and use it only when
projects can be appropriately modeled with this software.
3. Switch software packages completely.

One of the challenges in this industry is that people are used to getting
software for free. eQuest, DOE-2, EnergyPlus have all be developed in
large part with public funding. When that funding goes away, support
stops, but people still have the expectation that the software should be
free.

--
Karen

Dear Joe,
Absolutely no condescension intended. I apologize for even coming close.
It was supposed to be tongue in cheek; I realize that switching software is
a tough thing and not necessarily a good plan for all.

It's days of the week that matter. You'll never get your calibration to
line up on daily or hourly usage if your days of the week get off. I was
trying to say in my earlier response that your days of the week are not off
by a day if you just let it ride and have your simulation still run through
days 1-365. What's off by a day is your month start and end days and
potentially your holidays, which you can relatively easily set to alternate
dates.

Excellent insights in this thread!
Our "two cents":

- We measure key variables such as larger motor kW and outdoor airflow.
The idea is to eliminate uncertainty for energy aspects that we know are
"big". Because it's easy to obtain, we use the actual weather data. It's
not always a big impact, but eliminates needless uncertainty.
- We don't trust people's memory about almost anything; it's amazing how
different the story can be when told by two different people, both of whom
should be knowledgeable.
- We also don't trust sensor calibration. Not all sensors matter as
much, however; discharge sensors on reheat systems, for example, matter
more than room temp sensors.
- More data is better. Smart meters are very helpful, as is trend data
if the client has taken time to set them up. Not many do :(
- We have not yet tinkered with sensitivity analysis for uncertain
variables such as boiler efficiency or infiltration, but we do run GenOpt
for the best R-squared fit to each (exact) billing period and review /
revise the output for reasonableness.
- On the one hand, it bothers me that calibrated models are "grossly
incorrect" in Maria's experience. My bias is that those models may have
too little measured data. On the other hand, this is a brave new world and
we probably all have a lot to learn.

Jim,

FYI attached is a presentation from 2014 ASRHAE/IBPSA summarizing our findings on accuracy of the calibrated simulations based on projects that went through an incentive program in NJ. (You presented there also ? loved your Sherlock Holmes hat and pipe J.) To clarify the language, ECM savings estimated via calibrated simulation are referred in the presentation as ?projected savings?; ECM savings estimated via ?whole building? approach are referred to as ?realized savings?. Slide 7 summarizes high level results ? while there was a relatively good alignment between the total projected and realized savings for the entire sample, individual projects were all over the map.

I agree with your last point below that the greatest source of error is too little measured data, but given the finite budget that each project has, I hoped that documents such as Guideline 14 would emphasize the link between the minimum scope of measurements and ECMs considered for a given project. For example Guidelines 14 says the following concerning simulation defaults, which in my opinion is impractical:

5.3.3.3.7 Minimizing Default Values. Check and thoroughly

understand all default input variables in the simulation

program, as many of the default values have little resemblance

to the actual building being simulated. The fewer the

number of default values used, the more representative the

simulation will be?but only if the changes are well reasoned.

This also includes inspection of the default performance

curves of the various systems and plant equipment, because

such curves can significantly impact the results of the simulation.

Any program default values that are altered, however,

should be well documented.

There is also a section on adjusting infiltration / ventilation rates to calibrate the model (quoted below), but it should only apply if project does not include ECMs involving air-sealing (which are common in multifamily and school projects) or ventilation controls such as DCV. If project includes infiltration or ventilation related measures, the related inputs cannot be used to calibrate the model even as ?a last resort?.

5.3.3.3.6 Estimating Infiltration Rates. Infiltration

rates are difficult to measure and may be treated as an

unknown that is iteratively solved with the simulation program

once the other major parameters are determined. This

approach is only recommended as a last resort. To solve for

the infiltration rate and/or the ventilation rate iteratively, conduct

a series of simulation runs such that only the infiltration

and/or ventilation rates are changed from one-tenth to as

much as ten times the expected rates. Next, compare the simulation

outputs produced to the measured building data as discussed

below. In addition, supporting evidence should be

used to justify the final choice of variables.

I think calibrated simulation should be treated as an extension of retrofit isolation approach, and site measurements performed in support of simulation must at minimum include (a) parameters that are modified to model ECM savings, and (b) simulation inputs that do not change but drive ECM savings. For lighting fixture replacement ECM, pre-retrofit wattage would be an example of (a), and lighting runtime hours would be an example of (b). On the other hand, if there are no ECMs that reduce base load (e.g. plug load, lighting, etc.), but there are ECMs involving HVAC controls, site measurements should focus on operation of existing controls, and establishing existing lighting wattage is not necessary because model calibrated to capture the overall base load (lighting + plug loads) usage may be good enough in this case. We had to develop guidelines for incentive programs that rely on calibrated simulation to specify minimum scope of measurements and acceptable ?estimated? inputs for each common ECM type. I hope that a similar guidance would be included at some point in one of ASHRAE standards / guidelines. As it stands now, there is a sea of difference between the specificity of simulation requirements in 90.1 Appendix G compared to calibrated simulation of existing buildings. Unless I am missing some key reference?

Maria

--

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net

Phone 860.430.1909

41C New London Turnpike

Glastonbury, CT 06033

Since we have your ear Jeff, I have wondered if Guideline 14 describes any penalty to apply to the goodness of fit metric when allowing unknown parameters to be varied to improve the fit to monthly data? Something like how adjusted R-squared or AIC penalize models with more parameters? It seems that if you achieve a given level of goodness of fit after fiddling with a bunch of parameters then that is not as good as achieving the same level of fit with fewer free parameters. Is there any way to estimate the number of degrees of freedom lost when changing a single numeric model parameter like ACH vs a more complicated one like changing out entire HVAC curves?

Dan