Bayesian Rules of Thumb: Robust Uncertainty Quantification in Early Project Cost Estimation
Systems engineers often make use of cost Rules ofThumb in order to estimate cost during early phases of projectformulation. These Rules of Thumb typically take the form ofa sequence of percentages over which a total cost is allocatedacross NASA WBS elements. Rules of Thumb can then be usedto extrapolate cost from one or more known WBS elements tothe remaining unknown WBS elements, assisting early projectformulation architecture studies (such as those in JPL’s Team Xand A Team).A number of issues can arise when generating and using costRules of Thumb. For example, many records of project costsconsist of incomplete data. Typical methods of dealing withincomplete cost allocation data include (a) ignoring missionswith incomplete data, or (b) taking averages of the non-zero percentagesacross missions, but both of these methods can result inbiased estimates if the existence of incomplete data correlateswith total mission cost or any particular WBS element. Anothercommon example is cost reported in one or more incorrect WBSelements. This is especially prevalent in smaller missions whereit is more common for engineers to perform tasks that fall underthe purview of multiple WBS elements.Furthermore, a Rule of Thumb estimate is typically reported asa point estimate; there is no reported uncertainty around thepercentages used to generate an allocation. Even in the rarecase in which confidence intervals around mean percentages areprovided, there may be positive or negative correlations betweenWBS elements which can skew estimates.Here we attempt to address these problems by formulatingprobabilistic Rules of Thumb in which a distribution of allocationschemes, rather than a single allocation scheme, is generated.We use a bootstrap imputation method to simultaneouslyaccount for uncertainty in the missing data while using allavailable information contained in the dataset. The imputeddatasets are then input into a multivariate Bayesian modelwhich accounts for correlations between WBS elements andproperly accounts for uncertainty in the final Rule of Thumbpercentages and predictions. We describe the mathematicalmodel and provides snippets of R code utilizing the brms(Bayesian Regression Models using Stan) package. To illustratethis model, we generate a Bayesian Level 2 WBS Cost Rule ofThumb for MIDEX (Medium-Class Explorers) missions withdata extracted from NASA’s CADRe. We then compare thismethod’s performance with the classical Rule of Thumb method.