BioE PhD Defense Presentation- Hyoann Choi

Advisor: Dr. Wilbur Lam, M.D., Ph.D. 

Committee Members: Dr. Anant Madabhushi, Dr. Kamran Paynabar, Dr. Eva Dyer, Dr. David Myers

 

CONFIDENCE: A System of Computed, Inspectable Reliability for Biomedical Conclusions

False discovery in biomedical research is a well-known problem, and what undermines a conclusion is usually something nobody thought to check. A conclusion can rest on a correctly computed yet biased, inflated or entangled statistic and still not be supported by the study that produced it. Methods and reporting guidelines exist for each individual threat, from a confounded design to an inadequate sample, a violated assumption or an unstable result. However, they are fragmented, and whether a conclusion is reliable is still decided by hand, differently by each analyst, and often without a record anyone can review. No convention says which of them must be checked or how thoroughly, so each researcher decides alone, based on their own best knowledge and educated assumptions. High-dimensional data carry more of these threats at once, while their complexity is what makes them harder to examine. 

In this study, I developed CONFIDENCE, a computational framework and software platform that operationalizes reliability assessment of biomedical analyses. It assesses whether the data are sufficient in both quality and quantity, which variables and relationships have to be accounted for, how the analysis should be specified, and whether the results are consistent and stable. Every check returns its own judgment with the evidence under it, and those readings aggregate into verdicts. Delivered as software, the assessment is automated, consistent and repeatable by a third party. 

Demonstrations establish that the assessment responds specifically and changes what a study may conclude. Simulated data carrying planted uncertainties show that the obliged checks fire on each one and stay quiet on the matched clean data. On an in-house human platelet cohort assessed before publication, the framework surfaces uncertainties a competent biomedical researcher would not have known to check. For a published human muscle aging atlas, the framework tells a reader what to consider to build on the published results. The same 

framework finds different obstacles in each study, showing that what limits a conclusion is a property of the study, to be judged on its own terms. Reliability becomes a known, routinely testable property rather than a matter of whichever analyst or reviewer happens to look.