Hi Everyone - I'm glad to see a lot of discussion around this topic.  Remember that this is a "Request for Information".  One of the purposes of these is to see what folks think about the topic and how to help evolve the topic as something researchable (and
help clear up misconceptions or anything vague in the topic).  Given the variety of ideas/insights discussed in this thread, I'd encourage some of you to consider drafting a response to this RFI as I'm sure the program manager would appreciate your input on
the topic. I'm attaching a PDF of the full RFI as it provides more detail about what they're initially considering. As an example of what people could discuss, there is the issue of "confidence level" as described in the attached, and "confidence interval"
as used in meta-analyses.  "Confidence Intervals" are one of many sources that could inform what DARPA is calling "Confidence Levels". And the RFI also describes other potential examples of exogenous variables that might be included
in assessment of Confidence Levels.  Anyway, responses are not due until August 24th (see pp. 4-5 of the attached).  These are often a fun way to help make one's ideas more concrete, help bring together a new research team, and, quite possibly, have some influence
on future funding opportunities.  
Best,
Steve Fiore

Request for Information (RFI) 

DARPA-SN-17-57 Confidence Levels for the Social and Behavioral Sciences 

The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on new ideas and approaches for creating (semi)automated capabilities to assign "Confidence Levels" to specific studies, claims, hypotheses, conclusions,
models, and/or theories found in social and behavioral science research. These social and behavioral science Confidence Levels should rapidly enable a non-expert to understand and quantify the confidence they can have in a specific research result or claim's
reliability, reproducibility, and robustness.

https://www.fbo.gov/index?s=opportunity&mode=form&id=3f2885886c97d05491ff3ac8f7968912&tab=core&_cview=0

--------

Stephen M. Fiore, Ph.D.

Professor, Cognitive Sciences, Department
of Philosophy (philosophy.cah.ucf.edu/staff.php?id=134)

Director, Cognitive Sciences Laboratory, Institute for Simulation & Training (http://csl.ist.ucf.edu/)

University of Central Florida

sfiore@ist.ucf.edu

Author, e-mail: 
sfiore@IST.UCF.EDU
Author, name: 
Fiore, Steve