Hi Everyone - there is another interesting DARPA RFI out continuing its theme of research on improving research in the social sciences.  This comes on the heals of DARPA's recent Ground Truth program (http://www.darpa.mil/program/ground-truth)
and the original program that is now titled Next Generation Social Science (http://www.darpa.mil/program/next-generation-social-science).
 A bit of the history is below in earlier emails (along with additional links).



Stephen M. Fiore, Ph.D.

Professor, Cognitive Sciences, Department of Philosophy (philosophy.cah.ucf.edu/staff.php?id=134)

Director, Cognitive Sciences Laboratory, Institute for Simulation & Training (http://csl.ist.ucf.edu/)

University of Central Florida


Request for Information (RFI) DARPA-SN-17-57 Confidence Levels for the Social and Behavioral Sciences
Office: Defense Advanced Research Projects Agency
Solicitation Number:  DARPA-SN-17-57
"Confidence Levels" for the Social and Behavioral Sciences 
The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on new ideas and approaches for creating (semi)automated capabilities to assign "Confidence Levels" to specific studies, claims, hypotheses, conclusions,
models, and/or theories found in social and behavioral science research. These social and behavioral science Confidence Levels should rapidly enable a non-expert to understand and quantify the confidence they can have in a specific research result or claim's
reliability, reproducibility, and robustness.


From: Fiore, Steve <sfiore@IST.UCF.EDU>
Sent: Saturday, April 8, 2017 12:18 PM
Subject: Finding Ground Truth...

Hi Everyone - The evolution of the social sciences marches onward.  DARPA just announced a new call in their line of research supporting the development of new methods and tools for understanding humanity.  The new call is "Putting
Social Science Modeling Through Its Paces" (http://www.darpa.mil/news-events/2017-04-07).  To put this in context, below is an email I sent
last year.  In that one I was trying to connect some of what DARPA is pursuing with the other meta-scientific work going on around the country where various people/organizations are trying to improve the process of science, writ large.  In brief,
in this larger research space, DARPA now has two programs designed to examine and improve research in the social sciences.  The first started last year and is titled "Next Generation Social Science" (see

 and http://www.darpa.mil/program/next-generation-social-science).  We now have
this new one focused on capabilities for improving model testing through simulated social systems.  Please share this with anyone you think might be interested.


Steve Fiore
Conference Chair, Science of Team Science 2017
June 12-14, Clearwater Beach, Florida

Putting Social Science Modeling Through Its Paces
New program seeks to develop simulated social systems of varying complexity against which to test the accuracy of social science modeling methods


From: Fiore, Steve
Sent: Thursday, September 8, 2016 3:19 PM
Subject: DARPA RFI "Forensic Social Science Supercolliders (FS3)"

Hi Everyone - there is a new Request for Information out from DARPA.  It's titled "Forensic Social Science Supercolliders (FS3) and can be found here (http://bit.ly/2csENve).
 I'm sharing it with the list for a couple of reasons. 

First, it is generally an interesting topic in that it is around the development of new approaches for 'doing' social science research.  Historically, it draws from the idea of "strong inference", but fits within the current context of the push for open science
and improving the way we do and verify science. This includes creating an infrastructure for open science, reproducibility, and registered replications as is being promoted by the Center for Open Science (see https://cos.io/).
 Relatedly, it fits with what is being done conceptually with the idea of "metascience" (see "Metascience could rescue the ‘replication crisis’" -- http://go.nature.com/2cJbWGr),
and methodologically with Stanford's "Meta-Research Innovation Center" -- http://metrics.stanford.edu/). I've
cut-and-pasted some of the relevant text from the RFI below, but, if you're interested in this topic, I encourage you to read it all.  

Second, I thought folks might also be interested from a science of science policy perspective, in that this appears to be an evolution of an RFI out last year.  That one was titled "New
Capabilities for Experimental Falsifiability in Social, Behavioral and Economic Sciences" (see http://bit.ly/2bVY9Gg).
 The PM had a very engaging workshop on that in the Fall of 2015 and the ideas around that topic were further developed and refined.  I find this new framing of "forensic social science", along with the idea of developing
simulation capabilities as an analog to supercolliders, to be quite intriguing.  So you can compare the two RFIs and see how it has evolved into this newer refinement/request.


Steve Fiore

Request for Information (RFI) DARPA-SN-16-70 

Forensic Social Science Supercolliders (FS3)i 

The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on new ideas, approaches, and capabilities for developing interactive simulations that can be used to calibrate the validity
of different social science research methods and tools in drawing “strong inference” about causal mechanisms that can lead to emergent complex behaviors in human social systems.


Today, social scientists are increasingly incorporating
simulation as a research method, particularly agent-based models (ABMs) and system-level simulations (which may include large-scale distributed online games). Where simulation is used currently, however, it generally appears as part of a larger collection
of social science research methods, usually as an early exploratory mechanism to help identify or refine hypotheses that can then guide further data collection and observation in the “real world.” Hence simulations currently also suffer from the above limitations
due to a lack of ground truth. Rather than simply being another research method, simulations – if advanced to a sufficient level of sophistication – might provide new capabilities for calibrating the inferential validity of social science research methods
in the first place. Indeed, DARPA hypothesizes that ABMs, system DARPA-SN-16-70, FS3 RFI 2 simulations, and games – rather than being complementary research tools – might provide initial capabilities as a kind of “social supercollider” in which other methods
and tools can themselves be tested. If successful, such simulations could provide testbeds in which social science research methods can be forensically evaluated for their capabilities and their limitations to correctly identify and characterize different
causal mechanisms and dynamics that give rise to observed complex behaviors and systems. Simulations might also engender new opportunities to test, calibrate, and explore a wide range of combinations of existing methods and tools, and potentially enable the
discovery of novel hybrid social science research methods with unique capabilities for correctly inferring causality. DARPA is referring to this potential capability as “forensic” for two reasons. First, if successful, the “supercollider” could support simulations
that, while artificial, allow for complex social behavior to emerge from relatively simple first principles, where these first principles are known because they were coded into the simulation from the beginning. Provided these simulations are sufficiently
sophisticated, they will enable the testing and evaluation of inferences derived from different social science research methods against ground truth with precision and certainty almost never available in the “real world”. Second, this kind of social science
supercollider could allow simulations of sudden, disruptive changes in key parameters or system behaviors (phase transitions, tipping points, etc.) in order to further calibrate the accuracy of different methods to correctly infer “what really happened” to
cause the observed simulated behaviors. 

From: Fiore, Steve
Sent: Thursday, September 3, 2015 8:44 PM
Subject: RFI out from DARPA on… "New Capabilities for Experimental Falsifiability in Social, Behavioral and Economic Sciences"


Hi Everyone - I just got notice about an interesting development.  It looks like DARPA is considering funding empirical work with a Popperian philosophy of science twist.  So I thought
some folks on this list might be interested
in contributing to this RFI as it could really benefit from expert input.


Stephen M. Fiore, Ph.D.

President, INGRoup 

INGRoup 11th Annual Conference

July 14-16, 2016 - Helsinki, Finland


RFI: New Capabilities for Experimental Falsifiability in Social, Behavioral and Economic Sciences
"The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on and suggestions
for research to develop novel methods, including new tools, platforms, techniques, and/or approaches, that could contribute to the development of unprecedented capabilities for testing the experimental falsifiability of (i.e., disconfirming) models, theories,
and hypotheses in social, behavioral and economic (SBE) sciences."

Here is a link to the RFI:  https://www.fbo.gov/index?s=opportunity&mode=form&id=c5f4210eb10ece263d3510f99472badc&tab=core&_cview=0

Author, e-mail: 
Author, name: 
Fiore, Steve