Governing Automated Strategic Intelligence

Frontier AI systems can now fuse satellite imagery, shipping records, social-media traces, and corporate filings into analyst-grade answers in minutes. This paper asks who that shifts power toward and what governance should do before the answer becomes obvious.

What automated intelligence is

Intelligence analysis has traditionally required human teams that spend weeks manually collecting reports, imagery, signals, and financial records, then reconcile contradictions across those sources before producing a finished assessment. The paper's central claim is that frontier multimodal AI systems can now automate a significant part of this process — not by replacing the analyst's judgment, but by compressing the collection and synthesis phases from weeks to minutes.

The paper defines automated strategic intelligence as a system capable of ingesting heterogeneous open-source streams, representing their contents in a machine-queryable form, retrieving relevant evidence in response to a strategic question, reasoning over that evidence with a language model, and delivering a structured assessment to a decision-maker with provenance attached.

The five-stage pipeline

A concrete example

To make this tangible, consider one of the actual tasks used in the study: estimating what percentage of Starlink terminals shipped to Ukraine showed active signals near the frontline during a specified period.

From the paper's introduction (adapted from Gupta, 2024): publicly available satellite imagery combined with object-detection models can expose strategically relevant asset positions without any classified access.

Exploratory uplift study

The empirical component is an exploratory study rather than a definitive experiment. The paper is transparent about this — the goal is to produce enough signal to motivate the governance argument, not to settle the empirical question.

Study design

Twenty novice participants and two skilled intelligence analysts worked on twelve intelligence-analysis tasks. Each task had a three-hour time limit. Novices were randomly assigned to either a control condition (no LLM) or a treatment condition (public LLM access permitted). Because most tasks lack clean public ground-truth answers, the outcome could not be a standard accuracy score.

The outcome measure

The paper defines an expert-similarity score: a normalised measure of how closely a participant's answer resembles the analyst-derived reference answer. If we denote the raw similarity for participant $i$ on task $j$ as $s_{ij} \in [0, 1]$, the score is normalised across tasks to have mean zero and unit variance:

This controls for task difficulty — a task where all participants score similarly contributes less variance to the estimate than one where scores spread widely. The normalised score is the dependent variable in all statistical models.

It is important to keep in mind what this measures. The study is not measuring whether LLM-assisted novices are correct in some objective sense. It is measuring whether they produce outputs that look more like what skilled analysts write. That is a meaningful proxy in a domain where ground truth is often unavailable, but it is not the same thing as accuracy.

Statistical model

The primary analysis uses a two-way fixed-effects model. Let $\text{LLM}_i \in \{0,1\}$ indicate whether participant $i$ was in the treatment group. The model is:

where $\alpha_i$ is a participant fixed effect absorbing individual baseline differences, $\gamma_j$ is a task fixed effect absorbing differences in task difficulty, $\beta$ is the treatment effect of interest, and $\varepsilon_{ij}$ is the error term. The fixed effects are the key design choice: they let the model compare each participant to themselves across tasks rather than relying on between-participant variation, which is helpful given the small sample.

The estimate of interest is $\hat{\beta}$, interpreted as the average improvement in expert-similarity score (in standard-deviation units) attributable to LLM access, after removing participant and task baseline differences.

Results

The primary fixed-effects estimate is $\hat{\beta} = 0.148$ with $t = 2.44$ and a one-sided $p = 0.036$. Two robustness checks are reported: a paired residual-difference test ($\hat{\beta} = 0.153$, $p = 0.025$) and a participant-stratified permutation test ($\hat{\beta} = 0.086$, $p = 0.219$ over 10,000 permutations).

The permutation test not reaching conventional significance is the main reason the paper frames this as preliminary evidence. All three estimates point in the same direction. Two of three are conventionally significant. One is not. The honest conclusion is that the signal is consistent and positive but not yet conclusive. A larger, pre-registered study with more participants and a wider task set would be needed to settle it.

Even at face value, a $0.148$ standard-deviation shift in expert-similarity is strategically meaningful if it scales. If public models on consumer hardware can compress part of the novice-to-expert gap without any specialised infrastructure, then the barrier to entry for capable intelligence analysis is already lower than most governance frameworks assume.

Strategic implications

The paper argues the shift matters in at least three ways, each of which follows from the pipeline structure rather than from the uplift estimate alone.

Policy recommendations

The paper proposes five governance areas that follow from the pipeline analysis rather than from the uplift study alone. The argument is that even if the empirical results are weaker than presented, the pipeline architecture makes these recommendations prudent.

A sixth operational recommendation cuts across all five: continuous quantitative benchmarking. Any state or institution deploying AI for intelligence synthesis needs institutions that routinely measure accuracy, calibration, hallucination rates, and adversarial robustness — not just capability demos under favourable conditions.

Limitations

References