AI Investment Backtesting: A Data Leakage Test Harness

An AI investment backtest is credible only if every feature, label, universe member, and trading assumption was available at the simulated decision time. Otherwise, the model may be learning from the future.

This guide provides a point-in-time data contract, walk-forward validation design, cost model, and ten-test harness for detecting common failures. It is a research-control framework—not a trading recommendation or evidence that a strategy will earn returns.

Educational information only. Backtests are hypothetical, omit real-world frictions, and cannot guarantee future performance. Investment decisions can result in loss.

Why AI investment backtests fail

Machine-learning pipelines add failure paths to familiar backtest problems:

• revised fundamentals overwrite values originally known to the market;
• delisted securities disappear from the training universe;
• features are normalized using future observations;
• overlapping labels leak information across train and test windows;
• hyperparameters are selected after repeated testing on the same period;
• a language model uses knowledge published after the decision date;
• turnover, spread, market impact, borrow cost, or rejected orders are omitted.

The CFA Institute Research Foundation’s AI and big-data handbook emphasizes that investment use of machine learning still depends on sound data, economic reasoning, and validation. Recent research on temporal leakage in financial language models further shows why model knowledge dates must be treated as a data boundary.

Step 1: write a point-in-time data contract

Do this before feature engineering.

```yaml decision: timestamp: “market_close_t” action_earliest: “next_tradable_event” rebalance_frequency: “monthly”

universe: membership_source: “point-in-time security master” include_delisted: true liquidity_rule: “defined before test”

features:

• name: “example_feature” source_release_timestamp: required first_observed_timestamp: required revision_policy: “use value known at decision time” transformation_fit_window: “training only”

labels: horizon: “precommitted” overlap_embargo: “defined from horizon”

execution: price_rule: “first price actually available after decision” costs: [“fees”, “spread”, “slippage”, “market impact”, “borrow”]

model: training_cutoff: “strictly before test window” pretrained_knowledge_cutoff: documented prompt_and_version: immutable ```

The contract turns “no look-ahead bias” into testable fields. A dataset without release timestamps cannot prove point-in-time integrity merely because it contains historical dates.

Step 2: separate research, validation, and final test

A single chronological split is better than random shuffling, but it does not control repeated researcher decisions.

Use three roles:

1. Research windows: build features and candidate models.
2. Validation windows: select settings under a walk-forward protocol.
3. Locked final window: run once after the method and thresholds are frozen.

For each walk-forward fold, train only on observations available before the validation start. Fit scalers, encoders, feature selection, and imputers inside that fold. If outcomes overlap in time, purge overlapping samples and use an embargo appropriate to the label horizon.

```text Fold 1: [train------------] [embargo] [validate] Fold 2: [train--------------] [embargo] [validate] Fold 3: [train---------------] [embargo] [validate] Locked: [final test] ```

Do not let a model summarize or retrieve post-cutoff documents. A citation date is not enough if the underlying model already encodes later events.

Step 3: model transaction costs and implementability

At minimum, calculate:

```text Net strategy return = gross position return

• commissions and fees
• bid/ask spread
• slippage and market impact
• borrow and financing costs
• taxes, when relevant to the stated use ```

Costs should depend on turnover, instrument, liquidity, order size, and market conditions where data permits. Test a base assumption and adverse cases rather than using one optimistic constant.

Execution timing must also match reality. A signal calculated from a closing-price file cannot transact at that same close unless the complete input and order were actually available beforehand.

Step 4: run the ten-test leakage harness

Test	Procedure	Failure signal
Timestamp audit	Compare event, release, ingestion, and decision times	Feature available only after decision
Revision audit	Rebuild with first-known rather than latest values	Material result collapse
Universe audit	Restore delisted and acquired securities	Performance depends on survivors
Pipeline-fit audit	Fit transforms inside each training fold	Test-period information influenced transform
Label-overlap audit	Purge/embargo overlapping horizons	Performance vanishes after separation
Model-knowledge audit	Document pretraining/retrieval cutoffs	Model may know future narrative or events
Placebo-label test	Shuffle labels within valid blocks	Model still appears predictive
Feature-delay test	Delay every feature by one realistic period	Implausible sensitivity reveals timing error
Cost stress test	Increase spread, slippage, and impact	Edge is smaller than plausible friction
Subperiod test	Evaluate regimes and instruments separately	Result comes from one narrow episode

Automate each test and save pass/fail evidence. A verbal “we checked leakage” is not reproducible.

Step 5: require a baseline and ablation study

Compare the AI strategy with simple alternatives under the same universe, dates, costs, and rebalance rules:

• cash or an appropriate passive benchmark;
• a naive historical mean;
• a simple linear or rules-based model;
• the same pipeline with each feature family removed;
• the same portfolio construction using random or delayed signals.

The question is not whether a complex model produces a positive chart. It is whether complexity adds stable information after identical constraints and costs.

For research conclusions, convert results into a reviewable AI investing decision card rather than a buy/sell command.

Step 6: precommit the acceptance gate

Define thresholds before viewing the locked test. A defensible gate can require:

• every temporal and universe test passes;
• performance remains after the adverse cost case;
• improvement is present across multiple walk-forward folds;
• no single security, short period, or feature dominates the conclusion;
• economic rationale survives an ablation study;
• capacity and operational constraints are documented;
• an independent reviewer can reproduce the run from frozen inputs.

Use uncertainty intervals and distributions, not one headline metric. If many models or feature combinations were tried, disclose the search process and adjust how much confidence you place in the winner.

A reproducible evidence pack

Store:

1. data manifest with vendor, version, hashes, timestamps, and licenses;
2. universe membership and corporate-action rules;
3. feature/label definitions and source-availability logic;
4. code commit, environment lockfile, seed policy, and model version;
5. every attempted configuration, not only the winner;
6. fold-level predictions, positions, orders, costs, and failures;
7. leakage-harness results;
8. reviewer comments, overrides, and final decision.

If language models contributed qualitative evidence, apply a separate portfolio-commentary validation workflow and retain claim-to-source mappings.

Regulatory and fraud caution

The SEC, FINRA, and NASAA investor alert warns that AI-related claims are used in investment fraud. The SEC’s predictive-data-analytics materials also illustrate regulatory concern where technology may place firm interests ahead of investors.

Do not market a backtest as guaranteed, cherry-pick favorable periods, or imply regulatory approval. Document material limitations prominently.

Frequently asked questions

What is data leakage in an investment backtest?

It occurs when training or simulated decisions use information that would not have been available at the historical decision time.

What is walk-forward validation?

It repeatedly trains on earlier data and validates on the next chronological window, mimicking how a strategy would be updated through time.

Why is random train-test splitting dangerous for financial data?

Financial observations are time-dependent and labels can overlap. Random splits can place related or later information in both training and test sets.

How do you test an LLM investment strategy for temporal leakage?

Control document dates, retrieval cutoffs, model knowledge cutoff, prompt/version, and decision time; then test delayed inputs and unseen locked periods.

Which transaction costs belong in a backtest?

Include fees, bid/ask spread, slippage, market impact, borrow or financing costs, and any other friction relevant to the stated implementation.

Does a successful backtest prove an AI strategy works?

No. It is hypothetical evidence conditional on data and assumptions. Live behavior, market structure, costs, and relationships can change.