Backtesting Alternative Data Signals: Does the Score Actually Work?

Any alternative data signal can be made to look brilliant in hindsight. The only way to know whether a signal actually carries an edge — rather than a compelling narrative — is to replay it forward over history, under the constraints a real strategy would have faced, and measure what it would have done. That replay is a backtest, and doing it honestly is harder than it looks.

Point-in-time, or nothing

The single most important rule: use the signal as it was known on each day, never as it was later revised. Alternative data is especially prone to look-ahead bias because signals get re-scored, sources get backfilled, and entities get re-linked over time. If your backtest sees today's cleaned-up score on a date in the past, it is cheating, and the results are worthless. This is why serious platforms store observations append-only and never overwrite history — so the backtest sees exactly what was knowable at the time.

Always measure against a benchmark

A strategy that returned 30% tells you nothing until you know what doing nothing returned. The right comparison for a signal-driven book is usually an equal-weight benchmark of the same universe: if the score adds value, the score-driven book should beat simply owning everything in the basket. The gap between them — the excess return — is the real measure of whether the signal works, because it strips out the market beta both share.

The metrics that matter

Total return is only half the story; risk is the other half. A complete read includes:

• Sharpe ratio — return per unit of total volatility. Above 1 is good; it tells you whether the return was earned or just borrowed from risk.
• Sortino — like Sharpe, but only penalizes downside volatility.
• Maximum drawdown — the worst peak-to-trough loss, i.e. how much pain you had to hold through.
• VaR and CVaR — the daily loss you would exceed 5% of the time, and the average loss on those worst days (tail risk).
• Calmar — return per unit of drawdown, which rewards smooth compounding over lucky spikes.

The overfitting trap

Sweep enough parameters — signal, thresholds, rebalance cadence, holding count — and you will always find a combination that looks spectacular in-sample. That is not an edge; it is curve-fitting to the past. Defenses: keep the parameter grid small and economically motivated, prefer configurations that rank well across multiple objectives (strong Sharpe and return, not just the single highest return), test on windows the optimizer never saw, and treat every backtested number as a hypothesis rather than a promise.

Honest by construction

The point of a disciplined backtest is not to produce an impressive chart — it is to earn the right to trust a signal. Point-in-time data, a fair benchmark, a full risk panel and a healthy fear of overfitting are what separate an alternative data strategy from a good story.

Backtest it live

Prismetric's Strategy Lab backtests every score-driven strategy on real price history — point-in-time, against an equal-weight benchmark, with the full metric panel — then tracks it live. See how it works, or read about the frontier signals going into the newest strategies.