The question behind the screenshot
An AI trading agent posts a backtest: a large language model (LLM) read the news and the charts, made the calls, and the equity curve went up. Maybe you built the agent yourself with a coding assistant. Maybe another agent is about to recommend it to you, or act on it for you.
The question is not “is this AI clever?”. It is narrower and easier to answer: if someone else runs the same procedure, do they get the same result, and does it hold outside the window it was shown on? That is what re-verifying means. This guide shows what happened when researchers asked that question of published AI trading agents in 2025 and 2026, then gives the steps to ask it yourself.
Short answer: re-verifying an AI agent’s backtest
- Get the rules out as rules. Entries, exits, position size, fees, coins, timeframe, dates. If the agent cannot state its decisions as rules, a backtest cannot be re-run, only forward-tested.
- Reproduce first. Run the same rules on the same window with the same costs. If you cannot get the published number back, stop there.
- Move the window. Run it again on a longer window, and on windows before and after. A result that only exists on one stretch of dates belongs to those dates.
- Use data the model could not have seen. An LLM trained on text up to some date may already “know” what markets did before that date. Test after the model’s training cutoff.
- Count the tries. Ask how many prompts, models or parameter sets were tried before the one shown. The best of fifty is not one test.
- Price the execution. LLM agents can take minutes to decide. Use realistic fees, slippage, funding and fill timing, not a perfect fill at the close.
- Then go forward. Paper trade or demo trade after the rules are frozen, and compare with the backtest over the same dates.
The rest of this guide explains why each step is there, using what outside studies measured.
What happened when researchers re-ran AI trading agents
These are other people’s studies. Each one has its own market, period and method, and we state them as the authors report them. Most were run on US stocks, not crypto, which we mark each time.
Two more months changed the sign (stocks)
FINSABER (arXiv:2505.07078, version 6, 2026-06-26, published in the ACM SIGKDD conference proceedings) is a backtesting framework built to re-test LLM investing strategies. In its Appendix D the authors put the figures originally reported for the LLM agent FinMem, over 2022-10-06 to 2023-04-10, next to figures for the same agent over a window two months longer, 2022-10-05 to 2023-06-10 (the longer-window LLM figures are taken from a later paper, Yu et al. 2024):
- MSFT: cumulative return 23.261% reported → −22.036% on the longer window; Sharpe ratio 1.440 → −1.247.
- NFLX: Sharpe ratio 2.017 → −0.478.
The same paper ran timing strategies over 20 years and 100+ stocks and split the results by market regime. Sharpe ratios as reported: FinAgent 0.12 in bull markets and −0.38 in bear markets; FinMem −0.19 and −0.97; Buy and Hold 0.61 in bulls, 0.48 in sideways markets and −0.28 in bears. The authors’ summary is that previously reported LLM advantages “deteriorate significantly” under a broader cross-section and a longer evaluation.
What this shows: a short window can carry a result. It does not show that LLM agents cannot trade; it shows that a short-window number, on its own, does not tell you which.
Backtest, paper and real money on a crypto exchange
“Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets” (arXiv:2609.34510, version 1, 2026-09-28) is a preprint, so it has not been through peer review yet. It is crypto, and it ran on OKX: 10 large-cap coins, the same taker fees in the backtest as on the exchange, and three stages for machine-learning, reinforcement-learning, LLM and agent methods:
- Backtest on 2025 data, after training on 2021 to 2023 and selecting on 2024.
- Paper trading on the exchange from 2026-06-04 to 2026-09-21. Over the same dates, the authors report large gaps between the backtest and the paper result for the same method, in both directions: 10 of the 14 methods did better in paper trading than in the backtest. For some methods the sign of the conclusion changed, for example from a negative backtest return to a positive paper return. The split also ran by family: all eight machine-learning methods were profitable in the 2025 backtest and two of them stayed profitable in paper trading, while none of the six reinforcement-learning, LLM and agent methods were profitable in the backtest and all six were in paper trading.
- Real-money trading from 2026-08-24 to 2026-09-21 for four methods: three machine-learning models (LSTM, TRA, XGBoost) and one LLM (Qwen). The authors report that one of the four, Qwen, ended with a profit, that none beat holding Bitcoin over that period, and that all four had negative alpha.
Two cautions from the numbers themselves: the live stage is four weeks and four methods, only one of them an LLM, a small sample; and the paper-trading gap ran in both directions (10 of 14 methods higher in paper than in the backtest). The lesson is not “backtests are too optimistic”. It is that the backtest did not predict the paper or live result well enough to stand in for it.
The same paper measured how long each method took from decision to execution in paper trading. As reported: about 14.6 seconds for an XGBoost model, against about 501 seconds for FinAgent and 545 seconds for FinMem. A backtest that fills an LLM agent at the price of the bar it decided on assumes those minutes away. The authors add a caution of their own: across methods, the larger execution discrepancies did not line up with larger backtest-to-paper gaps, so delay alone does not explain the differences.
A leaky strategy can pass the statistics (stocks and ETFs)
“What survives honest evaluation?” (arXiv:2608.27734, 2026-08-27, a preprint) built a system where an LLM searches for trading strategies and every attempt is recorded. Two findings, as the authors report them, on a 453-stock US universe and a 39-ETF multi-asset universe with transaction, impact and borrowing costs:
- A deliberately leaky strategy, one that could see the future, posted a Sharpe ratio of 35 and still passed the Deflated Sharpe Ratio and probability-of-backtest-overfitting tests. Statistical corrections did not catch this leak; the authors argue the data setup has to.
- Once performance was deflated by the number of strategies the search had tried, none of the LLM-discovered strategies kept an out-of-sample edge the authors could certify, across two frontier models, up to 100 candidates and five repeated runs. Passive benchmarks did.
The model may remember the past (stocks)
An LLM trained in 2024 may already have read about 2020. “Summoning the Oracle to Slay It” (arXiv:2605.24564, version 2, 2026-08-29; listed as EMNLP 2026 main conference) calls this parametric look-ahead bias. Across five 7B to 14B models and five mega-cap stocks, reducing the model’s use of memorised outcomes changed the in-sample return by as much as −67.1% (the largest model-level mean the authors report). A backtest that runs an LLM over years it was trained on can be measuring its memory.
The fill rule can reorder the agents (synthetic data)
“How Execution Assumptions Change Short-Horizon Sharpe Rankings” (arXiv:2610.05077, 2026-10-04, a preprint) traded five LLM policies and seven classical baselines on the same synthetic price paths under six execution settings. On a two-asset board in the high-volatility regime, the ranking under near-ideal fills and the ranking under default stress agreed only weakly (Kendall τ_b 0.21, against 0.82 in the calm regime, with wide, overlapping intervals; with ten assets, agreement was higher). The authors’ point: on a short horizon, the fill convention can become part of the headline.
Four ways an AI agent’s backtest goes wrong that a standard checklist can miss
A general backtest checklist, like the seven questions for any strategy, still applies. LLM agents add four problems of their own.
1. Memorised history. Test windows that sit before the model’s training cutoff can reward memory rather than reasoning. Ask for the model name, its cutoff date and the test dates. Prefer results on data after the cutoff.
2. Code the agent wrote. If an AI assistant wrote the backtest, check the indexing yourself. In a 2026-02-05 question on Quantitative Finance Stack Exchange, a developer described asking a coding assistant for signals from a rolling window ending at the previous bar; the code ran cleanly and quietly included the current bar, a look-ahead bias. As they put it, you often cannot tell from the backtest result alone.
3. Untracked tries. An agent can try many prompts, models and settings in minutes. If only the best run is published, the result carries a selection effect: in the strategy-search study above, the bar a result had to clear rose with each try faster than the best in-sample Sharpe ratio did. Ask for the full count, including the runs that were dropped.
4. Decision time and fills. An agent that waits minutes for a model response does not trade at the bar it read. Ask when each order was filled, at what price, with what fee and slippage, and, on perpetual futures, what funding was paid.
How to re-verify with the same procedure
“Same procedure” means anyone can repeat the steps and compare. In practice:
- Write down the inputs before you run anything: rules, coins, timeframe, start and end dates, fee per side, slippage model, funding. Save that list with a date. It is your pre-registration, even if it is only a text file.
- Reproduce the claimed window. Same inputs, same window. Note any difference and its cause before you go further.
- Extend and shift the window. Add months on both sides, then run separate windows. Keep every result, including the ones that look bad.
- Run after the model’s cutoff if the strategy depends on an LLM’s judgement.
- Change the costs one at a time. Raise slippage, delay the fill by one bar, add funding. See which assumption the result depends on.
- Freeze and go forward. Paper or demo trade the frozen rules, then compare the forward result with a backtest over the same dates.
An AI agent can do all of this programmatically if the source gives it machine-readable inputs, dates and versions. If it does not, the agent should report the result as not re-verified rather than repeat it.
Doing it on PRUVIQ, and what it cannot do
PRUVIQ is one place to run the rule-based part of this procedure. It is not the only one; an open-source backtester you control works too.
- Re-run the rules on any window. Enter an agent’s entry and exit rules in the strategy builder, pick the coins, timeframe, and start and end dates, and run it. Every run applies trading fees, a slippage model tiered by market-cap rank (smaller coins cost more; the tier uses today’s rank, which the methodology page lists as a known limit), and funding for every coin that has funding-rate history. Then move the dates and run it again.
- Let an agent run it. The same simulator is available through the public API, so an agent can submit the same rules on several windows and compare the outputs itself.
- Re-check our own records. Strategy records, verification records with their criteria version and pinned engine and data, and a re-verification script are listed on the page for AI agents and developers. How each number is produced, and the biases we know about, are on the methodology page.
- Fix the rules before the data exists. Our forward tests are pre-registered with SHA-256 hashes and OpenTimestamps proofs; the checking steps are on /verify/forward/.
What it cannot do:
- It re-runs rules, not an LLM’s bar-by-bar reasoning, and it does not replay the news an agent read. An agent whose decisions cannot be written as rules has to be judged on a forward record.
- It covers OKX USDT perpetual futures only. It says nothing about stocks, spot markets or other exchanges.
- Its coin universe is the pairs listed on OKX today, so coins delisted along the way are missing (survivorship bias).
- A backtest here is still a backtest. The crypto study above is a reminder that paper and live results can differ from it in either direction.
What a clean re-run does not prove
A strategy whose result holds across windows, after its model’s cutoff, with honest costs and a counted number of tries, can still lose money next month. Markets change regime. What the re-run buys you is that the result was not a single window, a memory, a coding slip, a lucky try or a perfect fill. That is a reason to paper trade with small size, not a reason to skip it.
Questions people ask
How do I know if an AI trading bot’s backtest is real? Ask for the rules, the inputs and the dates, then reproduce the number yourself on the same window. If you cannot reproduce it, it is a claim. If you can, extend the window, test after the model’s training cutoff, and ask how many runs were tried before this one.
Do LLM trading agents work in crypto? The evidence so far is thin and mixed. In the 2026-09-28 crypto preprint above, backtests did not predict paper or live results reliably. In paper trading all six LLM, agent and reinforcement-learning methods ended positive. In the four-week real-money stage, the one LLM among four methods (Qwen) was the only one to end with a profit, and none beat holding Bitcoin. One LLM over four weeks is too small a sample to settle the question either way.
What is look-ahead bias in an LLM backtest? Using information that was not available at the time of the decision. With LLMs it has two forms: ordinary data leaks in the backtest code, and the model’s own memory of events in its training data. The second needs test data from after the model’s cutoff.
Can my AI agent re-verify a strategy before it recommends it? Yes, if the source publishes machine-readable inputs, as-of dates and versions, and a way to re-run. The agent should fetch them, re-run or reproduce, compare, and say “not re-verified” when it cannot. The fields to require are listed in our verification checklist.
Sources, as published: FINSABER, arXiv:2505.07078 v6 (2026-06-26), Appendix D and regime analysis; arXiv:2609.34510 v1 (2026-09-28); arXiv:2608.27734 v1 (2026-08-27); arXiv:2605.24564 v2 (2026-08-29); arXiv:2610.05077 v1 (2026-10-04); Quantitative Finance Stack Exchange question 85456 (2026-02-05). Publication status as listed on arXiv on 2026-10-26. The oldest figures here are FINSABER’s FinMem windows, which end in 2023 and are older than six months.