Can We Lastly Use ChatGPT as a Quantitative Analyst?
In two of our earlier articles, we explored the concept of utilizing synthetic intelligence to backtest buying and selling methods. Since then, AI has continued to develop, with instruments like ChatGPT evolving from easy Q&A assistants into extra complicated instruments which will assist in creating and testing funding methods—not less than, in keeping with among the extra optimistic voices within the subject. Over a yr has handed since our first experiments, and with all the present hype across the usefulness of huge language fashions (LLMs), we consider it’s the best time to critically revisit this matter. Subsequently, our objective is to judge how effectively immediately’s AI fashions can carry out as quasi-junior quantitative analysts—highlighting not solely the promising use circumstances but additionally the constraints that also stay.
Mannequin choice
First, we wanted to pick out a mannequin appropriate for the duty. We explored the choices of utilizing Claude AI, Gemini Superior (previously Deep Analysis), and ChatGPT, as these are among the most generally used AI instruments immediately. Progress within the AI fashions goes actually quick; a few of them are higher in selective sub-tasks, and others are worse; nonetheless, from our perspective, now we have not seen vital variations between them. Subsequently, primarily based on our wants – knowledge imputation, code interpretation, and reasoning, we selected ChatGPT as a major device wherein carried out our evaluation. When deciding which particular model to make use of, we chosen the GPT-4o mannequin, because it proved to be probably the most versatile total. We additionally thought-about the GPT-4.5 mannequin (which is marketed by the OpenAI as higher mannequin for analytical duties), however since it’s anticipated to be depreciated quickly, we felt this text wouldn’t provide lasting relevance if primarily based on it.
What we wish to accomplish
Because the title of this text suggests, our objective was to search out out whether or not the method of making a buying and selling technique will be assisted by AI, or if not the entire course of, then if not less than some a part of the method will be outsourced to the AI and and if we nonetheless can belief the outcomes. For that, we determined to stay to the straightforward mannequin – we labored with ChatGPT and requested him to help us in creating an asset allocation technique utilizing three belongings – equities, mounted earnings and commodities.
Our checks had been carried out on knowledge from 07.07.2015 to 17.04.2025 for SPY (SPDR S&P 500 ETF Belief), IEF (iShares 7-10 Yr Treasury Bond ETF) and DBC (Invesco DB Commodity Index Monitoring Fund) as funding universe.
First iterations
When the information had been ready (we bumped into some points, however we are going to summarize them later), implementing a easy buying and selling technique, like fixed-percentage allocation, was a comparatively simple activity. Easy methods contain assigning a set portion of capital to totally different belongings, no matter market situations. For instance, you would possibly allocate 60% to shares, 30% to bonds, and 10% to commodities. In code, this simply means multiplying every asset’s return by its goal weight and summing them as much as get the portfolio return. You don’t want complicated indicators or dynamic rebalancing, simply primary arithmetic operations on time sequence knowledge. This sort of technique is good for the beginning of AI automation and testing as a result of the logic is simple and will be utilized constantly over the dataset.
The AI mannequin additionally does slightly bit extra. Not solely can it write code for such a primary technique, however it could recommend a few of them by itself. Subsequently, we began with a naive technique and requested AI to recommend us modification of allocation ratios, that are rational and cheap and recommend us methods, which can be extra worthwhile when it comes to returns, Sharpe ratio and Calmar ratio.
Ideas
After working the fundamental mounted asset allocation methods and checking their efficiency, the subsequent step was clear: can we do higher? It’s one factor to create a easy portfolio with mounted weights, however markets are not often that cooperative. So we requested ChatGPT not simply to check the naive technique (and variations) but additionally to assist provide you with cheap modifications which may enhance the outcomes with out making the entire thing overly sophisticated.
That is the place issues get extra fascinating. As a substitute of simply assigning static weights, we explored small variations: what occurs if we shift a bit extra into bonds throughout tough intervals or barely enhance fairness publicity in sturdy uptrends? We intentionally prevented leaping into complicated machine-learning fashions or regime-switching strategies. The objective right here was modest – introduce simply sufficient construction to replicate real-world pondering, like adapting to current efficiency or volatility. ChatGPT may deal with that, (as soon as once more, not with out issues), however in the long run, it was in a position to recommend methods to re-weight the portfolio or apply primary filters to keep away from main drawdowns. Because of these prompts, we obtained the next fairness curves:
Combining and optimising
As soon as we noticed that energetic asset allocation methods may enhance efficiency, the subsequent problem was to discover a extra balanced technique – one which not solely performs effectively on paper but additionally feels strong and smart. It’s simple to get caught up in tuning parameters and selecting the most effective interval for indicators to squeeze out a barely greater Sharpe ratio, however there’s all the time a trade-off. A method that appears nice in a single interval would possibly collapse in one other.
To discover this, we requested ChatGPT to assist us take a look at totally different variations of the technique by adjusting key parameters – in our case, largely timeframes. The concept wasn’t to blindly optimize for the most effective outcome however to grasp how delicate the technique is to adjustments. If small shifts in a parameter result in large swings in efficiency, that’s a crimson flag.
Remaining iteration of the Asset Allocation Technique In line with ChatGPT is as follows:
Described technique has the next properties:
And these are fairness curves of steered energetic methods and ultimate technique (brown):
And right here is the results of the AI performing the robustness checks to make it possible for the parameter home windows we used, like lookback intervals or rebalancing intervals, weren’t simply conveniently chosen values that occurred to provide distinctive outcomes by probability.
What went good
To this point, it appears a contented story, proper? We requested ChatGPT for the technique, and in the long run, we obtained one. It’s undoubtedly a major improve after we examine the entire course of with the evaluation we carried out roughly 18 months in the past. ChatGPT orientates itself effectively in quant finance and might recommend plenty of variations for the asset allocation methods after which all the time provide you with strategies for the subsequent steps within the evaluation. The exploratory a part of the quant evaluation is well-handled. ChatGPT is an AI chatbot, and as such, it could talk plenty of concepts and focus on them eloquently.
Nonetheless, right here comes the catch – it’s nonetheless a chatbot, not an information analyst, and the chatbot’s major focus is to make you pleased with the “chatting.” What does it imply? It tends to be over-optimistic and sycophantic – it doesn’t “assume”, it solutions questions and tries to make you prepared to proceed within the dialog. Quite a lot of the time, ChatGPT introduced its concepts or evaluation and made extraordinarily naive errors in it; nonetheless, it introduced outcomes as the most effective technique/thought ever in existence. The fixed re-checking of the person steps within the evaluation was actually tiring.
What went improper
So, what had been the problems we encountered, and what do you have to take note of once you experiment with chatbots as assistants in quantitative finance?
Knowledge preparation
We encountered a couple of points when working with knowledge. Initially, we tried to acquire the information instantly from the web by way of ChatGPT, however that wasn’t possible-so we had to offer the information ourselves. This led to some sudden issues. Since we used dates within the format DD.MM.YYYY and numbers with a comma because the decimal separator, ChatGPT actually struggled to interpret the information appropriately. Probably the most dependable method turned out to be offering the information in a format that ChatGPT is extra acquainted with-typically utilizing YYYY-MM-DD for dates and a dot because the decimal level. Getting ready the dataset on this manner will make the interplay smoother and scale back misunderstandings throughout evaluation.
Knowledge corruption
After working a number of fashions on the inputted dataset, we skilled a couple of points. In some circumstances, the order of the information modified unexpectedly; in others, total sections of information had been misplaced. This led to outputs that had been clearly incorrect or inconsistent with what we anticipated. The outcomes appeared like this:
This difficulty is carefully associated to how reminiscence works when dealing with our knowledge. We continuously needed to re-upload the identical dataset, because it was both forgotten through the evaluation course of or turned corrupted in varied methods (and we didn’t perceive the rationale for corruption). This may make it tougher sooner or later to keep up consistency throughout checks and highlights the constraints of working with bigger datasets in this sort of setup.
In the long run, if you need to do your personal take a look at evaluation, we’d undoubtedly advocate offering a chatbot with your personal knowledge. As ChatGPT tends to make errors within the preliminary knowledge dealing with, for those who depend on the information from ChatGPT itself, you wouldn’t be capable to catch among the errors it makes.
Want for validation
When utilizing AI to create a technique, you typically wish to plot fairness curves, calculate primary efficiency metrics, and so forth. Nonetheless, the mannequin could interpret these duties in its personal manner, which doesn’t all the time match your expectations. Generally the problems are apparent at first look, however extra typically, it is advisable to examine the code fastidiously. The most typical errors often happen in knowledge formatting, the implementation of the technique operate, and the way returns, threat, and drawdowns are calculated.
One other associated difficulty is overpromising on the theoretical aspect whereas underdelivering within the precise code. This typically signifies that the mannequin describes, for instance, a technique consisting of three guidelines utilized to a dataset, however solely implements two of them. In our case, the technique was supposed to include momentum, volatility, and correlations. Nonetheless, correlations weren’t used within the implementation.
Hallucinations
Within the context of AI, it usually refers to when a mannequin generates info that’s factually incorrect or fabricated, despite the fact that it might sound believable.
ChatGPT
In our case, we had been exploring a number of methods directly and aimed to research simply the efficiency of probably the most profitable amongst them. This setup elevated the danger of errors going unnoticed-especially when the mannequin appeared to execute every step appropriately, however had truly skipped or misapplied components of the technique logic. With out cautious evaluate, these inconsistencies can result in deceptive conclusions a few technique’s effectiveness.
After we obtained the code for this technique and ran our personal evaluation, the outcomes we obtained had been considerably totally different.
After importing the information into the mannequin a second time, the outcomes it produced matched our personal. How the ChatGPT calculated higher ratios within the first time? And why had been they totally different? We don’t know.
This introduced us again to an necessary a part of the method – we (customers, people) must validate ends in every step of the evaluation. Regardless of how small or insignificant step it appears. It’s completely essential. ChatGPT typically produces completely made-up numbers (even when the code it suggests for calculation of these numbers is right).
Cyclic conversations
After we found errors within the calculated efficiency metrics, we wished to grasp why they occurred. After a couple of follow-up prompts, the mannequin circled round varied explanations-differences in knowledge, discrepancies within the construction of the technique, or changes to its parameters. Nonetheless, we identified (appropriately) that none of those utilized, since we had merely run the precise code supplied by ChatGPT on the identical dataset we had initially equipped. Even after asking the mannequin to re-run its code on the identical enter, we discovered ourselves in a loop, the place the AI continued to deflect the problem slightly than acknowledge or right the defective calculations. This expertise illustrates a key limitation of utilizing AI to debug or take a look at a technique: whereas it might appear assured, it doesn’t all the time reliably hint the basis of its personal errors.
If we take a step again and use AI only for brainstorming technique concepts, we could encounter the same difficulty. The mannequin typically will get caught on one primary idea and tends to construct every thing round it. For instance, if we start with a technique that includes deciding on the highest N belongings primarily based on a sure criterion, the mannequin could proceed to recommend solely variations that deal with this choice step as important. Until we explicitly state that we wish to keep away from utilizing that criterion, it’ll seemingly stay a core a part of each new proposal. This highlights a typical limitation: AI tends to anchor on the preliminary path and struggles to discover completely totally different concepts until firmly guided to take action.
Tendention to over-optimization
ChatGPT, as an analyst, tends to be an optimization machine. Ideas it offers, or concepts it presents as worthwhile to research have a tendency so as to add levels of freedom into the technique, and as such, the technique turns into an increasing number of over-optimized to the previous knowledge. ChatGPT doesn’t generalize effectively (as of now) and often picks the best-performing model of the technique after which appears out for the reason of why it’s the most effective and tries to enhance it much more. It’s logical (from the chatbot’s standpoint), nevertheless it’s not the most effective thought if you wish to construct a sturdy buying and selling technique. Subsequently, typically, ChatGPT’s strategies have a restricted worth, and it’s often higher to immediate it to proceed in several instructions than it suggests. All in all, it’s higher when a human is in cost than relying blindly on a chatbot throughout evaluation.
Conclusion
Synthetic intelligence is a robust device that may help with many duties. It’s good at suggesting top-down concepts, drafting code outlines for testing, and sometimes serving to you discover a new path once you’re caught on an issue. Nonetheless, there are a number of necessary limitations to bear in mind. As an illustration, you continue to must supply your personal knowledge for evaluation, fastidiously verify the code for lots of potential errors, and keep away from absolutely trusting the efficiency metrics (and even charts) printed by the mannequin with out verification.
Since our earlier article, AI has made vital progress. What it could do is assist automate components of the workflow and avoid wasting treasured time. Nonetheless, even with these developments, the potential for errors stays excessive. That’s a threat that must be calculated once you attempt to work with it. AI is a classical device, like a pointy knife – you can also make plenty of helpful issues with it, or for those who have no idea what you might be doing, then you possibly can lower your personal finger with it.
Authors: David Belobrad, Quant Analyst, QuantpediaRadovan Vojtko, Head of Analysis, Quantpedia
Are you in search of extra methods to examine? Join our publication or go to our Weblog or Screener.
Do you wish to be taught extra about Quantpedia Premium service? Test how Quantpedia works, our mission and Premium pricing provide.
Do you wish to be taught extra about Quantpedia Professional service? Test its description, watch movies, evaluate reporting capabilities and go to our pricing provide.
Are you in search of historic knowledge or backtesting platforms? Test our listing of Algo Buying and selling Reductions.
Would you want free entry to our companies? Then, open an account with Lightspeed and luxuriate in one yr of Quantpedia Premium without charge.
Or observe us on:
Fb Group, Fb Web page, Twitter, Linkedin, Medium or Youtube
Share onLinkedInTwitterFacebookConsult with a buddy
