The first experience with stock trading I had when my friend and I decided to create a company for algorithmic trading by using AI technology (it was called AlgoBrain). The software for data processing and training adaptive components (including ANN) was written in C++, and the part interfacing with the broker via its website was written in C#. The trading system was fully automated; it could function without manual intervention for days in a row.
When I was invited to work for a startup company that was developing a trading model for ultra-high frequency of trading, I brought in my expertise in statistics and created a few novel stock-specific indicators based on a book of public orders as well as special indicators based on stock clusters. I also developed and programmed (C++) a procedure for automatic identification of stable stock clusters. In addition, I developed a method for optimization of the trading model’s complexity with respect to minimization of overfit and reduction of drawdowns. Furthermore, I developed special, custom filters for reduction of noise in the input data on public orders and public trades. All those improvements helped to increase the statistical reliability of predictions on the out-of-sample period and improve the profitability considerably.
During my stint at Jefferies (in the department of strategic investments), I developed a trading model for an ultra-high frequency trading system. Since by that time (end of 2012) the usual auto-regression moving average-type models became much less profitable than before, I had to search for alternative trading models, and my search was successful. The solution I found was based on a special, fuzzy type clusterization of stocks and custom statistical indicators using those clusters. However, the administration decided to close the HFT direction after realizing that the cost of required infrastructure was higher than they expected.
During my employment with Migdal Research and later, Fresnel Research, in addition to market indicators, I developed more complex functional components that I call “sensors” (using my experience in AI solutions), which, besides input data processing, included sensor-specific trade decision making. This technology considerably increased robustness for the trading model. I also developed and programmed (using Python in combination with C++) routines for efficient optimization of trading model parameters.
After disintegration of Fresnel Research, I had time to work on pursuing my old dream, finding an AI-based “island of complexity” in the area of trading models. In the field of deep learning (ANN-based), it is well known that significant noise in the input data severely hinders attempts to increase the ANN complexity. Since the stock market is rather noisy, there is a conundrum: simpler models, e.g., of auto-regression moving average (ARMA) type are stable with respect to the noise but are not “smart” enough to capture the complexity of stock market dynamics, resulting in considerable drawdowns. And more complex models suffer from overfit. Therefore, it is very desirable to find a “safe island of model complexity” where the model is both sufficiently complex and sufficiently stable with respect to the noise. It took me a long time to search for such an island. I had to try different approaches and methods, each time making sure that the computer program implementing the method works correctly and optimizing the method’s parameters. Only recently I found what I was looking for. As I’ve been anticipating for a long time, the solution is based on an idea similar to AI approaches utilized for training a computer to play antagonistic games (such as Chess or Go). I’ve tested it on historical stock market data and look for an opportunity to employ it for real trading.