Artificial Intelligence-Driven Systems for Financial Forecasting

In recent years, machine learning algorithms have been successfully employed to leverage the potential of identifying hidden patterns of financial market behavior and, consequently, have become the norm in financial applications. This thesis proposes a statistical arbitrage trading strategy with two key elements: an ensemble of regression algorithms for asset return prediction, followed by a dynamic asset selection. More specifically, the extreme heterogeneity of the ensemble is achieved by ensuring model diversity using state-of-the-art machine learning algorithms, data diversity by using diverse input features, and method diversity}by using individual models for each asset, as well as models that learn cross-sectional across multiple assets. Moreover, the ensemble is constructed using a novel model selection approach. Then, the predicted results are fed into a quality assurance mechanism that prunes assets that have poor forecasting performance in recent history. However, relying blindly on machine learning algorithms for decision-making can have negative consequences, especially in critical areas such as finance. At the same time, it is well acknowledged that converting data into actionable insights can be a complex task. As a particular example, the practitioners in the financial domain, find it difficult to manage large quantities of data linked to an impressive number of stocks. Given these motivations, this dissertation introduces machine learning approaches based on eXplainable Artificial Intelligence techniques that are integrated into a financial forecasting and trading pipeline. Specifically, there are presented three strategies for excluding irrelevant features for the prediction task, with the goal being to increase the prediction performance not only at the stock level but also globally, at the stock-set level. For the proposed trading strategies, the analysis that was carried out reveals that the use of the feature selection approaches improve the portfolio performance. This is achieved by using only the predictive signals which are less noisy than the original content of the entire feature set while preserving enough information. To demonstrate the utility of the proposed approaches, their performance is evaluated in real-world scenarios, i.e., historical data of stocks composing the S&P500 index or custom stock-set combining various other indexes. The thesis contributes to the scientific literature in terms of results and the novel manner of exploiting machine learning in an algorithmic trading context.