Hierarchical Multi-Agent System with Bayesian Neural Networks for Portfolio Optimization

Financial markets are inherently dynamic and complex, presenting significant challenges for portfolio management [1, 2]. Conventional models rely on static assumptions about asset returns and correlations, which often fail to capture the dynamic, non-linear behaviors of markets. These models typically struggle to adapt to sudden market shifts, making them suboptimal for decision-making in highly volatile environments [3, 4]. Moreover, as the volume of available financial data increases, traditional approaches often fall short in processing and utilizing these vast amounts of information effectively, limiting their predictive accuracy and adaptability. Recent advancements in machine learning, particularly reinforcement learning (RL), offer promising alternatives [5, [6]. Multi-Agent Reinforcement Learning (MARL) [7] has demonstrated its potential by enabling multiple agents to independently learn and interact in dynamic environments [8-10]. While MARL-based systems allow for better adaptability compared to traditional models, they often neglect the complexities of uncertainty and risk management in highly volatile markets. In addition, existing approaches frequently rely on Conventional Neural Networks (CNNs), which offer deterministic predictions that fail to account for the inherent uncertainty in financial forecasting. Furthermore, these systems lack mechanisms to prioritize critical information dynamically, limiting their ability to focus on the most relevant data amidst the increasing volume of financial information.

The Integrated Portfolio Strategist (IPS) system presented in this paper is a novel framework designed to bridge these gaps by combining a hierarchical multi-agent system with Bayesian Neural Networks (BNNs) within the Proximal Policy Optimization (PPO) framework. Unlike traditional models that rely on static assumptions or single-agent systems, the IPS dynamically adjusts portfolio allocations by leveraging a high-level agent to analyze macroeconomic trends and low-level agents to manage sector-specific asset groups. The use of BNNs [11] enhances decision-making by incorporating uncertainty [12, 13], providing the system with a probabilistic framework that allows it to account for market volatility and forecast errors. Attention mechanisms further improve the system’s performance by enabling agents to communicate and collaborate effectively, dynamically prioritizing the most critical macroeconomic signals and sector-specific trends. This combination of uncertainty-aware modeling and dynamic communication ensures robust adaptability in unpredictable market conditions. Moreover, the IPS system utilizes Genetic Algorithms (GAs) for hyperparameter optimization [14, 15], ensuring that the system remains responsive to market shifts without relying on manual tuning. This integration allows the IPS to adapt its configuration and optimize performance under different market conditions, an advancement over traditional systems that often require constant manual intervention or rely on preset parameters.

In comparison to existing portfolio optimization systems, the IPS offers several key advancements:

• Dynamic Adaptability: The hierarchical structure and multi-agent approach enable the IPS to respond to both macroeconomic signals and sector-specific dynamics, a significant improvement over static models.

• Uncertainty Modeling: The incorporation of BNNs allows the IPS to quantify uncertainty, providing a more reliable basis for decision-making in volatile markets.

• Enhanced Information Prioritization: Attention mechanisms enable agents to dynamically prioritize relevant data while facilitating inter-agent communication, enhancing collaboration and decision accuracy in complex and data rich environments.

• Enhanced Risk Management: The IPS demonstrates superior risk-adjusted returns and drawdown management, particularly in stress test scenarios like the COVID-19 market crisis, where traditional models often perform poorly.

• Automated Hyperparameter Optimization: The use of GAs automates the fine-tuning of the system, optimizing the allocation of resources across agents and sectors without human intervention.

Thus, the IPS system fills a critical gap in portfolio management by offering an adaptive, uncertainty-aware approach that balances risk and return more effectively in complex and volatile financial environments.

The IPS framework employs a MARL [7] approach to handle complex financial decision-making under dynamic and uncertain conditions. The architecture includes two types of agents: The high-level agent, which analyzes macroeconomic trends and provides strategic insights and the Low-Level agents that manage sector-specific asset groups, balancing group performance with overall portfolio objectives. The overall architecture of this system is illustrated in Figure 1.

1.png

Figure 1. The IPS framework with the high-level agent generating macroeconomic insights and the three low-level agents managing sector-specific decisions for portfolio allocation and rebalancing

3.1 Hierarchical agent structure and decision making

The hierarchical structure of the IPS system is modeled using the Hierarchical Markov Decision Process (HMDP) framework, a formal extension of the standard MDP applied in Hierarchical Reinforcement Learning (HRL) [18]. HMDPs have been widely used to capture multi-level temporal abstraction in decision-making processes by decomposing a global policy into sub-policies at different hierarchies of control [19]. In this formulation, each decision-making level corresponds to a semi-Markov process operating at its own temporal resolution. The high-level policy selects abstract actions (e.g., strategic allocations), which invoke lower-level policies that carry out more granular decisions (e.g., asset reallocations). In our framework, the high-level agent operates over a slower time scale and captures macroeconomic signals to make strategic portfolio-level decisions. In contrast, low-level agents act more frequently within their group-specific environments, executing sectoral or asset-level actions that refine the high-level strategy. Formally, each agent solves an MDP defined by the tuple ⟨S, A, P, R, γ⟩, with state and action spaces tailored to its decision level. The high-level agent uses macroeconomic state space S^macro and action space A^macro, while low-level agents operate over group-specific spaces S^group and A^group. The overall IPS policy is modeled as a joint hierarchical policy:

$\begin{gathered}\pi\left(s_t\right)=\pi^{\text {macro}}\left(a_t^{\text {macro}} \mid s_t^{\text {macro}}\right). \\ \pi^{\text {group}}\left(a_t^{\text {group}} \mid s_t^{\text {group}}, a_t^{\text {macro}}\right)\end{gathered}$           (2)

This formulation ensures coordinated decision-making across both strategic and operational levels, enabling the IPS to integrate macroeconomic insights with sector-specific portfolio actions.

3.1.1 High-level agent: Macro-economic signal processing

The high-level agent in the IPS framework uses PPO [20] to update its policy in a stable and reliable manner. PPO constrains the magnitude of policy updates while maximizing long-term returns, making it well-suited for financial environments where overreaction to short-term changes can be costly. The PPO algorithm aims to maximize the following objective:

$\begin{gathered}L^{P P O}\left(\theta^{macro}\right) =\mathbb{E}_t\left[\min \left(r_t\left(\theta^{macro}\right) A_t^{macro}, clip\left(r_t\left(\theta^{macro}\right), 1\right.\right.\right. \left.\left.-\varepsilon, 1+\varepsilon) A_t^{macro}\right)\right]\end{gathered}$        (3)

The high-level agent provides strategic guidance for group-level portfolio adjustments, enabling hierarchical coordination with:

Inputs: The high-level agent processes macroeconomic indicators $X_t^{macro}$, including economic and market signals and global events influencing market movements.

Reward Function: Designed to balance risk-adjusted returns, portfolio stability, sectoral performance, and leverage management. The reward function incentivizes decisions aligned with market conditions. It integrates:

$R_t^{macro}=\alpha \cdot S R_t+\beta \cdot R_t^{stability}+\gamma \cdot S_t^{sector}-\delta \cdot L_t$          (4)

where, SR_t  is the Sharpe Ratio, calculated as:

$S R_t=\frac{\mathbb{E}\left[R_t\right]-R_f}{\sigma_t}$         (5)

here, $\mathbb{E}\left[R_t\right]$ is the expected return, $R_f$ is the risk-free rate, and $\sigma_t$ is the standard deviation of returns.

$R_t^{stability}$: The stability term, defined as the inverse of portfolio volatility:

$R_t^{stability}=\frac{1}{\sigma_t}$              (6)

This term rewards agents for maintaining a stable portfolio with lower volatility.

$S_t^{sector}$: Sector-specific Sharpe Ratios, which measure the performance of capital allocation to individual sectors. This term rewards effective sector allocation based on macroeconomic conditions, such as increasing exposure to defensive sectors during market downturns.

L_t: The leverage ratio, representing the extent of portfolio leverage. A penalty term is included to discourage excessive leverage during volatile markets.

The parameters α, β, γ, and δ control the relative importance of each component in the reward function, enabling the system to adapt its priorities dynamically. This reward function aims to ensure that high-level agents optimize portfolio strategies across macroeconomic, stability, sectoral, and leverage dimensions.

Actions: The high-level agent interprets macroeconomic signals to guide portfolio adjustments, including: Adjusting risk exposure (risk-on/risk-off), reallocating capital across asset classes, setting sectoral preferences, and modifying leverage ratios to adapt to market trends.

3.1.2 Low-level agents: group-level signal processing

In the IPS framework, each sector is managed by a dedicated low-level agent, which processes sector-specific signals to capitalize on shared market characteristics and optimize decision-making at a granular level. These agents are specialized to exploit common dynamics within their respective sectors, allowing IPS to adapt granularly to sectoral trends while aligning with overarching macroeconomic strategies.

Technology Sector Agent: The low-level agent for the technology sector focuses on high-growth opportunities, targeting assets with significant potential for value growth. This agent specializes in navigating the inherent volatility of the sector by optimizing asset allocation within subsectors such as semiconductors, cloud computing, and artificial intelligence.

Healthcare Sector Agent: The healthcare sector agent prioritizes stability and defensive performance, particularly during periods of economic uncertainty. This agent focuses on assets such as pharmaceutical companies and biotechnology firms, which are known for their resilience during market downturns.

Energy Sector Agent: The energy sector agent is tailored to manage the cyclical nature of this industry, focusing on macroeconomic indicators such as oil prices and geopolitical trends. This agent dynamically adjusts allocations within the sector to capitalize on market cycles.

To guide their decision-making, these agents rely on:

Inputs: These agents receive sector-specific inputs, including aggregated Open, High, Low, Close, Volume (OHLCV) market data and technical signals that capture trends and momentum for all assets within its assigned sector. This input structure enables low-level agents to refine their decisions based on sector-specific factors, while aligning their actions with the strategic guidance provided by the high-level agent.

Reward Function: The reward function for low-level agents in the IPS framework is designed to optimize sector-level decisions by balancing risk and growth objectives. The reward for a low-level agent managing a specific sector, R_i, is calculated as:

$R_i=\omega_i^{\text {risk }} \cdot \frac{E\left[R_i\right]}{\sigma_i}+\omega_i^{\text {growth }} \cdot \lambda \cdot C_i$               (7)

where, $\omega_i^{\text {risk}}:$ A weighting parameter that determines the emphasis on risk-adjusted returns; $\frac{E\left[R_i\right]}{\sigma_i}$: The Sharpe Ratio, representing the expected return $E\left[R_i\right]$ per unit of risk (volatility $\left.\sigma_i\right) ; \omega_i^{\text {growth}}$: A weighting parameter that prioritizes growth-focused objectives; λ: A growth scaling factor that controls the contribution of growth metrics to the reward function; C_i: A growth-oriented measure, such as capital allocation to high-potential sectors or industries.

Actions: Based on the received signals and computed rewards, each low-level agent undertakes sector-specific actions, such as adjusting portfolio weights for its assigned sector, reallocating resources among sector-specific assets, and managing exposure within the sector. These actions directly influence the portfolio's structure by optimizing risk and return within the agent’s assigned sector, while adhering to the guidelines set by the high-level agent.

This architecture ensures that low-level agents respond precisely to sector-specific dynamics while remaining synchronized with the strategic guidance of the high-level agent, enabling fine-grained control and improved portfolio coherence.

3.2 Bayesian neural networks for risk assessment in IPS

In the IPS framework, BNNs [11] are integrated into both the policy and value networks of each PPO agent, replacing deterministic neural networks. BNNs model epistemic uncertainty by treating network parameters as probability distributions rather than fixed values, allowing agents to avoid overconfident decisions in volatile financial environments.

Mathematically, this is formalized through Bayes’ theorem, where a prior distribution $p(\theta)$ over network weights is updated with data D to produce a posterior:

$p(\theta \mid D)=\frac{p(D \mid \theta) p(\theta)}{p(D)}$               (8)

To implement this in practice, we use Monte Carlo Dropout (MC Dropout) as an approximation to Bayesian inference, following [21]. During training and inference, dropout is applied at each forward pass, and multiple stochastic passes are used to estimate predictive distributions. The prior is implicitly defined by a Bernoulli distribution from the dropout mask, while the posterior is approximated using variational inference with stochastic gradient descent.

Although BNNs introduce additional computational overhead (approximately 1.5×compared to deterministic networks), this cost is justified by their ability to improve risk-aware decision-making and robustness in uncertain market conditions.

3.3 Attention-based communication for agent coordination in IPS

In many standard MARL frameworks, agents operate independently based only on local observations, without explicitly accounting for broader system-level information. While this may work in simple environments, financial markets are complex, dynamic systems with interdependent signals — such as macroeconomic indicators, sector-level trends, and correlated asset movements. Without effective inter-agent communication, agents risk missing broader market contexts, leading to suboptimal portfolio decisions.

To address this, the IPS architecture integrates agent-specific attention mechanisms that enable selective information exchange and prioritization. For the high-level agent, inputs include macroeconomic indicators such as GDP growth, interest rates, and global indices. For low-level agents, inputs consist of sector-specific OHLCV data and technical indicators. To enable selective communication, each agent first linearly projects its input feature vector X into query (Q), key (K), and value (V) representations using learned weight matrices:

$Q=W_Q \cdot X, K=W_k \cdot X, V=W_v \cdot X$             (9)

During communication, each agent a_i receives its own observation o_i and shared signals x_j from other agents. The attention mechanism computes relevance scores e_ij using the scaled dot-product:

$e_{i j}=\frac{q_i \cdot k_j}{\sqrt{d}}$             (10)

These scores are normalized using the Softmax function to produce attention weights:

$a_{i j}=\frac{\exp \left(e_{i j}\right)}{\sum_{k=1}^n \exp \left(e_{i k}\right)}$             (11)

Finally, the context vector c_i, which represents the weighted aggregation of shared signals, is computed as:

$c_i=\sum_{j=1}^n a_{i j} v_j$                (12)

where, v_j=w_vx_j  is the value vector derived from x_j.

3.4 Hyperparameter optimization in IPS

In IPS, hyperparameters are dynamically adjusted using GAs, which are well-suited for exploring high-dimensional spaces, avoiding local optima, and accommodating both continuous and discrete parameter types (Figure 2).

GAs optimize hyperparameters such as learning rates, discount factors, and sector-specific allocations by encoding them into chromosomes.

The GA workflow includes:

1. Initialization: Generating an initial population of candidate solutions (chromosomes).

2. Fitness Evaluation: Evaluating each chromosome using cumulative returns and risk-adjusted metrics.

3. Selection, Crossover, and Mutation: Selecting top-performing chromosomes, combining traits via crossover, and introducing mutations to maintain diversity and avoid local optima.

4. Iteration and Convergence: Repeating these steps until a termination criterion is met.

Table 1 presents possible hyperparameter settings for high-level and low-level agents in IPS.

2.png

Figure 2. Integration of genetic algorithms in the IPS system for agent configuration and optimization

Table 1. Possible hyperparameter settings for high-level and low-level agents in IPS