| 234 | 0 | 103 |
| 下载次数 | 被引频次 | 阅读次数 |
针对股指期货市场，本文基于优势演员-评论家（advantage actor-critic,A2C）算法与近端策略优化（proximal policy optimization ,PPO）算法，设计了集成强化学习量化交易策略。首先，采用极端梯度提升（extreme gradient boosting,XGBoost）模型筛选关键技术因子作为模型的状态输入变量，并结合交易成本约束完成奖励函数的优化设计。其次，分别对A2C和PPO模型进行超参数调优，确定隐藏层结构、学习率、批处理参数等关键超参数的最优取值。在此基础上，构建集成模型框架，通过验证集夏普比率对比实现基模型动态选择。回测结果表明：所提出的集成模型在沪深300指数、上证50指数和中证500指数这3个指数上的累计收益分别为106 766、84 739和78 408元；年化收益率达32.03%、25.42%和23.52%；夏普比率分别为1.95、0.63和1.39；各项指标显著优于单一A2C模型、单一PPO模型及买入持有策略、动量策略两类基准策略。本文提出的策略可通过动态融合各基模型的性能优势，在市场波动环境中实现更低的最大回撤和更稳健的收益表现，尤其在中证500指数下跌行情中仍能保持正收益，充分验证了其抗风险能力和市场适应性。
Abstract：This paper focuses on stock index futures and designs an integrated reinforcement learning quantitative trading strategy based on advantage actor critic(A2C) model and proximal policy optimization(PPO) model. First, the extreme gradient boosting(XGBoost) model is used to select key technical factors as state inputs, and the reward function is designed based on transaction cost optimization. Second, parameter tuning is conducted for both A2C and PPO models to determine key hyperparameters such as the hidden layer structure, learning rate, and batch processing parameters. On this basis, an ensemble model framework is designed, and dynamic selection of base models is achieved by comparing the Sharpe ratios on the validation set. The backtesting results show that the integrated model yielded cumulative returns of 106 766 RMB, 84 739 RMB, and 78 408 RMB, alongside annualized returns of 32.03%,25.42%, and 23.52%, and Sharpe ratios of 1.95, 0.63, and 1.39 for the CSI 300, SSE 50, and CSI 500 indices,respectively, significantly outperforming single A2C and PPO models, as well as benchmark buy-and-hold and momentum strategies. By dynamically blending the distinct advantages of each base model based on validation-set performance, the proposed integrated strategy achieves lower maximum drawdowns and more robust returns amid market volatility. In particular, it can still maintain positive returns during the downward trend of the CSI 500 index,fully validating its risk mitigation capabilities and market adaptability.
[1]SAHU S K, MOKHADE A, BOKDE N D. An overview of machine learning, deep learning, and reinforcement learning-based techniques in quantitative finance:recent progress and challenges[J]. Applied Sciences, 2023, 13(3):1956.
[2]卜湛，张善凡，李雪延，等.基于深度强化学习的自适应股指预测研究[J].管理科学学报，2023, 26(4):148-174.BU Z, ZHANG S F, LI X Y, et al. Adaptive stock index prediction based on deep reinforcement learning[J]. Journal of Management Sciences in China , 2023 , 26(4):148-174.(in Chinese)
[3]符甲鑫，刘磊，钱成.基于注意力机制的A3C量化交易策略[J].南通大学学报（自然科学版），2023, 22(2):43-49.FU J X, LIU L, QIAN C. A3C quantitative trading strategy based on attention[J]. Journal of Nantong University(Natural Science Edition), 2023, 22(2):43-49.(in Chinese)
[4]WU X , CHEN H L , WANG J J , et al. Adaptive stock trading strategies with deep reinforcement learning methods[J]. Information Sciences, 2020, 538:142-158.
[5]TANG L L. An actor-critic-based portfolio investment method inspired by benefit-risk optimization[J]. Journal of Algorithms&Computational Technology, 2018, 12(4):351-360.
[6]许波，贺一峻，温健城，等.深度强化学习应用于金融市场量化交易研究综述[J].智能科学与技术学报，2024,6(4):416-428.XU B, HE Y J, WEN J C, et al. Review:the application of deep reinforcement learning to quantitative trading in financial market[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(4):416-428.(in Chinese)
[7]AN B, SUN S, WANG R D. Deep reinforcement learning for quantitative trading:challenges and opportunities[J].IEEE Intelligent Systems, 2022, 37(2):23-26.
[8]SUN S, WANG R D, AN B. Reinforcement learning for quantitative trading[J]. ACM Transactions on Intelligent Systems and Technology, 2023, 14(3):1-29.
[9]司伟钰.基于深度强化学习的交易策略技术研究[D].上海：上海交通大学，2018.SI W Y. Research on the trading strategy based on deep reinforcement learning[D]. Shanghai:Shanghai Jiao Tong University, 2018.
[10]TH魪ATE T, ERNST D. An application of deep reinforcement learning to algorithmic trading[J]. Expert Systems with Applications, 2021, 173:114632.
[11]KIM M. Adaptive trading system integrating machine learning and back-testing:Korean bond market case[J].Expert Systems with Applications, 2021, 176:114767.
[12]WIERING M A, van HASSELT H. Ensemble algorithms in reinforcement learning[J]. IEEE Transactions on Systems,Man, and Cybernetics, Part B(Cybernetics), 2008, 38(4):930-936.
[13]TAGHIAN M, ASADI A, SAFABAKHSH R. Learning financial asset-specific trading rules via deep reinforcement learning[J]. Expert Systems with Applications, 2022, 195:116523.
[14]龙杰，谢良，徐海蛟.集成的深度强化学习投资组合模型[J].计算机应用，2024, 44(1):300-310.LONG J, XIE L, XU H J. Integrated deep rein forcement learning portfolio model[J]. Journal of Computer Applications, 2024, 44(1):300-310.(in Chinese)
[15]YANG H Y, LIU X Y, ZHONG S, et al. Deep reinforcement learning for automated stock trad ing:an ensemble strategy[C]//Proceedings of the First ACM International Conference on AI in Finance, October 15-16, 2020, New York. New York:Association for Computing Machinery,2020:1-8.
[16]MA C, ZHANG J S, LIU J M, et al. A par allel multimodule deep reinforcement learning algorithm for stock trading[J]. Neurocomputing, 2021, 449:290-302.
[17]刘健.基于强化学习的自动交易代理[D].哈尔滨：哈尔滨工业大学，2020.LIU J. Autonomous trading agent with reinforcement learning[D]. Harbin:Harbin Institute of Technology, 2020.
[18]JING L, KANG Y. Automated cryptocurrency trading approach using ensemble deep reinforcement learning:learn to understand candlesticks[J]. Expert Systems with Applications, 2024, 237:121373.
[19]VISHAL M, RAVI V, LAL R. Ensemble deep reinforcement learning for financial trading[M]//MAGLARAS L A,DAS S, TRIPATHY N, et al. Machine learning approaches in financial analytics. Cham:Springer Nature Switzerland, 2024:191-207.
[20]冀中，张文嘉.集成深度强化学习在股票指数投资组合优化中的应用分析[J].计算机科学与探索，2025, 19(1):237-244.JI Z, ZHANG W J. Application analysis of ensemble deep reinforcement learning in portfolio opti-mization of stock index[J]. Journal of Frontiers of Computer Science&Technology, 2025, 19(1):237-244.(in Chinese)
基本信息:
DOI：10.12194/j.ntu.20240929001
中图分类号:TP18;F832.5
引用信息:
[1]刘磊,张凤敏,钱成.基于模型集成强化学习方法的股指期货交易策略[J].南通大学学报(自然科学版),2026,25(02):14-23.DOI:10.12194/j.ntu.20240929001.
基金信息:
教育部人文社会科学研究项目(25YJAZH108)
2025-07-24
2025-07-24
2025-07-24