Coder–Critic–Tester Multi-Agent Code Generation with Dynamic Role Arbitration: An Execution-Grounded Evaluation on MBPP
Keywords:
Multi-Agent Code Generation, Program Synthesis, Code Review, Execution Feedback, Automated Testing, Program Repair, Dynamic Arbitration, MBPPAbstract
Multi-agent code generation can improve functional correctness by separating generation, review, execution, and repair, but unconditional collaboration may spend computation on tasks that are already solved. This study evaluates a dynamic Coder–Critic–Tester architecture in which an Arbiter invokes Repair only when execution of two visible assertions exposes a failure. The controlled local implementation combines word-and-bigram TF–IDF retrieval over the MBPP training partition, task-conditioned Python sketches, static abstract-syntax review, timeout-guarded execution, and a compact synthesis library. The evaluation follows the official MBPP split and covers all 500 test tasks. The third basic assertion and every available challenge assertion remain sealed during candidate selection. Coder and Critic each achieve 21.0% sealed Pass@1, while the Tester raises performance to 27.2%. Dynamic arbitration reaches 31.2% sealed Pass@1 and 25.6% full-suite Pass@1. Relative to the Tester endpoint, the final policy records 22 gains and 2 losses, a net improvement of 4.0 percentage points (exact McNemar p = 3.59 × 10⁻⁵). It matches an always-repair policy exactly while skipping 168 of 500 repair calls, reducing candidate executions by 26.2%, lexical code tokens processed by 6.3%, and mean selection time by 6.4%. The results show that execution feedback, rather than static review alone, supplies the principal accuracy gain, and that a simple failure-aware Arbiter can preserve that gain while removing avoidable repair work.
References
[1] J. Austin et al., “Program Synthesis with Large Language Models,” arXiv:2108.07732, 2021.
[2] M. Chen et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374, 2021.
[3] Y. Li, “Findable then Explainable: Retrieval-Summary Integration for Code Intelligence on a Lightweight CodeSearchNet Subset,” J. Adv. Comput. Syst., vol. 4, no. 7, pp. 65-82, Jul. 2024, doi: 10.69987/JACS.2024.40706.
[4] Z. Feng et al., “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” in Findings of the Association for Computational Linguistics: EMNLP, 2020, pp. 1536-1547.
[5] D. Guo et al., “GraphCodeBERT: Pre-training Code Representations with Data Flow,” in Proc. Int. Conf. Learning Representations, 2021.
[6] Y. Wang et al., “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation,” in Proc. EMNLP, 2021, pp. 8696-8708.
[7] E. Nijkamp et al., “A Conversational Paradigm for Program Synthesis,” arXiv:2203.13474, 2022.
[8] Y. Li et al., “Competition-Level Code Generation with AlphaCode,” Science, vol. 378, no. 6624, pp. 1092-1097, 2022.
[9] H. Le et al., “CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 21314-21328.
[10] D. Fried et al., “InCoder: A Generative Model for Code Infilling and Synthesis,” in Proc. Int. Conf. Learning Representations, 2023.
[11] R. Li et al., “StarCoder: May the Source Be with You!” Trans. Mach. Learn. Res., 2023.
[12] B. Rozière et al., “Code Llama: Open Foundation Models for Code,” arXiv:2308.12950, 2023.
[13] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the Naturalness of Software,” in Proc. 34th Int. Conf. Software Engineering, 2012, pp. 837-847.
[14] J. Nie and D. Zheng, “Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal,” J. Adv. Comput. Syst., vol. 3, no. 1, pp. 66-80, Jan. 2023, doi: 10.69987/JACS.2023.30105.
[15] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A Survey of Machine Learning for Big Code and Naturalness,” ACM Comput. Surv., vol. 51, no. 4, Art. no. 81, 2018.
[16] A. Madaan et al., “Self-Refine: Iterative Refinement with Self-Feedback,” in Advances in Neural Information Processing Systems, vol. 36, 2023.
[17] N. Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023.
[18] S. Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” in Proc. Int. Conf. Learning Representations, 2023.
[19] Y. He, Y. Pan, Y. Wang, S. Du, and Q. Xin, “Intelligent Fault Analysis with AIOps Technology,” J. Theory Pract. Eng. Sci., vol. 4, no. 1, pp. 94-100, Feb. 2024, doi: 10.53469/jtpes.2024.04(01).13.
[20] K. Cobbe et al., “Training Verifiers to Solve Math Word Problems,” arXiv:2110.14168, 2021.
[21] Z. S. Zhong, R. Ma, and H. Zhao, “Human-Uncertainty Distillation for Calibrated Vision Models on CIFAR-10H,” J. Adv. Comput. Syst., vol. 3, no. 2, pp. 77-89, Feb. 2023, doi: 10.69987/JACS.2023.30206.
[22] Y. Du et al., “Improving Factuality and Reasoning in Language Models through Multiagent Debate,” arXiv:2305.14325, 2023.
[23] Q. Wu et al., “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation,” arXiv:2308.08155, 2023.
[24] S. Hong et al., “MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework,” arXiv:2308.00352, 2023.
[25] S. Zhou, Z. Li, and E. Wang, “Evidence-Grounded RAG for Tokenized Trade Receivable Disclosure QA under U.S. Capital Market Standards,” J. Adv. Comput. Syst., vol. 3, no. 7, pp. 41-57, Jul. 2023, doi: 10.69987/JACS.2023.30704.
[26] C. Qian et al., “ChatDev: Communicative Agents for Software Development,” arXiv:2307.07924, 2023.
[27] D. Huang et al., “AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation,” arXiv:2312.13010, 2023.
[28] Y. Wang, S. Du, Q. Xin, Y. He, and W. Qian, “Autonomous Driving System Driven by Artificial Intelligence Perception Fusion,” Acad. J. Sci. Technol., vol. 9, no. 2, pp. 193-198, Feb. 2024, doi: 10.54097/e0b9ak47.
[29] G. Fraser and A. Arcuri, “EvoSuite: Automatic Test Suite Generation for Object-Oriented Software,” in Proc. 19th ACM SIGSOFT Symp. Foundations of Software Engineering, 2011, pp. 416-419.
[30] H. Zhang, “DriftGuard: Multi-Signal Drift Early Warning and Safe Re-Training/Rollback for CTR/CVR Models,” J. Adv. Comput. Syst., vol. 3, no. 7, pp. 24-40, 2023, doi: 10.69987/JACS.2023.30703.
[31] S. Lukasczyk and G. Fraser, “Pynguin: Automated Unit Test Generation for Python,” in Proc. IEEE/ACM 44th Int. Conf. Software Engineering: Companion Proceedings, 2022.
[32] W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest, “Automatically Finding Patches Using Genetic Programming,” in Proc. 31st Int. Conf. Software Engineering, 2009, pp. 364-374.
[33] Q. Xin, R. Song, Z. Wang, Z. Xu, and F. Zhao, “Enhancing Bank Credit Risk Management Using the C5.0 Decision Tree Algorithm,” J. Comput. Technol. Appl. Math., vol. 1, no. 4, pp. 100-107, Nov. 2024, doi: 10.5281/zenodo.14032041.
[34] R. Gupta, S. Pal, A. Kanade, and S. Shevade, “DeepFix: Fixing Common C Language Errors by Deep Learning,” in Proc. AAAI Conf. Artificial Intelligence, vol. 31, no. 1, 2017.
[35] Z. S. Zhong and S. Ling, “Improved Theoretical Guarantee for Rank Aggregation via Spectral Method,” Inf. Inference: J. IMA, vol. 13, no. 3, Art. no. iaae020, Sep. 2024, doi: 10.1093/imaiai/iaae020.
[36] T. Yang, Q. Xin, X. Zhan, S. Zhuang, and H. Li, “Enhancing Financial Services through Big Data and AI-Driven Customer Insights and Risk Analysis,” J. Knowl. Learn. Sci. Technol., vol. 3, no. 3, pp. 53-62, Jul. 2024, doi: 10.60087/jklst.vol3.n3.p53-62.
[37] S. Zhou, Z. Li, and E. Wang, “Long-Document RAG for Contractual and Insurance Clause Analysis in Receivables RWA Structures,” J. Adv. Comput. Syst., vol. 4, no. 8, pp. 88-104, Aug. 2024, doi: 10.69987/JACS.2024.40810.
[38] B. Wang, Y. He, Z. Shui, Q. Xin, and H. Lei, “Predictive Optimization of DDoS Attack Mitigation in Distributed Systems Using Machine Learning,” in Proc. 6th Int. Conf. Computing and Data Science (ICCDS), 2024, pp. 89-94, doi: 10.54254/2755-2721/64/20241350.
[39] Z. S. Zhong and S. Ling, “Uncertainty Quantification of Spectral Estimator and MLE for Orthogonal Group Synchronization,” arXiv:2408.05944, Aug. 2024.
[40] Q. Xin, Z. Xu, L. Guo, F. Zhao, and B. Wu, “IoT Traffic Classification and Anomaly Detection Method Based on Deep Autoencoders,” in Proc. 6th Int. Conf. Computing and Data Science (ICCDS), 2024, pp. 64-70, doi: 10.54254/2755-2721/69/20241511.
[41] J. Nie and D. Zheng, “Noisy-Neighbor-Aware VM Degradation Risk Modeling with Unsupervised Residual Fusion,” J. Adv. Comput. Syst., vol. 4, no. 4, pp. 112-123, Apr. 2024, doi: 10.69987/JACS.2024.40409.
[42] J. Chen, J. Xiong, Y. Wang, Q. Xin, and H. Zhou, “Implementation of an AI-Based MRD Evaluation and Prediction Model for Multiple Myeloma,” Front. Comput. Intell. Syst., vol. 6, no. 3, pp. 127-131, 2024, doi: 10.54097/zJ4MnbWW.
[43] A. G. S. Raj et al., “Impact of Bilingual CS Education on Student Learning and Engagement in a Data Structures Course,” in Proc. 19th Koli Calling Int. Conf. Computing Education Research, 2019, Art. no. 24, pp. 1-10, doi: 10.1145/3364510.3364518.
[44] Z. Ling, Q. Xin, Y. Lin, G. Su, and Z. Shui, “Optimization of Autonomous Driving Image Detection Based on RFAConv and Triplet Attention,” in Proc. 2nd Int. Conf. Software Engineering and Machine Learning (SEML), 2024, pp. 210-217, doi: 10.54254/2755-2721/77/2024MA0067.
[45] H. Zhang, “Risk-Aware Budget-Constrained Auto-Bidding under First-Price RTB: A Distributional Constrained Deep Reinforcement Learning Framework,” J. Adv. Comput. Syst., vol. 4, no. 6, pp. 30-47, Jun. 2024, doi: 10.69987/JACS.2024.40603.
[46] J. Wang, Q. Xin, Y. Liu, J. Wang, and T. Yang, “Predicting Enterprise Marketing Decision Making with Intelligent Data-Driven Approaches,” J. Ind. Eng. Appl. Sci., vol. 2, no. 3, pp. 12-19, Jun. 2024, doi: 10.5281/zenodo.11357252.
[47] Z. Zhong, M. Zheng, H. Mai, J. Zhao, and X. Liu, “Cancer Image Classification Based on DenseNet Model,” J. Phys.: Conf. Ser., vol. 1651, no. 1, Art. no. 012143, Nov. 2020, doi: 10.1088/1742-6596/1651/1/012143.
[48] K. Xu, H. Zhou, H. Zheng, M. Zhu, and Q. Xin, “Intelligent Classification and Personalized Recommendation of E-Commerce Products Based on Machine Learning,” in Proc. 6th Int. Conf. Computing and Data Science (ICCDS), 2024, doi: 10.54254/2755-2721/64/20241365.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Wei Zhang, Michael Foster, Jessica Nguyen, Bin He (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Creative Commons Attribution 4.0 International (CC BY 4.0).
This work is licensed under a 