Corrective RAG with Iterative Evidence Verification and Query Reformulation for Scientific Fact Checking
Keywords:
Scientific Fact Checking, Corrective RAG, Iterative Retrieval, Query Reformulation, Evidence Verification, Scifact, Calibrated Gating, Reciprocal Rank FusionAbstract
Scientific claim verification requires a system to retrieve relevant literature, identify evidentiary sentences, and assign a SUPPORT, CONTRADICT, or not-enough-information (NEI) label. A single retrieval pass is inexpensive but leaves recoverable evidence outside the candidate set, whereas unconditional iterative retrieval increases cost and can introduce query drift. This study presents a corrective retrieval-augmented verification pipeline that couples stemmed BM25 retrieval, evidence sentence ranking, a calibrated verifier agent, training-fold evidence memory, query reformulation, and reciprocal-rank fusion. The corrective gate is calibrated inside each training fold. It initiates a second retrieval pass only for low-confidence or weak-evidence cases, accepts a changed label only when the second verifier provides a calibrated confidence gain, and preserves a previously evidence-backed label when the second pass returns NEI. Experiments use the SciFact corpus of 5,183 abstracts and 1,109 labeled claims under the provided five-fold partition. The second pass raises document Recall@5 from 90.3% to 92.8%. The gated system invokes that pass for 17.0% of claims, reaches 68.4% accuracy, 0.656 Macro-F1, and the highest evidence-conditioned joint score of 0.597. Compared with always applying two passes, it improves accuracy by 4.78 points and Macro-F1 by 5.51 points while reducing processed tokens by 41.8%. The results show that iterative retrieval is most effective when evidence accumulation, label replacement, and cost are governed by separate verification decisions.
References
[1] D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi, “Fact or Fiction: Verifying Scientific Claims,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing, 2020, pp. 7534–7550.
[2] S. Zhou, Z. Li, and E. Wang, “Evidence-Grounded RAG for Tokenized Trade Receivable Disclosure QA under U.S. Capital Market Standards,” J. Adv. Comput. Syst., vol. 3, no. 7, pp. 41–57, Jul. 2023, doi: 10.69987/JACS.2023.30704.
[3] J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: A Large-Scale Dataset for Fact Extraction and VERification,” in Proc. 2018 Conf. North American Chapter of the Association for Computational Linguistics, 2018, pp. 809–819.
[4] S. Zhou, Z. Li, and E. Wang, “Long-Document RAG for Contractual and Insurance Clause Analysis in Receivables RWA Structures,” J. Adv. Comput. Syst., vol. 4, no. 8, pp. 88–104, Aug. 2024, doi: 10.69987/JACS.2024.40810.
[5] Y. Li, “Findable then Explainable: Retrieval–Summary Integration for Code Intelligence on a Lightweight CodeSearchNet Subset,” J. Adv. Comput. Syst., vol. 4, no. 7, pp. 65–82, Jul. 2024, doi: 10.69987/JACS.2024.40706.
[6] J. Vladika and F. Matthes, “Scientific Fact-Checking: A Survey of Resources and Approaches,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 6215–6230.
[7] D. Wadden and K. Lo, “Overview and Insights from the SCIVER Shared Task on Scientific Claim Verification,” in Proc. Second Workshop on Scholarly Document Processing, 2021, pp. 124–129.
[8] R. Pradeep, X. Ma, R. Nogueira, and J. Lin, “Scientific Claim Verification with VerT5erini,” in Proc. 12th International Workshop on Health Text Mining and Information Analysis, 2021, pp. 94–103.
[9] D. Wadden, K. Lo, L. L. Wang, A. Cohan, I. Beltagy, and H. Hajishirzi, “MultiVerS: Improving Scientific Claim Verification with Weak Supervision and Full-Document Context,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 61–76.
[10] I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A Pretrained Language Model for Scientific Text,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th International Joint Conf. Natural Language Processing, 2019, pp. 3615–3620.
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics, 2019, pp. 4171–4186.
[12] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th International Joint Conf. Natural Language Processing, 2019, pp. 3982–3992.
[13] Z. Zhong, M. Zheng, H. Mai, J. Zhao, and X. Liu, “Cancer Image Classification Based on DenseNet Model,” J. Phys.: Conf. Ser., vol. 1651, no. 1, Art. no. 012143, Nov. 2020, doi: 10.1088/1742-6596/1651/1/012143.
[14] J. Chen, J. Xiong, Y. Wang, Q. Xin, and H. Zhou, “Implementation of an AI-Based MRD Evaluation and Prediction Model for Multiple Myeloma,” Front. Comput. Intell. Syst., vol. 6, no. 3, pp. 127–131, Jan. 2024, doi: 10.54097/zJ4MnbWW.
[15] Z. S. Zhong, R. Ma, and H. Zhao, “Human-Uncertainty Distillation for Calibrated Vision Models on CIFAR-10H,” J. Adv. Comput. Syst., vol. 3, no. 2, pp. 77–89, Feb. 2023, doi: 10.69987/JACS.2023.30206.
[16] Q. Xin, “Uncertainty-Aware Late Fusion for 3D Perception (Confidence Calibration + Fusion Rule Learning),” J. Technol. Informatics Eng., vol. 4, no. 1, pp. 215–238, Apr. 2025, doi: 10.51903/jtie.v4i1.485.
[17] Z. S. Zhong and S. Ling, “Uncertainty Quantification of Spectral Estimator and MLE for Orthogonal Group Synchronization,” arXiv:2408.05944, Aug. 2024.
[18] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474.
[19] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang, “REALM: Retrieval-Augmented Language Model Pre-Training,” in Proc. 37th International Conf. Machine Learning, 2020, pp. 3929–3938.
[20] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense Passage Retrieval for Open-Domain Question Answering,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing, 2020, pp. 6769–6781.
[21] G. Izacard and E. Grave, “Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering,” in Proc. 16th Conf. European Chapter of the Association for Computational Linguistics, 2021, pp. 874–880.
[22] J. Nie and D. Zheng, “Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal,” J. Adv. Comput. Syst., vol. 3, no. 1, pp. 66–80, Jan. 2023, doi: 10.69987/JACS.2023.30105.
[23] Q. Xin, “Explaining OpenStack Failure-Injection Log Anomalies with Retrieved Normal Prototypes,” Emerg. Inf. Sci. Technol., vol. 6, no. 2, pp. 125–146, Nov. 2025, doi: 10.18196/eist.v6i2.31232.
[24] H. Zhang, “LLM-Driven CI Failure Diagnosis and Automated Repair: From GitHub Actions Logs to Patch Recommendation,” J. Technol. Informatics Eng., vol. 4, no. 1, pp. 190–214, Apr. 2025, doi: 10.51903/jtie.v4i1.484.
[25] Y. He, Y. Pan, Y. Wang, S. Du, and Q. Xin, “Intelligent Fault Analysis with AIOps Technology,” J. Theory Pract. Eng. Sci., vol. 4, no. 1, pp. 94–100, Feb. 2024, doi: 10.53469/jtpes.2024.04(01).13.
[26] S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009.
[27] G. V. Cormack, C. L. A. Clarke, and S. Büttcher, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods,” in Proc. 32nd International ACM SIGIR Conf. Research and Development in Information Retrieval, 2009, pp. 758–759.
[28] G. Salton, A. Wong, and C. S. Yang, “A Vector Space Model for Automatic Indexing,” Communications of the ACM, vol. 18, no. 11, pp. 613–620, 1975.
[29] J. J. Rocchio, “Relevance Feedback in Information Retrieval,” in The SMART Retrieval System—Experiments in Automatic Document Processing, G. Salton, Ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1971, pp. 313–323.
[30] V. Lavrenko and W. B. Croft, “Relevance-Based Language Models,” in Proc. 24th Annual International ACM SIGIR Conf. Research and Development in Information Retrieval, 2001, pp. 120–127.
[31] Z. S. Zhong and S. Ling, “Improved Theoretical Guarantee for Rank Aggregation via Spectral Method,” Inf. Inference: J. IMA, vol. 13, no. 3, Art. no. iaae020, Sep. 2024, doi: 10.1093/imaiai/iaae020.
[32] L. Wang, N. Yang, and F. Wei, “Query2doc: Query Expansion with Large Language Models,” in Proc. 2023 Conf. Empirical Methods in Natural Language Processing, 2023, pp. 9414–9423.
[33] Z. S. Zhong, X. Pan, and Q. Lei, “Bridging Domains with Approximately Shared Features,” in Proc. 28th Int. Conf. Artificial Intelligence and Statistics, PMLR, vol. 258, pp. 559–567, 2025.
[34] X. Ma, Y. Gong, P. He, H. Zhao, and N. Duan, “Query Rewriting in Retrieval-Augmented Large Language Models,” in Proc. 2023 Conf. Empirical Methods in Natural Language Processing, 2023, pp. 5303–5315.
[35] Z. Jiang, F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig, “Active Retrieval Augmented Generation,” in Proc. 2023 Conf. Empirical Methods in Natural Language Processing, 2023, pp. 7969–7992.
[36] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions,” in Proc. 61st Annual Meeting of the Association for Computational Linguistics, 2023, pp. 10014–10037.
[37] L. Gao, X. Ma, J. Lin, and J. Callan, “Precise Zero-Shot Dense Retrieval without Relevance Labels,” in Proc. 61st Annual Meeting of the Association for Computational Linguistics, 2023, pp. 1762–1777.
[38] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” arXiv:2310.11511, 2023.
[39] K. Xu, H. Zhou, H. Zheng, M. Zhu, and Q. Xin, “Intelligent Classification and Personalized Recommendation of E-Commerce Products Based on Machine Learning,” Appl. Comput. Eng., vol. 64, no. 1, pp. 143–149, May 2024, doi: 10.54254/2755-2721/64/20241365.
[40] H. Zhang, “Risk-Aware Budget-Constrained Auto-Bidding under First-Price RTB: A Distributional Constrained Deep Reinforcement Learning Framework,” J. Adv. Comput. Syst., vol. 4, no. 6, pp. 30–47, Jun. 2024, doi: 10.69987/JACS.2024.40603.
[41] H. Wang, Y. Ren, and X. Chang, “Layout-Aware Progressive PDF Rendering: AI Prioritization of PDF Slices to Reduce Time-to-Functional-First-Frame on FUNSD,” J. Technol. Informatics Eng., vol. 4, no. 2, pp. 425–446, Aug. 2025, doi: 10.51903/jtie.v4i2.523.
[42] H. Zhang, “Counterfactual Learning-to-Rank for Ads: Off-Policy Evaluation on the Open Bandit Dataset,” J. Adv. Comput. Syst., vol. 5, no. 12, pp. 1–11, Dec. 2025, doi: 10.69987/JACS.2025.51201.
[43] Q. Xin, “Hybrid Cloud Architecture for Efficient and Cost-Effective Large Language Model Deployment,” J. Inf. Syst. Informatics, vol. 7, no. 3, pp. 2182–2195, Sep. 2025, doi: 10.51519/journalisi.v7i3.1170.
[44] B. Zhou, H. Wang, and X. Chang, “Distilling VMAF into an Edge-Deployable Quality Predictor: A Pilot Shot-Level Proxy with LLM-Ready Quality Tokens,” J. Technol. Informatics Eng., vol. 4, no. 2, pp. 447–463, Aug. 2025, doi: 10.51903/jtie.v4i2.522.
[45] L. Zhang, R. Ma, and P. Greg, “Digital-Twin Dispatching for Urban Mobility via Spatio-Temporal Transformers and Offline Reinforcement Learning,” J. Technol. Informatics Eng., vol. 4, no. 2, pp. 337–363, Aug. 2025, doi: 10.51903/jtie.v4i2.501.
[46] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” in Proc. 34th International Conf. Machine Learning, 2017, pp. 1321–1330.
[47] Y. Geifman and R. El-Yaniv, “Selective Classification for Deep Neural Networks,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 4878–4887.
[48] H. Zhang, “DriftGuard: Multi-Signal Drift Early Warning and Safe Re-Training/Rollback for CTR/CVR Models,” J. Adv. Comput. Syst., vol. 3, no. 7, pp. 24–40, Jul. 2023, doi: 10.69987/JACS.2023.30703.
[49] J. Nie and D. Zheng, “Noisy-Neighbor-Aware VM Degradation Risk Modeling with Unsupervised Residual Fusion,” J. Adv. Comput. Syst., vol. 4, no. 4, pp. 112–123, Apr. 2024, doi: 10.69987/JACS.2024.40409.
[50] Q. Xin, Z. Xu, L. Guo, F. Zhao, and B. Wu, “IoT Traffic Classification and Anomaly Detection Method Based on Deep Autoencoders,” Appl. Comput. Eng., vol. 69, no. 1, pp. 64–70, Jul. 2024, doi: 10.54254/2755-2721/69/20241511.
[51] R. Dror, G. Baumer, S. Shlomov, and R. Reichart, “The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing,” in Proc. 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 1383–1392.
[52] B. Wang, Y. He, Z. Shui, Q. Xin, and H. Lei, “Predictive Optimization of DDoS Attack Mitigation in Distributed Systems Using Machine Learning,” Appl. Comput. Eng., vol. 64, no. 1, pp. 89–94, May 2024, doi: 10.54254/2755-2721/64/20241350.
[53] Z. Li, S. Zhou, and Z. Zhou, “Financial Risk Dashboard Design for Institutional RWA Investors: Visual Hierarchy, Chart Comprehension, and Explainability in FinChart-Bench,” Int. J. Graph. Des., vol. 3, no. 1, pp. 196–210, May 2025, doi: 10.51903/ijgd.v3i1.3715.
[54] Y. Wang, S. Du, Q. Xin, Y. He, and W. Qian, “Autonomous Driving System Driven by Artificial Intelligence Perception Fusion,” Acad. J. Sci. Technol., vol. 9, no. 2, pp. 193–198, Feb. 2024, doi: 10.54097/e0b9ak47.
[55] R. Ma, L. Zhang, and T. Song, “Computer-Vision-Informed Visual Explanation Cards for Autonomous-Driving Traffic-Sign Alerts: Localization, Classification, and Retrieved Evidence on GTSDB,” Int. J. Graph. Des., vol. 3, no. 2, pp. 455–473, Oct. 2025, doi: 10.51903/ijgd.v3i2.3991.
[56] Z. Ling, Q. Xin, Y. Lin, G. Su, and Z. Shui, “Optimization of Autonomous Driving Image Detection Based on RFAConv and Triplet Attention,” Appl. Comput. Eng., vol. 77, no. 1, pp. 210–217, Jul. 2024, doi: 10.54254/2755-2721/77/2024MA0067.
[57] Y. Lu, H. Zhou, and Y. Zhang, “A Constrained, Data-Driven Budgeting Framework Integrating Macro Demand Forecasting and Marketing Response Modeling,” J. Technol. Informatics Eng., vol. 4, no. 3, pp. 493–520, Dec. 2025, doi: 10.51903/jtie.v4i3.466.
[58] Q. Xin, R. Song, Z. Wang, Z. Xu, and F. Zhao, “Enhancing Bank Credit Risk Management Using the C5.0 Decision Tree Algorithm,” J. Comput. Technol. Appl. Math., vol. 1, no. 4, pp. 100–107, Nov. 2024, doi: 10.5281/zenodo.14032041.
[59] H. Zhou and K. Zhang, “News-Based Uncertainty and Macro-Market Fusion for VIX Direction Forecasting: Evidence from 2015–2024 FRED Panel,” J. Technol. Informatics Eng., vol. 4, no. 2, pp. 487–501, Aug. 2025, doi: 10.51903/jtie.v4i2.540.
[60] T. Yang, Q. Xin, X. Zhan, S. Zhuang, and H. Li, “Enhancing Financial Services through Big Data and AI-Driven Customer Insights and Risk Analysis,” J. Knowl. Learn. Sci. Technol., vol. 3, no. 3, pp. 53–62, Jul. 2024, doi: 10.60087/jklst.vol3.n3.p53-62.
[61] A. G. S. Raj, H. Zhang, V. Abhyankar, S. Mukherjee, E. Zhang, J. Williams, R. Halverson, and J. M. Patel, “Impact of Bilingual CS Education on Student Learning and Engagement in a Data Structures Course,” in Proc. 19th Koli Calling Int. Conf. Comput. Educ. Res., 2019, pp. 1–10, doi: 10.1145/3364510.3364518.
[62] J. Wang, Q. Xin, Y. Liu, J. Wang, and T. Yang, “Predicting Enterprise Marketing Decision Making with Intelligent Data-Driven Approaches,” J. Ind. Eng. Appl. Sci., vol. 2, no. 3, pp. 12–19, Jun. 2024, doi: 10.5281/zenodo.11357252.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Hao Wang, Laura Robinson, Brandon King (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Creative Commons Attribution 4.0 International (CC BY 4.0).
This work is licensed under a 