Numerical-Reasoning Guardrails for Financial Chart Vision-Language Models with Evidence Extraction, Calibrated Abstention, and Institutional Risk Dashboards
Keywords:
Financial Chart Question Answering, Vision-Language Models, Numerical Reasoning, OCR Evidence, Tool-Assisted Computation, Confidence Calibration, Selective Prediction, Institutional Risk DashboardAbstract
Financial chart questions require a model to align labels, periods, units, visual marks, and arithmetic before returning a compact answer. This study evaluates a backbone-agnostic numerical guardrail on the archived FinChart-Bench release containing 7,019 annotations over 1,202 chart keys. The system extracts OCR evidence, executes a restricted arithmetic trace, estimates answer correctness, and routes low-confidence cases to review. All questions were assigned to five source-document-grouped folds; each fold served once as test data, while a separate fold calibrated confidence and the remaining folds trained textual baselines. On 2,285 numerical QA records, the evidence executor reached 69.85% strict exact match and 93.39% precision-aware accuracy. Isotonic calibration reduced adaptive expected calibration error from 0.144 to 0.063 and area under the risk-coverage curve from 0.190 to 0.179. A threshold selected under a 10% calibration-fold risk budget answered 43.02% of QA records with 85.25% selective accuracy on held-out source documents. OCR recovered all operands for 30.28% of QA records and at least one operand for 68.80%, showing that arithmetic verification can remain useful even when chart text extraction is incomplete. The resulting dashboard exposes evidence completeness, calculation status, calibrated correctness, and review priority. These findings support selective, auditable assistance rather than unqualified automation in financial chart analysis.
References
[1] D. Shu, H. Yuan, Y. Wang, Y. Liu, H. Zhang, and M. Du, "FinChart-Bench: Benchmarking financial chart comprehension in vision-language models," in Proc. 64th Annu. Meeting Assoc. Comput. Linguistics, 2026, pp. 13447-13466, doi: 10.18653/v1/2026.acl-long.615.
[2] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, "ChartQA: A benchmark for question answering about charts with visual and logical reasoning," in Findings ACL, 2022, pp. 2263-2279, doi: 10.18653/v1/2022.findings-acl.177.
[3] N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar, "PlotQA: Reasoning over scientific plots," in Proc. IEEE WACV, 2020, pp. 1527-1536, doi: 10.1109/WACV45572.2020.9093523.
[4] K. Kafle, B. Price, S. Cohen, and C. Kanan, "DVQA: Understanding data visualizations via question answering," in Proc. IEEE/CVF CVPR, 2018, pp. 5648-5656, doi: 10.1109/CVPR.2018.00592.
[5] J. Luo, Z. Li, J. Wang, and C.-Y. Lin, "ChartOCR: Data extraction from charts images via a deep hybrid framework," in Proc. IEEE/CVF WACV, 2021, pp. 1916-1924, doi: 10.1109/WACV48630.2021.00196.
[6] F. Liu et al., "DePlot: One-shot visual language reasoning by plot-to-table translation," in Findings ACL, 2023, pp. 10381-10399, doi: 10.18653/v1/2023.findings-acl.660.
[7] F. Liu et al., "MatCha: Enhancing visual language pretraining with math reasoning and chart derendering," in Proc. ACL, 2023, pp. 12756-12770, doi: 10.18653/v1/2023.acl-long.714.
[8] K. Lee et al., "Pix2Struct: Screenshot parsing as pretraining for visual language understanding," in Proc. ICML, vol. 202, 2023, pp. 18893-18912.
[9] G. Kim et al., "OCR-free document understanding transformer," in Computer Vision-ECCV 2022, vol. 13688, pp. 498-517, doi: 10.1007/978-3-031-19815-1_29.
[10] A. Masry, P. Kavehzadeh, X. L. Do, E. Hoque, and S. Joty, "UniChart: A universal vision-language pretrained model for chart comprehension and reasoning," in Proc. EMNLP, 2023, pp. 14662-14684, doi: 10.18653/v1/2023.emnlp-main.906.
[11] Y. Han et al., "ChartLlama: A multimodal LLM for chart understanding and generation," arXiv:2311.16483, 2023, doi: 10.48550/arXiv.2311.16483.
[12] F. Meng et al., "ChartAssistant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning," in Findings ACL, 2024, pp. 7775-7803, doi: 10.18653/v1/2024.findings-acl.463.
[13] A. Masry, M. Shahmohammadi, M. R. Parvez, E. Hoque, and S. Joty, "ChartInstruct: Instruction tuning for chart comprehension and reasoning," in Findings ACL, 2024, pp. 10387-10409, doi: 10.18653/v1/2024.findings-acl.619.
[14] A. Masry et al., "ChartQAPro: A more diverse and challenging benchmark for chart question answering," in Findings ACL, 2025, pp. 19123-19151, doi: 10.18653/v1/2025.findings-acl.978.
[15] Z. Chen et al., "FinQA: A dataset of numerical reasoning over financial data," in Proc. EMNLP, 2021, pp. 3697-3711, doi: 10.18653/v1/2021.emnlp-main.300.
[16] F. Zhu et al., "TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance," in Proc. ACL-IJCNLP, 2021, pp. 3277-3287, doi: 10.18653/v1/2021.acl-long.254.
[17] Z. Chen, S. Li, C. Smiley, Z. Ma, S. Shah, and W. Y. Wang, "ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering," in Proc. EMNLP, 2022, pp. 6279-6292, doi: 10.18653/v1/2022.emnlp-main.421.
[18] P. Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen, "FinanceBench: A new benchmark for financial question answering," arXiv:2311.11944, 2023, doi: 10.48550/arXiv.2311.11944.
[19] Z. Gan et al., "MME-Finance: A multimodal finance benchmark for expert-level understanding and reasoning," in Proc. ACM Multimedia, 2025, pp. 12867-12874, doi: 10.1145/3746027.3758230.
[20] Z. Tang et al., "FinMMR: Make financial numerical reasoning more multimodal, comprehensive, and challenging," in Proc. IEEE/CVF ICCV, 2025, pp. 3245-3257.
[21] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in Proc. ICML, vol. 70, 2017, pp. 1321-1330.
[22] Y. Geifman and R. El-Yaniv, "Selective classification for deep neural networks," in Advances in Neural Information Processing Systems 30, 2017.
[23] Y. Geifman and R. El-Yaniv, "SelectiveNet: A deep neural network with an integrated reject option," in Proc. ICML, vol. 97, 2019, pp. 2151-2159.
[24] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, "Conformal risk control," in Proc. ICLR, 2024.
[25] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. I. Jordan, "Distribution-free, risk-controlling prediction sets," J. ACM, vol. 68, no. 6, Art. 43, pp. 1-34, 2021, doi: 10.1145/3478535.
[26] M. Mitchell et al., "Model cards for model reporting," in Proc. FAT*, 2019, pp. 220-229, doi: 10.1145/3287560.3287596.
[27] A. Crisan, M. Drouhard, J. Vig, and N. Rajani, "Interactive model cards: A human-centered approach to model documentation," in Proc. ACM FAccT, 2022, pp. 427-439, doi: 10.1145/3531146.3533108.
[28] M. Arnold et al., "FactSheets: Increasing trust in AI services through supplier's declarations of conformity," IBM J. Res. Develop., vol. 63, no. 4/5, pp. 6:1-6:13, 2019, doi: 10.1147/JRD.2019.2942288.
[29] J. Jin, "Evidence-Chain Reliable RAG: Hallucination Detection, Source Attribution, and Deterministic Provenance Explanations," J. Technol. Informatics Eng., vol. 4, no. 2, pp. 520-533, Aug. 2025, doi: 10.51903/jtie.v4i2.535.
[30] C. Li, J. Bai, and S. Wang, "Evidence-Chain Reliable RAG: Word-Level Hallucination Detection, Source Attribution, and Provenance Explanation for LLM Applications," J. Adv. Comput. Syst., vol. 4, no. 2, pp. 76-92, Feb. 2024, doi: 10.69987/JACS.2024.40207.
[31] B. Zhang, H. Rao, and D. Zhao, "Evidence-Grounded RAG for Cloud-Native DevOps: Hallucination-Resistant AIOps Question Answering over Private Operations Documents," J. Adv. Comput. Syst., vol. 4, no. 3, pp. 109-125, Mar. 2024, doi: 10.69987/JACS.2024.40308.
[32] W. Su, S. Chen, and C. Zhao, "Budgeted Multi-Hop Retrieval Agent for Compositional Question Answering: A Retrieval-Policy Evaluation on the Official MultiHop-RAG Benchmark," J. Technol. Informatics Eng., vol. 4, no. 3, pp. 649-662, Dec. 2025, doi: 10.51903/jtie.v4i3.543.
[33] G. Liu, C. Li, and E. Zhang, "OpsLLM for Cloud Incident Triage: Bilingual RAG-Based Root Cause Analysis and Alert Summarization for AI Infrastructure Operations," J. Adv. Comput. Syst., vol. 4, no. 4, pp. 97-111, Apr. 2024, doi: 10.69987/JACS.2024.40408.
[34] R. Zhang, Z. Wen, C. Wang, C. Tang, P. Xu, and Y. Jiang, "Quality analysis and evaluation prediction of RAG retrieval based on machine learning algorithms," arXiv:2511.19481, 2025, doi: 10.48550/arXiv.2511.19481.
[35] Z. S. Zhong, J. Chen, E. Zhong, and X. Sun, "Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0," J. Technol. Informatics Eng., vol. 4, no. 2, pp. 502-520, Aug. 2025, doi: 10.51903/jtie.v4i2.536.
[36] Z. Li, K. Zhang, and A. Wong, "Numerical-Reasoning Guardrails for a Quant Research Assistant: A Compact Reproducible Benchmark Using SEC and FRED Data," J. Technol. Informatics Eng., vol. 5, no. 2, pp. 75-90, Jun. 2026, doi: 10.51903/jtie.v5i2.541.
[37] S. Meng, J. Chen, and I. Zheng, "LLM-Inspired Offline Reranking for Financial Search: Query Rewriting, Hybrid Retrieval, and Listwise Relevance Ranking on FiQA," J. Technol. Informatics Eng., vol. 5, no. 1, pp. 361-378, Apr. 2026, doi: 10.51903/jtie.v5i1.537.
[38] Y. Chen, S. Zhou, and E. Lin, "Accounting-Aware Evidence Retrieval for Institutional Due Diligence of Tokenized Trade Receivable RWA," J. Technol. Informatics Eng., vol. 4, no. 3, pp. 649-663, Dec. 2025, doi: 10.51903/jtie.v4i3.542.
[39] K. Zhang, Y. Chen, and A. Qian, "Evidence-Grounded Accounting Disclosure Review Cards: A Visual Communication Framework for LLM-Style Explanations over SEC Financial Statements and Notes," Int. J. Graph. Des., vol. 3, no. 2, p. 395, Oct. 2025, doi: 10.51903/ijgd.v3i2.3710.
[40] S. Zhou, Y. Chen, and K. Lee, "Accounting-Aware Evidence-Constrained Agents for Disclosure, Settlement, and Secondary-Market Risk Monitoring in Tokenized," J. Technol. Informatics Eng., vol. 5, no. 2, pp. 60-74, 2026, doi: 10.51903/jtie.v5i2.544.
[41] Q. Xin, "Explainable and Fair Credit Risk Scoring with Counterfactual Explanations: A Reproducible Evaluation on the German Credit Dataset (HELOC-Motivated)," J. Inf. Technol., vol. 14, no. 2, pp. 215-231, Jun. 2026, doi: 10.32664/j-intech.v14i02.2228.
[42] J. Jin, T. Huang, and S. Lu, "A Model-Risk-Friendly Probability of Default Workflow: Calibration, Distribution-Free Uncertainty Quantification, and SHAP Explanations on the UCI Credit Card Default Dataset," J. Adv. Comput. Syst., vol. 4, no. 6, pp. 74-85, Jun. 2024, doi: 10.69987/JACS.2024.40606.
[43] Y. Chen, Y. Zhang, D. Chau, and M. Sherman, "Credit Card Default Risk Tiering with Probability Calibration and Uncertainty-Driven Rejection: A Reproducible Study on the UCI Credit Card Clients Dataset," J. Adv. Comput. Syst., vol. 3, no. 4, pp. 31-47, Apr. 2023, doi: 10.69987/JACS.2023.30403.
[44] Z. Li, S. Zhou, and Z. Zhou, "Financial Risk Dashboard Design for Institutional RWA Investors: Visual Hierarchy, Chart Comprehension, and Explainability in FinChart-Bench," Int. J. Graph. Des., vol. 3, no. 1, pp. 196-210, May 2025, doi: 10.51903/ijgd.v3i1.3715.
[45] Q. Xin, "Uncertainty-Aware Late Fusion for 3D Perception (Confidence Calibration + Fusion Rule Learning)," J. Technol. Informatics Eng., vol. 4, no. 1, pp. 215-238, Apr. 2025, doi: 10.51903/jtie.v4i1.485.
[46] S. He, H. Tu, and I. Liu, "Safe PD Capacity Forecasting with Time-Series Foundation Models and Calibrated Uncertainty for Heterogeneous GPU Clusters," J. Adv. Comput. Syst., vol. 3, no. 4, pp. 48-66, Apr. 2023, doi: 10.69987/JACS.2023.30404.
[47] J. Jin, "Calibrated Resume-Job Matching for Trustworthy LLM-Assisted Recruiter Screening: Pairwise Matching, Probability Calibration, and Selective Refusal on Two Public Recruitment Datasets," J. Technol. Informatics Eng., vol. 4, no. 3, pp. 625-648, Dec. 2025, doi: 10.51903/jtie.v4i3.529.
[48] S. Chen, S. He, and E. Sun, "Risk-Bounded GPU Resource Oversubscription via Conformal Demand Envelopes in Production AI Clusters," J. Adv. Comput. Syst., vol. 4, no. 5, pp. 119-134, May 2024, doi: 10.69987/JACS.2024.40509.
[49] Z. S. Zhong, C. Li, and H. Rao, "Trajectory Reliability Prediction for Generalist AI Agents: Tool-Use Failure Analysis and Success Forecasting on ZClawBench," J. Technol. Informatics Eng., vol. 5, no. 1, pp. 341-360, Apr. 2026, doi: 10.51903/jtie.v5i1.539.
[50] S. He, X. Chang, and E. Sun, "Cross-Cloud Transfer Learning for AI Training Capacity Forecasting under Workload and Topology Distribution Shift," J. Adv. Comput. Syst., vol. 4, no. 1, pp. 100-120, Jan. 2024, doi: 10.69987/JACS.2024.40108.
[51] Q. Xin, "Probabilistic Bike-Sharing Demand Forecasting under Changing Weather and Seasonal Regimes with Transformer-Based Models," Transport Findings, Mar. 2026, doi: 10.32866/001c.157499.
[52] J. Jin, "LLM-Style Evidence Cards for Scientific Search Interfaces: A UI/UX Design Framework for Retrieval Transparency, Ranking Trust, and Visual Evidence Hierarchy," Int. J. Graph. Des., vol. 3, no. 2, pp. 397-414, Oct. 2025, doi: 10.51903/ijgd.v3i2.3698.
[53] G. Liu, S. He, and H. Wong, "LLM-Compatible Visual Brief Cards for AI Infrastructure Capacity Dashboards: A UI/UX Framework for Turning Forecast Risk into Graphic Design Decisions," Int. J. Graph. Des., vol. 3, no. 1, pp. 196-213, May 2025, doi: 10.51903/ijgd.v3i1.3723.
[54] W. Su, H. Rao, and E. Ma, "Privacy and Data-Integrity Risk Cards for LLM Agents: A UI/UX Design Framework for Secure Human Oversight under Prompt-Injection Attacks," Int. J. Graph. Des., vol. 4, no. 1, pp. 186-191, Apr. 2026, doi: 10.51903/ijgd.v4i1.3699.
[55] Y. Li, S. Lu, and L. Zhao, "LLM-as-Design-Critic: Aligning AI-Generated UI Feedback with Human Graphic Design Judgment," Int. J. Graph. Des., vol. 3, no. 1, pp. 196-215, May 2025, doi: 10.51903/ijgd.v3i1.3661.
[56] J. Nie, G. Liu, C. Li, and T. Zou, "Evidence-Constrained Incident Visualization Cards for Distributed Cloud Logs: A UI/UX Framework for Turning Hadoop, OpenStack, and ZooKeeper Logs into Actionable SRE Design Interfaces," Int. J. Graph. Des., vol. 4, no. 1, pp. 179-185, Apr. 2026, doi: 10.51903/ijgd.v4i1.3703.
[57] Q. Xin, "Hybrid Cloud Architecture for Efficient and Cost-Effective Large Language Model Deployment," J. Inf. Syst. Informatics, vol. 7, no. 3, pp. 2182-2195, Sep. 2025, doi: 10.51519/journalisi.v7i3.1170.
[58] D. Zheng, B. Zhang, and J. Geibel, "VerifySafe: Toxicity-Safe Agent Responses under Adversarial Prompts with Evidence-Based Self-Verification," J. Adv. Comput. Syst., vol. 4, no. 1, pp. 67-82, Jan. 2024, doi: 10.69987/JACS.2024.40106.
[59] Q. Xin, "Explaining OpenStack Failure-Injection Log Anomalies with Retrieved Normal Prototypes," Emerg. Inf. Sci. Technol., vol. 6, no. 2, Nov. 2025, doi: 10.18196/eist.v6i2.31232.
[60] D. Zheng and C. Li, "Behavior-Level Jailbreak Resistance via Multi-Stage Refusal + Utility Preservation," J. Adv. Comput. Syst., vol. 4, no. 1, pp. 83-99, Jan. 2024, doi: 10.69987/JACS.2024.40107.
[61] Q. Xin, "Log Anomaly Detection with Conformal Alert Control and Evidence-Grounded Incident Ticket Generation," AVITEC, vol. 8, no. 2, pp. 247-264, May 2026, doi: 10.28989/avitec.v8i2.3974.
[62] D. Zheng, C. Li, and H. Davidson, "Continual Red-Teaming for In-the-Wild Jailbreaks via Online Guardrail Updates and Guardrail Distillation," J. Adv. Comput. Syst., vol. 3, no. 2, pp. 35-49, Feb. 2023, doi: 10.69987/JACS.2023.30203.
[63] Q. Xin, "LiDAR-Camera Object-Level Fusion for Multi-Target Tracking Using JPDA and EKF: A Reproducible Empirical Study on a PandaSet-Parameterised Five-Sequence Dataset," J. Technol. Informatics Eng., vol. 5, no. 1, pp. 54-76, Apr. 2026, doi: 10.51903/jtie.v5i1.486.
[64] Y. Li, "Findable then Explainable: Retrieval-Summary Integration for Code Intelligence on a Lightweight CodeSearchNet Subset," J. Adv. Comput. Syst., vol. 4, no. 7, pp. 65-82, Jul. 2024, doi: 10.69987/JACS.2024.40706.
[65] Q. Xin, "Host-Based Intrusion Detection with System Call Sequences: Window Localization and Forensic Narratives," AVITEC, vol. 8, no. 2, pp. 325-334, Jun. 2026, doi: 10.28989/avitec.v8i2.3973.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Benjamin Harris, Derek Guan, Charlotte Martin (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Creative Commons Attribution 4.0 International (CC BY 4.0).
This work is licensed under a 