Conformal GPU Demand Envelopes and Power-Aware Scheduling for Heterogeneous AI Clusters under Cold Start and Cross-Cloud Shift
Keywords:
AI Cluster Scheduling, Conformal Prediction, GPU Demand Forecasting, Heterogeneous Accelerators, Cold Start, Cross-Cloud Dispatch, Power-Aware Computing, Risk-Controlled Overbooking, Burstgpt, SustaindcAbstract
Production LLM traffic is bursty, and capacity decisions couple GPU availability with power and carbon. We convert probabilistic demand forecasts into risk-controlled admission, heterogeneous allocation, calibrated overbooking, and cross-site dispatch. The evaluation uses all 1,429,737 requests in the 61-day BurstGPT trace and SustainDC carbon, weather, workload, configuration, and physical power model. Requests form a complete five-minute grid with a weighted-token target. A chronological split assigns 42 days to fitting, nine to conformal calibration, and ten to testing. Point models include persistence, seasonality, Ridge, histogram gradient boosting, and Extra Trees; interval models include direct quantiles, split, rolling, regime-conditioned, and conformalized quantile regression.
Histogram gradient boosting attained 49.68% WAPE versus 115.81% for daily seasonality and reduced mean absolute error by 31,545.90 weighted tokens/bin (95% block-bootstrap interval, 23,152.07–37,746.37). Rolling conformal achieved 90.00% coverage at mean width 86,728.29. With a calibration-selected 0.80 overbooking factor, the proposed policy recorded 5.11% bin-level SLO violations and reduced emissions from 52.51 to 12.57 tCO2e through SustainDC-aware dispatch. The remaining 8.81% workload shortfall quantifies burst risk; GPU type, site, and price remain scenario inputs.
References
[1] Y. Wang, Y. Chen, Z. Li, X. Kang, Y. Fang, Y. Zhou, Y. Zheng, Z. Tang, X. He, R. Guo, X. Wang, Q. Wang, A. C. Zhou, and X. Chu, “BurstGPT: A real-world workload dataset to optimize LLM serving systems,” in Proc. 31st ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), vol. 2, pp. 5831–5841, 2025, doi: 10.1145/3711896.3737413.
[2] A. Naug, A. Guillen, R. Luna, V. Gundecha, C. Bash, S. Ghorbanpour, S. Mousavi, A. Ramesh Babu, D. Markovikj, L. D. Kashyap, D. Rengarajan, and S. Sarkar, “SustainDC: Benchmarking for sustainable data center control,” in Advances in Neural Information Processing Systems, vol. 37, Datasets and Benchmarks Track, 2024, doi: 10.52202/079017-3192.
[3] G. Ke et al., “LightGBM: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
[4] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” arXiv preprint arXiv:1803.01271, 2018.
[5] A. Das, W. Kong, R. Sen, and Y. Zhou, “A decoder-only foundation model for time-series forecasting,” in Proc. 41st Int. Conf. Machine Learning, PMLR, vol. 235, pp. 10148–10167, 2024.
[6] S. He, H. Tu, and I. Liu, “Safe PD capacity forecasting with time-series foundation models and calibrated uncertainty for heterogeneous GPU clusters,” J. Adv. Comput. Syst., vol. 3, no. 4, pp. 48–66, Apr. 2023, doi: 10.69987/JACS.2023.30404.
[7] S. Zhao, J. Bai, and D. Roberson, “Multi-horizon GPU demand forecasting with workload semantics and operational risk curves: An empirical study on Alibaba Clusterdata GPU Trace,” J. Technol. Informatics Eng., vol. 4, no. 3, pp. 544–571, Dec. 2025, doi: 10.51903/jtie.v4i3.498.
[8] C. Wang, Z. Wen, R. Zhang, P. Xu, and Y. Jiang, “GPU memory requirement prediction for deep learning task based on bidirectional gated recurrent unit optimization Transformer,” in Proc. 5th Int. Conf. Artificial Intelligence, Virtual Reality and Visualization (AIVRV), Chengdu, China, pp. 31–35, 2025, doi: 10.1109/AIVRV67401.2025.11350369.
[9] Q. Xin, “Probabilistic bike-sharing demand forecasting under changing weather and seasonal regimes with transformer-based models,” Findings, Mar. 2026, doi: 10.32866/001c.157499.
[10] S. He, C. Li, and H. Rao, “Few-shot cold-start workload forecasting for new AI inference tenants with time-series foundation models,” J. Technol. Informatics Eng., vol. 4, no. 1, pp. 306–324, Apr. 2025, doi: 10.51903/jtie.v4i1.546.
[11] Z. S. Zhong, X. Pan, and Q. Lei, “Bridging domains with approximately shared features,” in Proc. 28th Int. Conf. Artificial Intelligence and Statistics (AISTATS), PMLR, vol. 258, pp. 559–567, 2025.
[12] Q. Xin, “Hybrid cloud architecture for efficient and cost-effective large language model deployment,” J. Inf. Syst. Informatics, vol. 7, no. 3, pp. 2182–2195, Sep. 2025, doi: 10.51519/journalisi.v7i3.1170.
[13] Y. Romano, E. Patterson, and E. J. Candès, “Conformalized quantile regression,” in Advances in Neural Information Processing Systems, vol. 32, 2019.
[14] I. Gibbs and E. Candès, “Adaptive conformal inference under distribution shift,” in Advances in Neural Information Processing Systems, vol. 34, 2021.
[15] C. Xu and Y. Xie, “Sequential predictive conformal inference for time series,” in Proc. 40th Int. Conf. Machine Learning, PMLR, vol. 202, 2023.
[16] R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability,” Ann. Statist., vol. 51, no. 2, pp. 816–845, 2023, doi: 10.1214/23-AOS2276.
[17] Q. Xin, “Uncertainty-aware late fusion for 3D perception (confidence calibration + fusion rule learning),” J. Technol. Informatics Eng., vol. 4, no. 1, pp. 215–238, Apr. 2025, doi: 10.51903/jtie.v4i1.485.
[18] S. Chen, S. He, and E. Sun, “Risk-bounded GPU resource oversubscription via conformal demand envelopes in production AI clusters,” J. Adv. Comput. Syst., vol. 4, no. 5, pp. 119–134, May 2024, doi: 10.69987/JACS.2024.40509.
[19] W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” in Proc. 29th ACM Symp. Operating Systems Principles (SOSP), pp. 611–626, 2023, doi: 10.1145/3600006.3613165.
[20] G.-I. Yu et al., “Orca: A distributed serving system for Transformer-based generative models,” in Proc. 16th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 521–538, 2022.
[21] Z. Li et al., “AlpaServe: Statistical multiplexing with model parallelism for deep learning serving,” in Proc. 17th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 663–679, 2023.
[22] Y. Zhong et al., “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 193–210, 2024.
[23] B. Sun et al., “Llumnix: Dynamic scheduling for large language model serving,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 173–191, 2024.
[24] P. Patel et al., “Splitwise: Efficient generative LLM inference using phase splitting,” in Proc. 51st Annu. Int. Symp. Computer Architecture (ISCA), pp. 118–132, 2024, doi: 10.1109/ISCA59077.2024.00019.
[25] Y. Fu et al., “ServerlessLLM: Low-latency serverless inference for large language models,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 135–153, 2024.
[26] X. Miao et al., “SpotServe: Serving generative large language models on preemptible instances,” in Proc. 29th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS), vol. 2, pp. 1112–1127, 2024, doi: 10.1145/3620665.3640411.
[27] Y. Mei et al., “Helix: Serving large language models over heterogeneous GPUs and network via max-flow,” in Proc. 30th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS), vol. 1, pp. 586–602, 2025, doi: 10.1145/3669940.3707215.
[28] T. Griggs et al., “Mélange: Cost efficient large language model serving by exploiting GPU heterogeneity,” arXiv preprint arXiv:2404.14527, 2024.
[29] S. Zhao, Y. Ren, and X. Chang, “Profit-aware spot GPU admission control with cost-sensitive loss and evidence-grounded policy memos for AI workload supply-demand matching,” J. Technol. Informatics Eng., vol. 5, no. 2, pp. 45–59, 2026, doi: 10.51903/jtie.v5i2.545.
[30] S. He, J. Nie, and C. Li, “Power-aware inventory planning for AI infrastructure using job-level forecasting and LLM workload explanations,” J. Technol. Informatics Eng., vol. 5, no. 1, Apr. 2026, doi: 10.51903/jtie.v5i1.548.
[31] J. Nie and D. Zheng, “Noisy-neighbor-aware VM degradation risk modeling with unsupervised residual fusion,” J. Adv. Comput. Syst., vol. 4, no. 4, pp. 112–123, Apr. 2024, doi: 10.69987/JACS.2024.40409.
[32] D. Crankshaw et al., “InferLine: Latency-aware provisioning and scaling for prediction serving pipelines,” in Proc. 11th ACM Symp. Cloud Computing (SoCC), pp. 477–491, 2020, doi: 10.1145/3419111.3421285.
[33] K. Razavi et al., “FA2: Fast, accurate autoscaling for serving deep learning inference with SLA guarantees,” in Proc. IEEE 28th Real-Time and Embedded Technology and Applications Symp. (RTAS), pp. 146–159, 2022, doi: 10.1109/RTAS54340.2022.00020.
[34] C. Zhang et al., “MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving,” in Proc. USENIX Annu. Technical Conf. (USENIX ATC), pp. 1049–1062, 2019.
[35] H. Qiu et al., “Power-aware deep learning model serving with μ-Serve,” in Proc. USENIX Annu. Technical Conf. (USENIX ATC), pp. 75–93, 2024.
[36] J. Stojkovic et al., “DynamoLLM: Designing LLM inference clusters for performance and energy efficiency,” in Proc. IEEE Int. Symp. High-Performance Computer Architecture (HPCA), pp. 1348–1362, 2025, doi: 10.1109/HPCA61900.2025.00102.
[37] B. Acun et al., “Carbon Explorer: A holistic framework for designing carbon-aware datacenters,” in Proc. 28th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS), vol. 2, pp. 118–132, 2023, doi: 10.1145/3575693.3575754.
[38] A. Radovanović et al., “Carbon-aware computing for datacenters,” IEEE Trans. Power Syst., vol. 38, no. 2, pp. 1270–1280, 2023, doi: 10.1109/TPWRS.2022.3173250.
[39] W. A. Hanafy, Q. Liang, N. Bashir, D. Irwin, and P. Shenoy, “CarbonScaler: Leveraging cloud workload elasticity for optimizing carbon-efficiency,” Proc. ACM Meas. Anal. Comput. Syst., vol. 7, no. 3, Art. no. 57, 2023, doi: 10.1145/3626788.
[40] D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, and M. Zaharia, “Heterogeneity-aware cluster scheduling policies for deep learning workloads,” in Proc. 14th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 481–498, 2020.
[41] A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing, “Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning,” in Proc. 15th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 1–18, 2021.
[42] A. Patke, D. Reddy, S. Jha, H. Qiu, C. Pinto, C. Narayanaswami, Z. Kalbarczyk, and R. Iyer, “Queue management for SLO-oriented large language model serving,” in Proc. ACM Symp. Cloud Computing (SoCC), pp. 18–35, 2024, doi: 10.1145/3698038.3698523.
[43] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 117–134, 2024.
[44] S. J. Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, and G. R. Ganger, “Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling,” in Proc. 29th ACM Symp. Operating Systems Principles (SOSP), pp. 642–657, 2023, doi: 10.1145/3600006.3613175.
[45] A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gulavani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for LLM inference,” Proc. Mach. Learn. Syst., vol. 6, pp. 351–366, 2024.
[46] H. Moore, S. Qi, N. Hogade, D. S. Milojicic, C. E. Bash, and S. Pasricha, “Sustainable carbon-aware and water-efficient LLM scheduling in geo-distributed cloud datacenters,” in Proc. 35th Great Lakes Symp. VLSI (GLSVLSI), pp. 929–934, 2025, doi: 10.1145/3716368.3735301.
[47] A. Choudhury et al., “MAST: Global scheduling of ML training across geo-distributed datacenters at hyperscale,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), pp. 563–580, 2024.
[48] Z. Yang et al., “SkyPilot: An intercloud broker for sky computing,” in Proc. 20th USENIX Symp. Networked Systems Design and Implementation (NSDI), pp. 437–455, 2023.
[49] T. Sukprasert, A. Souza, N. Bashir, D. Irwin, and P. Shenoy, “On the limitations of carbon-aware temporal and spatial workload shifting in the cloud,” in Proc. 19th European Conf. Computer Systems (EuroSys), pp. 924–941, 2024, doi: 10.1145/3627703.3650079.
[50] G. Liu, S. He, and H. Wong, “LLM-compatible visual brief cards for AI infrastructure capacity dashboards: A UI/UX framework for turning forecast risk into graphic design decisions,” Int. J. Graph. Des., vol. 3, no. 1, pp. 196–213, May 2025, doi: 10.51903/ijgd.v3i1.3723.
[51] Q. Xin, “Log anomaly detection with conformal alert control and evidence-grounded incident ticket generation,” AVITEC, vol. 8, no. 2, pp. 247–264, 2026, doi: 10.28989/avitec.v8i2.3974.
[52] G. Liu, S. He, and I. Liu, “LLM-augmented multi-source root cause attribution for CPU and network faults in microservices,” J. Adv. Comput. Syst., vol. 3, no. 6, pp. 39–57, Jun. 2023, doi: 10.69987/JACS.2023.30604.
[53] B. Zhang, H. Rao, and D. Zhao, “Evidence-grounded RAG for cloud-native DevOps: Hallucination-resistant AIOps question answering over private operations documents,” J. Adv. Comput. Syst., vol. 4, no. 3, pp. 109–125, Mar. 2024, doi: 10.69987/JACS.2024.40308.
[54] Q. Xin, Z. Xu, L. Guo, F. Zhao, and B. Wu, “IoT traffic classification and anomaly detection method based on deep autoencoders,” Appl. Comput. Eng., vol. 69, no. 1, pp. 64–70, Jul. 2024, doi: 10.54254/2755-2721/69/20241511.
[55] X. Sun, Z. S. Zhong, and Q. Wu, “Retrieval-grounded HDFS log anomaly detection and deterministic failure narrative generation,” J. Comput. Syst. Appl., vol. 3, no. 1, pp. 15–30, Jun. 2026, doi: 10.64229/j6d7fr94.
[56] Q. Xin, “Self-supervised log anomaly detection with LogBERT-style transformers: Full empirical evaluation on a reproducible SynHDFS benchmark,” J. Electr. Eng. Comput. Sci., vol. 11, no. 1, pp. 23–35, May 2026, doi: 10.54732/jeecs.v11i1.3.
[57] J. Bai, S. Chen, D. Zheng, and M.-J. Kuo, “Interpretable attack-chain stage detection from AWS CloudTrail event sequences via linear models and HMM smoothing,” Inf. Electr. Electron. Eng., vol. 6, no. 1, pp. 28–43, May 2026, doi: 10.33474/infotron.v6i1.24923.
[58] Q. Xin, “Explaining OpenStack failure-injection log anomalies with retrieved normal prototypes,” Emerg. Inf. Sci. Technol., vol. 6, no. 2, pp. 125–146, Nov. 2025, doi: 10.18196/eist.v6i2.31232.
[59] W. Su, H. Rao, and E. Ma, “Privacy and data-integrity risk cards for LLM agents: A UI/UX design framework for secure human oversight under prompt-injection attacks,” Int. J. Graph. Des., vol. 4, no. 1, pp. 186–191, Apr. 2026, doi: 10.51903/ijgd.v4i1.3699.
[60] Q. Xin, “Host-based intrusion detection with system call sequences: Window localization and forensic narratives,” AVITEC, vol. 8, no. 2, pp. 325–334, Aug. 2026, doi: 10.28989/avitec.v8i2.3973.
[61] J. Nie, G. Liu, C. Li, and T. Zou, “Evidence-constrained incident visualization cards for distributed cloud logs: A UI/UX framework for turning Hadoop, OpenStack, and ZooKeeper logs into actionable SRE design interfaces,” Int. J. Graph. Des., vol. 4, no. 1, pp. 179–185, Apr. 2026, doi: 10.51903/ijgd.v4i1.3703.
[62] Z. S. Zhong, C. Li, and H. Rao, “Trajectory reliability prediction for generalist AI agents: Tool-use failure analysis and success forecasting on ZClawBench,” J. Technol. Informatics Eng., vol. 5, no. 1, pp. 341–360, Apr. 2026, doi: 10.51903/jtie.v5i1.539.
[63] B. Zhang, X. Sun, G. Liu, and B. Zhou, “LLM-style DevOps copilot for cloud-native troubleshooting: Retrieval-augmented runbook generation and command-safety evaluation,” J. Technol. Informatics Eng., vol. 5, no. 2, pp. 104–118, Aug. 2026, doi: 10.51903/jtie.v5i2.534.
[64] C. Li, G. Liu, and Z. Zhao, “Cost-aware LLM-style routing for AIOps log analysis: Log parsing, anomaly detection, fault diagnosis, and incident summarization on LogEval task files,” J. Technol. Informatics Eng., vol. 5, no. 2, pp. 91–103, 2026, doi: 10.51903/jtie.v5i2.538.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Sijia Chen, Marcus Reed, Hong Zhang (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Creative Commons Attribution 4.0 International (CC BY 4.0).
This work is licensed under a 