Private LLM Serving on Enterprise Container Platforms: A Reference Architecture for GPU Sharing, Version Governance and Policy-Enforced Rollouts in Regulated Industries
Main Article Content
Abstract
Institutions in banking, insurance and healthcare want the productivity gains of large language models (LLMs) but are frequently unable to send prompts, documents or customer records to public inference APIs. Open-weight models released in 2023 make on-premises and private-cloud hosting feasible, yet most enterprises lack a repeatable way to share scarce GPUs across teams, govern model versions and satisfy model-risk and data-residency controls. We present a five-layer reference architecture for private LLM serving on Kubernetes and OpenShift that treats the LLM as another governed workload on an existing enterprise container platform. The layers cover GPU partitioning (dedicated devices, time-slicing and NVIDIA Multi-Instance GPU), a serving runtime with paged key-value cache management and continuous batching, an OCI model registry with signed artifacts, blue-green promotion of model versions with automated latency gates, and policy-as-code admission controls on model sourcing, GPU quotas and network egress. Using a controlled evaluation with Llama 2 7B and 13B variants serving three enterprise workloads, we compare the three GPU sharing modes. MIG partitioning of one A100 80 GB device sustained 1,880 output tokens/s within latency objectives, 14% more than time-slicing, while keeping p95 time to first token 2.2 times lower and providing hard memory isolation; both shared modes cut GPU-hours per million tokens by two-thirds against dedicated devices at nominal load. We map each control to the NIST AI RMF and supervisory model-risk guidance and distil practitioner guidance for selecting a sharing mode
Article Details
Section
How to Cite
References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008.
[2] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, Jul. 2023.
[3] Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency, "Supervisory guidance on model risk management," SR Letter 11-7, Apr. 2011.
[4] National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, Jan. 2023.
[5] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, "Large-scale cluster management at Google with Borg," in Proc. 10th European Conf. Computer Systems (EuroSys), 2015, pp. 1–17.
[6] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, "Borg, Omega, and Kubernetes," ACM Queue, vol. 14, no. 1, pp. 70–93, 2016.
[7] J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Upper Saddle River, NJ, USA: Addison-Wesley, 2010.
[8] M. Fowler, "BlueGreenDeployment," martinfowler.com, Mar. 2010. [Online]. Available: https://martinfowler.com/bliki/BlueGreenDeployment.html
[9] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," in Proc. Machine Learning and Systems (MLSys), vol. 5, 2023.
[10] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, "Orca: A distributed serving system for transformer-based generative models," in Proc. 16th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2022, pp. 521–538.
[11] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with PagedAttention," in Proc. 29th ACM Symp. Operating Systems Principles (SOSP), 2023, pp. 611–626.
[12] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, "FlexGen: High-throughput generative inference of large language models with a single GPU," in Proc. 40th Int. Conf. Machine Learning (ICML), 2023, pp. 31094–31116.
[13] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, "FlashAttention: Fast and memory-efficient exact attention with IO-awareness," in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022.
[14] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, "Clipper: A low-latency online prediction serving system," in Proc. 14th USENIX Symp. Networked Systems Design and Implementation (NSDI), 2017, pp. 613–627.
[15] A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace, "Serving DNNs like Clockwork: Performance predictability from the bottom up," in Proc. 14th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2020, pp. 443–462.
[16] W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang, F. Yang, and L. Zhou, "Gandiva: Introspective cluster scheduling for deep learning," in Proc. 13th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2018, pp. 595–610.
[17] W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia, "AntMan: Dynamic scaling on GPU clusters for deep learning," in Proc. 14th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2020, pp. 533–548.
[18] The Kubernetes Authors, "Device plugins," Kubernetes Documentation, 2023. [Online]. Available: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
[19] NVIDIA Corporation, "Time-slicing GPUs in Kubernetes," NVIDIA GPU Operator Documentation, 2023.
[20] NVIDIA Corporation, "NVIDIA Multi-Instance GPU user guide," NVIDIA Documentation, 2023.
[21] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, "Hidden technical debt in machine learning systems," in Advances in Neural Information Processing Systems (NeurIPS), vol. 28, 2015, pp. 2503–2511.
[22] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, "Model cards for model reporting," in Proc. Conf. Fairness, Accountability, and Transparency (FAT)*, 2019, pp. 220–229.
[23] Z. Newman, J. S. Meyers, and S. Torres-Arias, "Sigstore: Software signing for everybody," in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2022, pp. 2353–2367.
[24] Open Source Security Foundation, "Supply-chain Levels for Software Artifacts (SLSA) specification, version 1.0," Apr. 2023. [Online]. Available: https://slsa.dev/spec/v1.0/
[25] M. Souppaya, J. Morello, and K. Scarfone, "Application container security guide," NIST Special Publication 800-190, Sep. 2017.
[26] Open Policy Agent Project, "Gatekeeper: Policy controller for Kubernetes," OPA Gatekeeper Documentation, 2023. [Online]. Available: https://open-policy-agent.github.io/gatekeeper/
[27] OWASP Foundation, "OWASP Top 10 for Large Language Model Applications, version 1.1," Oct. 2023.