A Systematic Analysis of MLOps Features and Platforms
Keywords:
MLOps, Design space, ML platformsAbstract
While many companies aim to use Machine Learning (ML) models, transitioning to deployment and practical application of such models can be very time-consuming and technically challenging. To address this, MLOps (ML Operations) offers processes, tools, practices, and patterns to bring ML models into operation. A large number of tools and platforms have been created to support developers in creating practical solutions. However, specific needs vary strongly in a situation-dependent manner, and a good overview of their characteristics is missing, making the architect’s task very challenging. We conducted a systematic literature review (SLR) of MLOps platforms, describing their qualities, features, tactics, and patterns. In this paper, we want to map the design space of MLOps platforms. We are guided by the Attribute-Driven Design (ADD) methodology. In this way, we want to provide software architects with a tool to support their work in the platform area.
References
M. Przybyla, “Data scientist vs machine learning ops engineer. here’s the difference.”, [Online]. Available: https://towardsdatascience.com
S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” in IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp. 291–300.
V. Maya and A. Felipe, “The state of MLOps,” 2021. [Online]. Available: https://repositorio.uniandes.edu.co/handle/1992/51495
Valohai, “MLOps - machine learning operations.” [Online]. Available: https://valohai.com/mlops/
MLOps: What it is, why it matters, and how to implement it. [Online]. Available: https://neptune.ai/blog/mlops-what-it-is-why-itmatters-and-how-to-implement-it-from-a-data-scientist-perspective
A. Hein, M. Schreieck, T. Riasanow, D. S. Setzke, M. Wiesche, M. Bohm, and H. Krcmar, “Digital platform ecosystems,” ¨ Electronic Markets, vol. 30, no. 1, pp. 87–98, 2020.
S. Oladele, “Best end-to-end MLOps platforms: Leading machine learning platforms that every data scientist need to know.” [Online]. Available: https://neptune.ai/blog/end-to-end-mlops-platforms
D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, and M. Young, “Machine learning: The high interest credit card of technical debt,” in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop), 2014.
J. George and A. Saha, “End-to-end machine learning using kubeflow,” in 5th Joint International Conference on Data Science; Management of Data. Association for Computing Machinery, 2022, pp. 336–338, event-place: Bangalore, India. [Online]. Available: https://doi.org/10.1145/3493700.3493768
K. K. Gupt, M. A. Raja, A. Murphy, A. Youssef, and C. Ryan, “GELAB – the cutting edge of grammatical evolution,” IEEE Access, vol. 10, pp.
694–38 708, 2022.
Z. Sun, L. Sandoval, R. Crystal-Ornelas, S. M. Mousavi, J. Wang, C. Lin, N. Cristea, D. Tong, W. H. Carande, X. Ma, Y. Rao, J. A. Bednar, A. Tan, J. Wang, S. Purushotham, T. E. Gill, J. Chastang, D. Howard, B. Holt, C. Gangodagamage, P. Zhao, P. Rivas, Z. Chester, J. Orduz, and A. John, “A review of earth artificial intelligence,” Computers and Geosciences, vol. 159, 2022.
T. D. Akinosho, L. O. Oyedele, M. Bilal, A. Y. Barrera-Animas, A.-Q. Gbadamosi, and O. A. Olawale, “A scalable deep learning system for monitoring and forecasting pollutant concentration levels on UK highways,” Ecological Informatics, vol. 69, 2022.
D. De Silva and D. Alahakoon, “An artificial intelligence life cycle: From conception to production,” Patterns, p. 100489, 2022.
G. Symeonidis, E. Nerantzis, A. Kazakis, and G. A. Papakostas, “MLOps - definitions, tools and challenges,” in IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), 2022, pp. 0453–0460.
N. Gift and A. Deza, Practical Mlops: Operationalizing Machine Learning Models. O’Reilly Media, Inc, USA, 2021.
S. Alla and S. K. Adari, Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker, Google Cloud, and Microsoft Azure, 1st ed. Apress, 2020.
V. Lakshmanan, S. Robinson, and M. Munn, Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. O’Reilly UK Ltd., 2020.
D. Kreuzberger, N. Kuhl, and S. Hirschl, “Machine learning operations ¨ (MLOps): Overview, definition, and architecture,” arXiv:2205.02302 [cs], 2022.
S. Choudhary, “Kubernetes-based architecture for an onpremises machine learning platform,” 2021. [Online]. Available: https://aaltodoc.aalto.fi:443/handle/123456789/110516
A. B. Kolltveit and J. Li, “Operationalizing machine learning models: a systematic literature review,” in SE4RAI ’22: Proceedings of the 1st Workshop on Software Engineering for Responsible AI. Association for Computing Machinery, 2022, pp. 1–8.
A. Lima, L. Monteiro, and A. P. Furtado, “Mlops: Practices, maturity models, roles, tools, and challenges-a systematic literature review.” ICEIS (1), pp. 308–320, 2022.
Y. Zhou, Y. Yu, and B. Ding, “Towards MLOps: A Case Study of ML Pipeline Platform,” in International Conference on Artificial Intelligence and Computer Engineering (ICAICE). IEEE, 2020, pp. 23–25.
R. Min˜on, J. Diaz-de Arcaya, A. I. Torre-Bastida, and P. Hartlieb, ´ “Pangea: An MLOps Tool for Automatically Generating Infrastructure and Deploying Analytic Pipelines in Edge, Fog and Cloud Layers,” Sensors, vol. 22, no. 12, p. 4425, 2022.
A. H. Nia, F. J. Kaleibar, F. Feizi, F. Rahimi, and H. Kashfi, “Unlocking the Power of Data in Telecom: Building an Effective MLOps Infrastructure for Model Deployment,” in 2023 7th Iranian Conference on Advances in Enterprise Architecture (ICAEA). IEEE, 2023, pp. 15–16.
G. Zarate, R. Mi ´ n˜on, J. D ´ ´ıaz-de Arcaya, and A. I. Torre-Bastida, “K2e: Building mlops environments for governing data and models catalogues while tracking versions,” in IEEE 19th International Conference on Software Architecture Companion (ICSA-C). IEEE, 2022, pp. 206– 209.
G. Recupito, F. Pecorelli, G. Catolino, S. Moreschini, D. Di Nucci, F. Palomba, and D. A. Tamburri, “A Multivocal Literature Review of MLOps Tools and Features,” in 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 2022, pp. 2022–02.
I. Kumara, R. Arts, D. Di Nucci, W. J. Van Den Heuvel, and D. A. Tamburri, “Requirements and Reference Architecture for MLOps:Insights from Industry,” Authorea Preprints, 2023.
H. Washizaki, H. Uchida, F. Khomh, and Y.-G. Gueh ´ eneuc, “Studying ´ software engineering patterns for designing machine learning systems,” in 2019 10th International Workshop on Empirical Software Engineering in Practice (IWESEP), 2019, pp. 49–495.
R. Subramanya, P. Rais ¨ anen, S. Sierla, and V. Vyatkin, “Cloud comput- ¨ ing design patterns for mlops: applications to virtual power plants,” in
IECON 2023-49th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2023, pp. 01–07.
B. Kitchenham, “Procedures for performing systematic reviews,” Keele, UK, Keele University, vol. 33, pp. 1–26, 2004.
L. Faubel and K. Schmid, “An mlops platform comparison,” Hildesheimer Informatik Berichte, no. 1/2024, SSE 1/24/E, 2024.
E. Raj, Engineering MLOps: Rapidly build, test, and manage productionready machine learning life cycles at scale. Packt Publishing, 2021.
A. Choudhury, “Top 8 alternatives to apache spark.” [Online]. Available: https://analyticsindiamag.com/top-8-alternatives-to-apache-spark/
Y. Gavrilova, “The best open-source MLOps tools you should
D. Meedeniya and H. Thennakoon, “Impact factors and best practices to improve effort estimation strategies and practices in DevOps,” in The 11th International Conference on Information Communication and Management. Association for Computing Machinery, 2021, pp. 11–17.
S. Rahman and E. Kandogan, “Characterizing practices, limitations, and opportunities related to text information extraction workflows: A human-in-the-loop perspective,” in CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3502068
S. Garg, P. Pundir, G. Rathee, P. Gupta, S. Garg, and S. Ahlawat, “On continuous integration / continuous delivery for automated deployment of machine learning models using MLOps,” in 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), 2021, pp. 25–28.
W. Wu and C. Zhang, “Towards understanding end-to-end learning in the context of data: Machine learning dancing over semirings codd’s table,” in Proceedings of the Fifth Workshop on Data Management for End-To-End Machine Learning. Association for Computing Machinery, 2021.
H. Jayalath and L. Ramaswamy, “Enhancing performance of operationalized machine learning models by analyzing user feedback,” in 4th International Conference on Image, Video and Signal Processing. Association for Computing Machinery, 2022, pp. 197–203.
A. Serban, K. van der Blom, H. Hoos, and J. Visser, “Adoption and effects of software engineering best practices in machine learning,” in Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). Association for Computing Machinery, 2020.
H. Cervantes and R. Kazman, Designing Software Architectures: A Practical Approach. Boston, MA, USA: Addison-Wesley Professional, 2016.
S. Giannakopoulou, M. Karpathiotakis, B. Gaidioz, and A. Ailamaki, “CleanM: An optimizable query language for unified scale-out data cleaning,” Proc. VLDB Endow., vol. 10, no. 11, pp. 1466–1477, 2022.
C. Poss, T. Irrenhauser, M. Prueglmeier, D. Goehring, F. Zoghlami, V. Salehi, and O. Ibragimov, “Enabling robot selective trained deep neural networks for object detection through intelligent infrastructure,” in Proceedings of the 4th International Conference on Automation, Control and Robotics Engineering. Association for Computing Machinery, 2019.
N. O. Nikitin, P. Vychuzhanin, M. Sarafanov, I. S. Polonskaia, I. Revin, I. V. Barabanova, G. Maximov, A. V. Kalyuzhnaya, and A. Boukhanovsky, “Automated evolutionary approach for the design of composite machine learning pipelines,” Future Gener. Comput. Syst., vol. 127, pp. 109–125, 2022, place: NLD Publisher: Elsevier Science Publishers B. V.
E. Raj, D. Buffoni, M. Westerlund, and K. Ahola, “Edge MLOps: An automation framework for AIoT applications,” in 2021 IEEE International Conference on Cloud Engineering (IC2E), 2021, pp. 191–200.
H. Liu, Q. Gao, J. Li, X. Liao, H. Xiong, G. Chen, W. Wang, G. Yang, Z. Zha, D. Dong, D. Dou, and H. Xiong, “JIZHI: A fast and costeffective model-as-a-service system for web-scale online inference at baidu,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery ; Data Mining. Association for Computing Machinery, 2021, pp. 3289–3298.
A. Bose and A. Aggarwal, “MLOps – ”Why is it required?” and ”What it is”? - KDnuggets,” 2022. [Online]. Available: https://www.kdnuggets.com/2020/12/mlops-why-required-what-is.html
Z. Li, X.-Y. Liu, J. Zheng, Z. Wang, A. Walid, and J. Guo, “FinRLpodracer: high performance and scalable deep reinforcement learning for quantitative finance,” in Proceedings of the Second ACM International Conference on AI in Finance. Association for Computing Machinery, 2021, pp. 1–9.
Z. Azad, R. Sen, K. Park, and A. Joshi, “Hardware acceleration for DBMS machine learning scoring: Is it worth the overheads?” in IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2021, pp. 243–253.
O. E. Oluyisola, S. Bhalla, F. Sgarbossa, and J. O. Strandhagen, “Designing and developing smart production planning and control systems in the industry 4.0 era: A methodology and case study,” J. Intell. Manuf., vol. 33, no. 1, pp. 311–332, 2022.
M. Oplenskedal, P. Herrmann, and A. Taherkordi, “DeepMatch2: A comprehensive deep learning-based approach for in-vehicle presence detection,” Information Systems, vol. 108, p. 101927, 2022.
J. N. van Rijn, B. Bischl, L. Torgo, B. Gao, V. Umaashankar, S. Fischer, P. Winter, B. Wiswedel, M. R. Berthold, and J. Vanschoren, “OpenML: A collaborative science platform,” in Advanced Information Systems Engineering. Springer Berlin Heidelberg, 2013, vol. 7908, pp. 645–649.
B. Brik, K. Boutiba, and A. Ksentini, “Deep learning for b5g open radio
access network: Evolution, survey, case studies, and challenges,” IEEE Open Journal of the Communications Society, vol. 3, pp. 228–250, 2022.
M. van der Goes, “Scaling enterprise recommender systems for decentralization,” in Fifteenth ACM Conference on Recommender Systems. Association for Computing Machinery, 2021, pp. 592–594.
D. L. Visengeriyeva, A. Kammer, I. Bar, A. Kniesz, and M. Pl ¨ od, ¨ “ml-ops.org.” [Online]. Available: https://ml-ops.org/
“LF AI & Data Landscape,” 2024, [Online; accessed 5. Apr. 2024]. [Online]. Available: https://landscape.lfai.foundation
K. S. d. Prado, “Awesome MLOps,” original-date: 2020-05- 25T22:53:26Z. [Online]. Available: https://github.com/kelvins/awesomemlops
D. Muiruri, L. E. Lwakatare, J. K. Nurminen, and T. Mikkonen, “Practices and infrastructures for ML systems – an interview study,” 2021, publisher: TechRxiv.
L. Silva and F. Osorio, “Flowi: A platform for ML development and management,” 2021, [Online; accessed 14. May 2024]. [Online]. Available: https://proceedings.science/wmecai/wmecai-2021/papers/flowi-aplatform-for-ml-development-and-management?lang=en
L. Cardoso Silva, F. Rezende Zagatti, B. Silva Sette, L. Nildaimon dos Santos Silva, D. Lucredio, D. Furtado Silva, and H. de Medeiros Caseli, ´ “Benchmarking machine learning solutions in production,” in 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 2020, pp. 626–633.
L. Faubel, K. Schmid, and H. Eichelberger, “MLOps Challenges in Industry 4.0,” SN Comput. Sci., vol. 4, no. 6, pp. 828–11, Oct. 2023.
H. Washizaki, H. Uchida, F. Khomh, and Y.-G. Gueh ´ eneuc, “Machine ´ learning architecture and design patterns,” IEEE Software, vol. 8, p. 2020, 2020.
L. Faubel, T. Woudsma, L. Methnani, A. G. Ghezeljhemeidan, F. Buelow, K. Schmid, W. D. van Driel, B. Kloepper, A. Theodorou, M. Nosratinia, and M. Bang, “Towards an mlops architecture for xai in industrial applications,” 2023.
Valohai, “Valohai | Take ML places it’s never been,” Feb. 2023, [Online accessed 23. Feb. 2023]. Available: https://valohai.com
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Leonhard Faubel, Klaus Schmid
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
License Terms:
Except where otherwise noted, content on this website is lincesed under a Creative Commons Attribution Non-Commercial License (CC BY NC)
Use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes, is permitted.
Copyright to any article published by WiPiEC retained by the author(s). Authors grant WiPiEC Journal a license to publish the article and identify itself as the original publisher. Authors also grant any third party the right to use the article freely as long as it is not used for commercial purposes and its original authors, citation details, and publisher are identified, in accordance with CC BY NC license. Fore more information on license terms, click here.