Event-Driven Enterprise Applications with Apache Kafka and Resilient Cloud-Native Architecture
Main Article Content
Abstract
Increasingly, the current business environment is requiring more real-time, scaled and fault tolerant application designs with the ability to accommodate large volume event in distributed cloud environments. Monolithic and request-based systems have been found to have issues with latency, lack of scalability and resilience when it comes to working with ever-generated enterprise data. The study introduces an Event-Driven Enterprise Application Framework that is based on an Apache Kafka and a Resilient Cloud-Native Architecture to facilitate the processing of events in a reliable, asynchronous and intelligent manner. The suggested framework is a combination of event producers, Apache Kafka clusters, distributed message topics, stream processing engines, microservices, API gateways, Kubernetes-based container orchestration, service mesh, cloud-native databases, distributed caching, observability services, and automated recovery mechanisms. The architecture uses event sourcing, Command Query Responsibility Segregation (CQRS), schema management, fault tolerance, horizontal auto-scaling, and AI-assisted monitoring to enhance the responsiveness of the system, its operational reliability, and resource usage. Cloud-native technologies provide dynamic workload management, self-healing deployments, continuous integration and continuous deployment (CI/CD) and secure communication between distributed services. Additionally, full-scale monitoring, distributed tracing, and predictive analytics allows detecting anomalies and optimization of performance proactively. The proposed architecture provides a versatile and strong foundation of enterprise applications utilized in the domain of finance and healthcare, retail, manufacturing, logistics, and IoT ecosystems in cases when low-latency event processing and high system availability are essential. The framework illustrates how the event-driven principles and cloud-native resiliency can contribute greatly to the agility of the enterprise, operational efficiency, fault tolerance, and business continuity as part of the contemporary digital transformation initiatives.
Article Details
Section
How to Cite
References
[1] A. Balalaie, A. Heydarnoori, and P. Jamshidi, “Microservices architecture enables DevOps: Migration to a cloud-native architecture,” IEEE Software, vol. 33, no. 3, pp. 42–52, May–Jun. 2016, doi: 10.1109/MS.2016.64.
[2] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and Kubernetes,” Communications of the ACM, vol. 59, no. 5, pp. 50–57, May 2016, doi: 10.1145/2890784.
[3] D. Gannon, R. Barga, and N. Sundaresan, “Cloud-native applications,” IEEE Cloud Computing, vol. 4, no. 5, pp. 16–21, Sep.–Oct. 2017, doi: 10.1109/MCC.2017.4250939.
[4] N. Kratzke and P.-C. Quint, “Understanding cloud-native applications after 10 years of cloud computing—A systematic mapping study,” Journal of Systems and Software, vol. 126, pp. 1–16, Apr. 2017, doi: 10.1016/j.jss.2017.01.001.
[5] J. Soldani, D. A. Tamburri, and W.-J. Van Den Heuvel, “The pains and gains of microservices: A systematic grey literature review,” Journal of Systems and Software, vol. 146, pp. 215–232, Dec. 2018, doi: 10.1016/j.jss.2018.09.082.
[6] S. Henning and W. Hasselbring, “Theodolite: Scalability benchmarking of distributed stream processing engines in microservice architectures,” Big Data Research, vol. 25, Art. no. 100209, Sep. 2021, doi: 10.1016/j.bdr.2021.100209.
[7] S. Henning and W. Hasselbring, “Scalable and reliable multi-dimensional sensor data aggregation in data-streaming architectures,” Data-Enabled Discovery and Applications, vol. 4, no. 1, 2020, doi: 10.1007/s41688-020-00041-3.
[8] M. Fragkoulis, P. Carbone, V. Kalavri, and A. Katsifodimos, “A survey on the evolution of stream processing systems,” arXiv preprint arXiv:2008.00842, 2020.
[9] H. Nasiri, S. Nasehi, and M. Goudarzi, “Evaluation of distributed stream processing frameworks for IoT applications in smart cities,” Journal of Big Data, vol. 6, no. 52, 2019, doi: 10.1186/s40537-019-0215-2.
[10] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache Flink: Stream and batch processing in a single engine,” IEEE Data Engineering Bulletin, vol. 38, no. 4, pp. 28–38, Dec. 2015.
[11] A. Avritzer, V. Ferme, A. Janes, B. Russo, W. van Hoorn, H. Schulz, D. Menasché, and V. Rufino, “Scalability assessment of microservice architecture deployment configurations: A domain-based approach leveraging operational profiles and load tests,” Journal of Systems and Software, vol. 165, Art. no. 110564, Jul. 2020, doi: 10.1016/j.jss.2020.110564.
[12] G. Brataas, A. Martini, G. K. Hanssen, and G. Ræder, “Agile elicitation of scalability requirements for open systems: A case study,” Journal of Systems and Software, vol. 182, Art. no. 111064, Dec. 2021, doi: 10.1016/j.jss.2021.111064.
[13] A. V. Papadopoulos, L. Versluis, A. Bauer, et al., “Methodological principles for reproducible performance evaluation in cloud computing,” IEEE Transactions on Software Engineering, vol. 47, no. 8, pp. 1528–1543, Aug. 2021, doi: 10.1109/TSE.2019.2927908.
[14] M. Kleppmann, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. Sebastopol, CA, USA: O'Reilly Media, 2017