Apache Flink Journey @Zomato: From Inception to Innovation

For restaurants, the choice of POS system is critical. POS players differ in terms of feature availability, speed of building new features, tech stability and scalability. POS systems also play a crucial role in ensuring smooth order processing and operations management. They have evolved to become integrated platforms that handle inventory management, handle staff payroll, and real-time analytics across multiple channels &#8211; online aggregators, dining, and take-away.



Over the last several years, Zomato, in partnership with various POS players, has worked deeply to innovate on new features, helping restaurants transform the way they manage and optimize their operations. As the restaurant industry scales, there is a need to enable POS players, both new and existing, to be able to solve problems in a quick and efficient manner for more restaurants.



As a first step to achieve this goal, Zomato has launched a developer platform for POS partners, to ensure quick, reliable integrations that enhance restaurant operations. This platform will offer easy-to-read documentation, real-time API testing, and an entire tool-kit for POS developers, thereby enabling POS partners to introduce many more features at scale.



Key Features



Zomato’s POS Developer Platform is built to streamline integration and make life easier for POS developers, with features that include:



<ol class="wp-block-list">
<li>Quick onboarding and integration: Developers can easily create an account and access easy-to-read integration guide; providing step-by-step instructions, reducing confusion and saving valuable development time.</li>



<li>Unified API documentation: The platform brings all API technical documentation together in one place, making it simple for developers to find what they need.</li>



<li>Real-time API testing: POS partners can test APIs and webhooks in real time, which speeds up the development process by allowing immediate troubleshooting.</li>
</ol>



With Zomato’s new developer platform, POS partners can now roll out new features for restaurants quickly in significantly shorter development cycles, allowing restaurants to focus on what matters most: making awesome food and offering a great customer experience.



If you’re a POS partner looking to streamline your integration with Zomato, the new Developer Platform is ready for you. Sign up, and begin your integration process today by visiting the POS<a href="https://www.zomato.com/developer/integration/"> Developer Platform</a>. If you are a restaurant looking for a POS system, visit the <a href="https://www.zomato.com/restaurant-services-hub/pos/">Restaurant Services Hub</a> to make an informed choice.



For any queries or feedback, please write to us on <a href="mailto:posintegarations@zomato.com">posintegarations@zomato.com</a>.

Introducing POS Developer Platform: Simplifying integration with easy-to-use tools

Migrating to VictoriaMetrics: A Complete Overhaul for Enhanced Observability

Imagine this: you&#8217;re in a war room atmosphere. Tension hangs thick in the air as your application team and you scramble to debug a sudden spike in OOM issues plaguing your application’s containers. No memory leaks seem to be the culprit, but further investigation reveals a shocking truth: the garbage collector (GC) wasn&#8217;t running when it should have been! This crucial cleanup process failed to free up memory before the memory breached the container’s hard limit, triggering those dreaded OOM issues.



Now, let&#8217;s look at a different scenario. You&#8217;re analyzing your application’s performance profile, and a particular pattern jumps out. A significant chunk of your application’s CPU usage is being devoured by the GC. While your service doesn&#8217;t necessarily require a ton of live heap memory, the constant GC activity is eating away at valuable resources. You have a hunch – what if you could streamline the GC process, making it more efficient?These two seemingly unrelated situations hold the key to unlocking a powerful new tool in GoLang: <a href="https://go.googlesource.com/proposal/+/master/design/48409-soft-memory-limit.md">GOMEMLIMIT</a>



The memory maze: Why did we need it?



While GoLang, like many languages, utilizes a garbage collector (GC) for automated memory management, pre-1.19 versions offered limited control, fostering the potential for OOM issues and inefficient GC management in high memory applications. The crux of the issue lay in:



GOGC and the Twice the Trouble: Previously, Go relied on the GOGC environment variable (defaulting to 100) to trigger GC cycles. This initiated garbage collection roughly at twice the live heap size (live heap is that part of the memory that is being used actively by your application and cannot be reclaimed by the garbage collector) as identified in the previous GC cycle, and it didn’t take into account your application’s memory hard limit



The OOM Trap: The problem arose when this doubled live heap size GC target surpassed the application’s memory hard limit, inevitably resulting in OOM issues.



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="196" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=640%2C196&#038;ssl=1" alt="" class="wp-image-13866" style="width:721px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?w=1178&amp;ssl=1 1178w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=300%2C92&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=1024%2C314&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=768%2C235&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



GC CPU wastage: For high memory applications with low live heap size, this doubled live heap size was lower compared to the memory hard limit, causing frequent GC cycle consumption, leading to high GC CPU usage.



The memory guardian: GOMEMLIMIT



<a href="https://go.googlesource.com/proposal/+/master/design/48409-soft-memory-limit.md">GOMEMLIMIT</a> is a godsend for memory-hungry Go applications like ours at Zomato. It acts as a soft memory limit for the heap, gently nudging the garbage collector to work its magic more frequently when memory usage gets close to a defined threshold (GOMEMLIMIT) which is set to be less than the application’s memory hard limit (to prevent OOM issue) and also saving CPU cycles by not running GC unnecessarily. This proactive approach prevents OOM crashes caused by inefficient garbage collection and keeps our applications running smoothly and consistently.



It&#8217;s important to remember that GOMEMLIMIT cannot solve memory leaks. A memory leak occurs when your application holds onto some memory (live heap) that the garbage collector cannot reclaim. If the live heap itself exceeds the application’s hard memory limit, even GOMEMLIMIT won&#8217;t be able to prevent an OOM issue.



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="235" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=640%2C235&#038;ssl=1" alt="" class="wp-image-13868" style="width:716px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?w=1122&amp;ssl=1 1122w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=300%2C110&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=1024%2C376&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=768%2C282&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



Introducing: Zomato/Go/Runtime Library



In order to implement GOMEMLIMIT for our applications, we took the initiative to develop our in -house Zomato/go/runtime library, simplifying the process for application developers. This library boasts the following functionalities:



Dynamic GOMEMLIMIT Calculation:



This feature eliminates the guesswork by dynamically calculating the optimal GOMEMLIMIT value during runtime. It considers the following factors:



<ul class="wp-block-list">
<li>ECS Task and Container Constraints: The library analyzes the memory limits set for both the ECS task and individual containers to arrive at a dynamic value for GOMEMLIMIT at runtime</li>



<li>Real-time Memory Usage: It retrieves up-to-date information on the current memory usage of the task at fixed intervals, this data is then used to dynamically calculate the optimal GOMEMLIMIT value with our homegrown algorithm using <a href="https://cs.opensource.google/go/go/+/go1.22.1:src/runtime/debug/garbage.go;l=236">runtime/debug.SetMemoryLimit</a> for setting the limit and the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html">ECS task Metadata Endpoint</a> for retrieving real-time container memory usage and memory limits.</li>
</ul>



GOMAXPROCS



Setting the GOMAXPROCS environment variable to an optimal value is required to avoid <a href="https://tip.golang.org/doc/gc-guide#:~:text=This%20situation%2C%20where,the%20heap%20spike.">thrashing</a> and to reduce unnecessary CPU throttling, our library achieves this by setting it based on ECS task level CPU hard limit.



Enhanced Monitoring with Runtime Metrics Integration



This library goes beyond dynamic GOMEMLIMIT calculation. It leverages the Go runtime library to export crucial metrics related to:



<ul class="wp-block-list">
<li>Garbage Collection: Track GC activity to identify potential bottlenecks or inefficiencies</li>



<li>Heap: Monitor memory allocation and utilization patterns to understand memory pressure within the application</li>



<li>Memory: Gain insights into overall memory usage and potential areas for optimization</li>



<li>CPU: Analyze CPU usage patterns and identify potential correlations with GC activity</li>



<li>ECS Task Limits: Monitor current memory usage compared to imposed ECS task limits, ensuring adherence to resource constraints</li>
</ul>



Unveiling the Impact: A deep dive into our memory management success 🚀



We integrated Zomato/go/runtime library in over 250 of our golang microservices , which yielded impressive outcomes:



Reduced GC CPU Usage by more than 95%: GOMEMLIMIT intelligently triggers garbage collection only when necessary, saving valuable CPU cycles for other critical tasks. This resulted in a remarkable reduction of overall CPU usage of our applications by up to 25%-50% (which in turn leads to equivalent reduction in our EC2 compute costs 📈). This substantial improvement empowers our applications to operate with greater efficiency handling higher loads with



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="342" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=640%2C342&#038;ssl=1" alt="" class="wp-image-13870" style="width:715px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?w=1015&amp;ssl=1 1015w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=300%2C160&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=768%2C410&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



increased ease, GC CPU usage dropped significantly, from a considerable 25%-60% (~25% on average) down to under 2% for most of our applications (a decrease of around 90% on average)



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="340" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=640%2C340&#038;ssl=1" alt="" class="wp-image-13871" style="width:720px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?w=1238&amp;ssl=1 1238w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=300%2C159&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=1024%2C543&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=768%2C408&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="377" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=640%2C377&#038;ssl=1" alt="" class="wp-image-13872" style="width:721px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?w=1225&amp;ssl=1 1225w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=300%2C177&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=1024%2C604&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=768%2C453&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



Enhanced Stability: By establishing a soft memory limit on when to run the GC, GOMEMLIMIT demonstrably reduced the risk of unexpected OOM ❌ issues and application crashes. This translates to a more reliable and stable user experience for our valued Zomato customers.



<figure class="wp-block-image size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="312" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=640%2C312&#038;ssl=1" alt="" class="wp-image-13874" style="width:628px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?w=1006&amp;ssl=1 1006w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=300%2C146&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=768%2C374&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure>



Reduced CPU throttling: By adjusting GOMAXPROCS to the optimal value using our library, we have reduced cpu throttling for our applications by up to 50%.


<div class="wp-block-image">
<figure class="aligncenter size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="232" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=640%2C232&#038;ssl=1" alt="" class="wp-image-13878" style="width:722px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?w=1238&amp;ssl=1 1238w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=300%2C109&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=1024%2C371&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=768%2C279&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure></div>


Enhanced connection management: By reducing CPU%, each application requires less number of containers to serve the same number of requests, so each downstream application has to maintain less number of connections to our application, reducing load on our service mesh.



Easy Debugging: Runtime library exports important metrics related to garbage collector, heap, memory, cpu and ecs task limits which help debug issues during incidents and also in setting up alerting on critical metrics



Cost Optimizations: By optimizing CPU usage and reducing the number of required AWS ECS tasks / EC2 instances, GOMEMLIMIT contributed to huge cost savings of around 30,000 USD per month.



The takeaway: Empowering developers



At Zomato, we&#8217;re constantly pushing the boundaries to enhance our platform&#8217;s stability and efficiency. Our latest innovation, GOMEMLIMIT, is a testament to our commitment to delivering seamless experiences through optimized performance.



GOMEMLIMIT empowers our developers with unparalleled control over memory and CPU usage. By significantly reducing out-of-memory (OOM) errors and optimizing garbage collection, this powerful feature not only improves application performance but also delivers substantial cost savings. It has proven invaluable in enhancing the scalability and reliability of our applications, ensuring that Zomato remains at the forefront of technological innovation.



As part of our ongoing commitment to innovation, we are excited to announce our plans to implement Profile-Guided Optimization (PGO) in GoLang, starting with the upcoming Go 1.20 release. PGO utilizes real-world profiling data to guide the compiler in making more informed optimization decisions, promising to further enhance runtime performance by 2-14%. This initiative underscores our dedication to leveraging cutting-edge technologies to drive continuous improvement across our platform.



Join us in shaping the future of technology



At Zomato, we believe in creating an environment where innovation thrives and where every team member contributes to our success. Our commitment to excellence is reflected in initiatives like GOMEMLIMIT and our upcoming implementation of PGO. If you are passionate about technology and seek to work with a team that values innovation and impact, consider joining us on our journey to redefine the future of technology in the food industry and beyond. Reach out to us at techrecruitment@zomato.com to explore exciting career opportunities.



All content provided in this blog is for informational and educational purposes only. It is not professional advice, and should not be treated as such. This blog was authored by <a href="https://www.linkedin.com/in/sakib-malik-836233198/">Sakib Malik</a> in collaboration with <a href="https://www.linkedin.com/in/saurabh-sabharwal-383b43108/">Saurabh Sabharwal</a> and Aniket Suri under the guidance of <a href="https://www.linkedin.com/in/rathorehimanshu/">Himanshu Rathore</a>.