Imagine this: you&#8217;re in a war room atmosphere. Tension hangs thick in the air as your application team and you scramble to debug a sudden spike in OOM issues plaguing your application’s containers. No memory leaks seem to be the culprit, but further investigation reveals a shocking truth: the garbage collector (GC) wasn&#8217;t running when it should have been! This crucial cleanup process failed to free up memory before the memory breached the container’s hard limit, triggering those dreaded OOM issues.



Now, let&#8217;s look at a different scenario. You&#8217;re analyzing your application’s performance profile, and a particular pattern jumps out. A significant chunk of your application’s CPU usage is being devoured by the GC. While your service doesn&#8217;t necessarily require a ton of live heap memory, the constant GC activity is eating away at valuable resources. You have a hunch – what if you could streamline the GC process, making it more efficient?These two seemingly unrelated situations hold the key to unlocking a powerful new tool in GoLang: <a href="https://go.googlesource.com/proposal/+/master/design/48409-soft-memory-limit.md">GOMEMLIMIT</a>



The memory maze: Why did we need it?



While GoLang, like many languages, utilizes a garbage collector (GC) for automated memory management, pre-1.19 versions offered limited control, fostering the potential for OOM issues and inefficient GC management in high memory applications. The crux of the issue lay in:



GOGC and the Twice the Trouble: Previously, Go relied on the GOGC environment variable (defaulting to 100) to trigger GC cycles. This initiated garbage collection roughly at twice the live heap size (live heap is that part of the memory that is being used actively by your application and cannot be reclaimed by the garbage collector) as identified in the previous GC cycle, and it didn’t take into account your application’s memory hard limit



The OOM Trap: The problem arose when this doubled live heap size GC target surpassed the application’s memory hard limit, inevitably resulting in OOM issues.



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="196" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=640%2C196&#038;ssl=1" alt="" class="wp-image-13866" style="width:721px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?w=1178&amp;ssl=1 1178w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=300%2C92&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=1024%2C314&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang.jpg?resize=768%2C235&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



GC CPU wastage: For high memory applications with low live heap size, this doubled live heap size was lower compared to the memory hard limit, causing frequent GC cycle consumption, leading to high GC CPU usage.



The memory guardian: GOMEMLIMIT



<a href="https://go.googlesource.com/proposal/+/master/design/48409-soft-memory-limit.md">GOMEMLIMIT</a> is a godsend for memory-hungry Go applications like ours at Zomato. It acts as a soft memory limit for the heap, gently nudging the garbage collector to work its magic more frequently when memory usage gets close to a defined threshold (GOMEMLIMIT) which is set to be less than the application’s memory hard limit (to prevent OOM issue) and also saving CPU cycles by not running GC unnecessarily. This proactive approach prevents OOM crashes caused by inefficient garbage collection and keeps our applications running smoothly and consistently.



It&#8217;s important to remember that GOMEMLIMIT cannot solve memory leaks. A memory leak occurs when your application holds onto some memory (live heap) that the garbage collector cannot reclaim. If the live heap itself exceeds the application’s hard memory limit, even GOMEMLIMIT won&#8217;t be able to prevent an OOM issue.



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="235" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=640%2C235&#038;ssl=1" alt="" class="wp-image-13868" style="width:716px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?w=1122&amp;ssl=1 1122w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=300%2C110&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=1024%2C376&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-1.jpg?resize=768%2C282&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



Introducing: Zomato/Go/Runtime Library



In order to implement GOMEMLIMIT for our applications, we took the initiative to develop our in -house Zomato/go/runtime library, simplifying the process for application developers. This library boasts the following functionalities:



Dynamic GOMEMLIMIT Calculation:



This feature eliminates the guesswork by dynamically calculating the optimal GOMEMLIMIT value during runtime. It considers the following factors:



<ul class="wp-block-list">
<li>ECS Task and Container Constraints: The library analyzes the memory limits set for both the ECS task and individual containers to arrive at a dynamic value for GOMEMLIMIT at runtime</li>



<li>Real-time Memory Usage: It retrieves up-to-date information on the current memory usage of the task at fixed intervals, this data is then used to dynamically calculate the optimal GOMEMLIMIT value with our homegrown algorithm using <a href="https://cs.opensource.google/go/go/+/go1.22.1:src/runtime/debug/garbage.go;l=236">runtime/debug.SetMemoryLimit</a> for setting the limit and the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html">ECS task Metadata Endpoint</a> for retrieving real-time container memory usage and memory limits.</li>
</ul>



GOMAXPROCS



Setting the GOMAXPROCS environment variable to an optimal value is required to avoid <a href="https://tip.golang.org/doc/gc-guide#:~:text=This%20situation%2C%20where,the%20heap%20spike.">thrashing</a> and to reduce unnecessary CPU throttling, our library achieves this by setting it based on ECS task level CPU hard limit.



Enhanced Monitoring with Runtime Metrics Integration



This library goes beyond dynamic GOMEMLIMIT calculation. It leverages the Go runtime library to export crucial metrics related to:



<ul class="wp-block-list">
<li>Garbage Collection: Track GC activity to identify potential bottlenecks or inefficiencies</li>



<li>Heap: Monitor memory allocation and utilization patterns to understand memory pressure within the application</li>



<li>Memory: Gain insights into overall memory usage and potential areas for optimization</li>



<li>CPU: Analyze CPU usage patterns and identify potential correlations with GC activity</li>



<li>ECS Task Limits: Monitor current memory usage compared to imposed ECS task limits, ensuring adherence to resource constraints</li>
</ul>



Unveiling the Impact: A deep dive into our memory management success 🚀



We integrated Zomato/go/runtime library in over 250 of our golang microservices , which yielded impressive outcomes:



Reduced GC CPU Usage by more than 95%: GOMEMLIMIT intelligently triggers garbage collection only when necessary, saving valuable CPU cycles for other critical tasks. This resulted in a remarkable reduction of overall CPU usage of our applications by up to 25%-50% (which in turn leads to equivalent reduction in our EC2 compute costs 📈). This substantial improvement empowers our applications to operate with greater efficiency handling higher loads with



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="342" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=640%2C342&#038;ssl=1" alt="" class="wp-image-13870" style="width:715px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?w=1015&amp;ssl=1 1015w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=300%2C160&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-2.jpg?resize=768%2C410&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



increased ease, GC CPU usage dropped significantly, from a considerable 25%-60% (~25% on average) down to under 2% for most of our applications (a decrease of around 90% on average)



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="340" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=640%2C340&#038;ssl=1" alt="" class="wp-image-13871" style="width:720px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?w=1238&amp;ssl=1 1238w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=300%2C159&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=1024%2C543&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-4.jpg?resize=768%2C408&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="377" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=640%2C377&#038;ssl=1" alt="" class="wp-image-13872" style="width:721px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?w=1225&amp;ssl=1 1225w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=300%2C177&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=1024%2C604&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-5.jpg?resize=768%2C453&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



Enhanced Stability: By establishing a soft memory limit on when to run the GC, GOMEMLIMIT demonstrably reduced the risk of unexpected OOM ❌ issues and application crashes. This translates to a more reliable and stable user experience for our valued Zomato customers.



<figure class="wp-block-image size-full is-resized"><img loading="lazy" decoding="async" width="640" height="312" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=640%2C312&#038;ssl=1" alt="" class="wp-image-13874" style="width:628px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?w=1006&amp;ssl=1 1006w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=300%2C146&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-6.jpg?resize=768%2C374&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



Reduced CPU throttling: By adjusting GOMAXPROCS to the optimal value using our library, we have reduced cpu throttling for our applications by up to 50%.


<div class="wp-block-image">
<figure class="aligncenter size-full is-resized"><img loading="lazy" decoding="async" width="640" height="232" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=640%2C232&#038;ssl=1" alt="" class="wp-image-13878" style="width:722px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?w=1238&amp;ssl=1 1238w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=300%2C109&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=1024%2C371&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/07/Building-Performant-Golang-7.jpg?resize=768%2C279&amp;ssl=1 768w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure></div>


Enhanced connection management: By reducing CPU%, each application requires less number of containers to serve the same number of requests, so each downstream application has to maintain less number of connections to our application, reducing load on our service mesh.



Easy Debugging: Runtime library exports important metrics related to garbage collector, heap, memory, cpu and ecs task limits which help debug issues during incidents and also in setting up alerting on critical metrics



Cost Optimizations: By optimizing CPU usage and reducing the number of required AWS ECS tasks / EC2 instances, GOMEMLIMIT contributed to huge cost savings of around 30,000 USD per month.



The takeaway: Empowering developers



At Zomato, we&#8217;re constantly pushing the boundaries to enhance our platform&#8217;s stability and efficiency. Our latest innovation, GOMEMLIMIT, is a testament to our commitment to delivering seamless experiences through optimized performance.



GOMEMLIMIT empowers our developers with unparalleled control over memory and CPU usage. By significantly reducing out-of-memory (OOM) errors and optimizing garbage collection, this powerful feature not only improves application performance but also delivers substantial cost savings. It has proven invaluable in enhancing the scalability and reliability of our applications, ensuring that Zomato remains at the forefront of technological innovation.



As part of our ongoing commitment to innovation, we are excited to announce our plans to implement Profile-Guided Optimization (PGO) in GoLang, starting with the upcoming Go 1.20 release. PGO utilizes real-world profiling data to guide the compiler in making more informed optimization decisions, promising to further enhance runtime performance by 2-14%. This initiative underscores our dedication to leveraging cutting-edge technologies to drive continuous improvement across our platform.



Join us in shaping the future of technology



At Zomato, we believe in creating an environment where innovation thrives and where every team member contributes to our success. Our commitment to excellence is reflected in initiatives like GOMEMLIMIT and our upcoming implementation of PGO. If you are passionate about technology and seek to work with a team that values innovation and impact, consider joining us on our journey to redefine the future of technology in the food industry and beyond. Reach out to us at techrecruitment@zomato.com to explore exciting career opportunities.



All content provided in this blog is for informational and educational purposes only. It is not professional advice, and should not be treated as such. The writer of this blog makes no representations as to the accuracy or completeness of any content or information contained here or found by following any link on this blog.

Go Beyond: Building Performant and Reliable Golang Applications

Unlocking Innovation: Zomato’s journey to seamless iOS code sharing & distribution with Swift Package Manager

Menu is one of the most important elements that defines a restaurant. Most restaurants tend to spend a disproportionate time creating a winning menu. In interactions with our restaurant partners as well, some questions we frequently hear are: &#8220;How do I make my menu more attractive? What changes can I make to my menu to improve orders from customers?” Hence, we have now compiled the best practices to help restaurant partners build an effective menu that appeals to new customers, help get better conversion, and help improve average order values.



A good menu is visually appealing, easy to navigate and communicates the right value. The menu should ideally have high quality dish images, with accurate descriptions, justifiable prices charged by the restaurant partners and be one that is not too long or too repetitive. We recognize that each restaurant is unique and so are its challenges and accordingly this tool has been designed to help each of our restaurant partners with personalized recommendations. Our goal is to empower restaurant partners with the knowledge to make informed decisions that could drive growth. Menu Score helps restaurant partners to convert their menu by drilling down into three key areas:



<ol class="wp-block-list">
<li>Menu Appeal: Visual Delight and Clarity The first impression is visual: dish photos, descriptions and item classifications. Menu score evaluates the completeness of the menu, analyzing elements such as photos, descriptions, add-ons, and side dishes. Engaging photos and clear descriptions not only captivate customers but also influence their decision-making process. </li>



<li>Menu Browsability: Navigating Success Menu score evaluates the number of items, identifies repeated items, and highlights items not being ordered and guides the restaurant partners in streamlining the menu to enhance customer satisfaction and encourage more orders.</li>



<li>Menu Pricing: Capturing &amp; Creating Value Pricing correctly is an art, and the optimal price charged by the restaurants can often determine a dish’s success. The menu score provides information about how many potential customers dropped off from the outlet after preparing a cart, which provides an insight to restaurant partners to accordingly decide their prices.</li>
</ol>


<div class="wp-block-image">
<figure class="aligncenter size-large is-resized"><img loading="lazy" decoding="async" width="640" height="398" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=640%2C398&#038;ssl=1" alt="" class="wp-image-13592" style="width:684px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=1024%2C636&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=300%2C186&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=768%2C477&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=1536%2C954&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?resize=1568%2C973&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?w=1999&amp;ssl=1 1999w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2024/04/image1-3.png?w=1920&amp;ssl=1 1920w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure></div>


The Menu Score Advantage



So, what sets Menu Score apart? It&#8217;s not just about analytics; it is about personalized, actionable insights. Here&#8217;s what you can expect:



<ul class="wp-block-list">
<li>Customized Recommendations: Receive tailored suggestions for improving menu completeness and navigability. </li>



<li>Real-time Updates: Keep the menu dynamic with real-time updates based on changing customer preferences and market trends like Navratri or New Year. Menu Score isn&#8217;t just a tool; it is a restaurant&#8217;s secret mantra for food and delivery ordering success which aims at providing the restaurant partners a higher menu score correlating with a higher menu to cart ratio. It analyzes the menu, pinpoints areas to improve, and unlocks a restaurant&#8217;s true growth potential. </li>
</ul>



So now, say goodbye to guesswork and say hello to Menu Score!&nbsp;



Let&#8217;s embark on this journey together towards culinary excellence and business prosperity.



All our restaurant partners can access their menu score using <a href="https://zom.page.link/r4xK">our merchant dashboard</a>.&nbsp;



Bon Appétit, Team Zomato