
<h2 class="wp-block-heading"><strong>Introduction</strong></h2>



<p>At Zomato, we develop and deliver fast, necessitating fast feedback loops. However, in a world of hundreds of microservices, testing the end-to-end flow can become time-consuming. To overcome this, we introduced Local Preview. This powerful development tool empowers developers to test their microservices directly on their local machines, eliminating the need to deploy to a shared environment. This tool is essential for accelerating the development process by providing immediate feedback on code changes and fostering better collaboration among developers working on different microservices.</p>



<h2 class="wp-block-heading"><strong>Ideal Development Environment</strong></h2>



<p>An ideal development environment champions Local-First Development, where developers can test their entire code flow locally. It ensures:</p>



<ol class="wp-block-list">
<li>Early Issue Detection &#8211; Testing locally enables developers to identify and resolve issues before deploying to shared environments, reducing the chances of integration failures and deployment delays.</li>



<li>Faster Feedback Loop &#8211; Immediate feedback on code changes speeds up development and debugging, eliminating time wasted waiting for CI/CD pipelines or remote environments.</li>



<li>Easier Code Debugging &#8211; We can leverage code debugging tools to debug code locally with ease, using breakpoints.</li>
</ol>



<h2 class="wp-block-heading"><strong>Problems with the Previous Workflow</strong></h2>



<p>At Zomato, we initiate the development process with coding and conducting initial testing on local machines. But the large number of microservices made it challenging to simulate every case locally. Several services have high dependencies on upstream services that are difficult to run locally. Moreover, local machines have inherent limitations on the number of services they can run simultaneously.</p>



<p>Once confident about the functionality of the code locally, the developers push changes to the remote repository, raise a pull request, and wait for reviews. Once the code is approved and merged, it will be deployed to the staging environment. The staging environment is a copy of the production environment (though not as scaled), with all services running. Only at this stage could we perform a comprehensive end-to-end (E2E) test and validate the changes on the UI.</p>



<p>If a bug was found during testing in the Staging Environment, the entire cycle had to be repeated:</p>


<div class="wp-block-image">
<figure class="aligncenter size-full is-resized"><img data-recalc-dims="1" loading="lazy" decoding="async" width="367" height="281" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image4.png?resize=367%2C281&#038;ssl=1" alt="" class="wp-image-14623" style="width:430px;height:auto" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image4.png?w=367&amp;ssl=1 367w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image4.png?resize=300%2C230&amp;ssl=1 300w" sizes="auto, (max-width: 367px) 100vw, 367px" /></figure></div>


<p>The time taken for each cycle was significant, as each iteration involved multiple steps and dependencies. Despite its slowness, this workflow helped maintain high code quality and uptime, even while meeting aggressive product deadlines.</p>



<h2 class="wp-block-heading"><strong>Solutions We Explored</strong></h2>



<p><strong>`rsync` local code to remote</strong></p>



<p>Our approach involved using rsync to synchronize the code from the local machine to a dedicated remote instance using our in-house command line tool ( zctl ). rsync copies the code present on the local machine to the remote machine, which is connected to the service mesh. This allows all flows to be tested on the remote machine while maintaining a faster feedback loop. We will deep dive into this in a future blog post.</p>



<p><strong>Introducing Preview Environment</strong></p>



<p>We launched the Preview Environment (which will be discussed in detail in a future blog post). It allows developers to deploy their code to an ephemeral environment that runs in parallel with staging, skipping the PR review process. However, even with the preview environment, the process still required us to push our code to the remote repository and wait for the application to be rebuilt and redeployed before testing could begin. This meant that while iteration time was significantly reduced, there was still substantial room for improvement.</p>


<div class="wp-block-image">
<figure class="aligncenter size-full"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="293" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image1.png?resize=640%2C293&#038;ssl=1" alt="" class="wp-image-14626" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image1.png?w=788&amp;ssl=1 788w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image1.png?resize=300%2C137&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image1.png?resize=768%2C352&amp;ssl=1 768w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure></div>


<p>To address this bottleneck, we developed Local Preview, which enables faster feedback by allowing developers to test changes locally without the overhead of remote builds and deployments.</p>



<h2 class="wp-block-heading"><strong>Local Preview &#8211; Bringing the Power of Preview to Everyone’s Local Machine</strong></h2>



<p>Local Preview eliminates even more friction by enabling developers to test their microservices locally without deploying them to staging. This tool significantly speeds up the development process by providing immediate feedback on code changes. It is powered by a service deployed in our staging environment, called Local Proxy.</p>



<p>Local Proxy is responsible for routing calls from the staging environment to the local machine. It acts as a proxy between staging and local, using TCP Tunneling. The Local Proxy leverages components such as SSH Daemon and Envoy to manage connections and effectively route requests.</p>



<p><strong>Here’s How it Works</strong></p>



<ol class="wp-block-list">
<li>The local Machine initiates a connection to the Local Proxy service to establish the reverse tunneling. Our zctl tool streamlines this process. It uses SSH to establish a secure TCP reverse tunnel between the local machine and the staging environment. This tunnel allows developers to test their services locally while maintaining a secure connection.</li>



<li>It relies on a unique HTTP header (<em>preview-id</em>) to distinguish requests intended for local services from those meant for standard staging services..To implement this, we are using the <a href="https://kuma.io/docs/2.10.x/policies/traffic-route/" data-type="link" data-id="https://kuma.io/docs/2.10.x/policies/traffic-route/">traffic-route</a> policy of Kuma service mesh.</li>



<li>If a call contains the required <em>preview-id</em>, it is then routed to the Local Proxy.</li>
</ol>



<pre class="wp-block-code"><code>type: TrafficRoute
mesh: dev
name: xyz-service
sources:
  - match:
      kuma.io/service: '*'
destinations:
  - match:
      kuma.io/service: xyz-service
conf:
  destination:
    kuma.io/service: xyz-service
    route_id: default
  http:
    - match:
        headers:
          preview-id:
            regex: .*xyz-service-29160.*
      destination:
        kuma.io/service: xyz-service
        route_id: xyz-service-29160
</code></pre>



<p>4. Local Proxy directs the call to the local machine via an SSH reverse tunnel. Envoy manages the routing of requests to the appropriate local service. It is configured to handle requests based on the <em>preview-id</em>, ensuring that each request reaches the correct destination.</p>


<div class="wp-block-image">
<figure class="aligncenter size-full"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="395" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image2.png?resize=640%2C395&#038;ssl=1" alt="" class="wp-image-14630" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image2.png?w=671&amp;ssl=1 671w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/06/image2.png?resize=300%2C185&amp;ssl=1 300w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure></div>


<p>5. Calls from the local service to the upstream microservices are routed to the Delegated Kuma-gateway via an internal VPN. The Kuma-gateway acts as an ingress gateway, routing requests based on headers. Staging datastores can also be connected from the local environment using the same internal VPN.<br>6. Local preview also has built-in failure recovery and monitoring systems. This is achieved by implementing two-way health checks and failure reports sent to both the developer’s machine and a central Grafana dashboard.</p>



<h2 class="wp-block-heading"><strong>Conclusion: Fast Feedback, Smarter Development Cycles</strong></h2>



<p>Local Preview is a huge step in shifting left in our development process, making early testing faster, easier, and closer to the developer. It enables developers to test E2E flow on their local machine, drastically improving the feedback cycles. Since the local machine hosts the server, testing won’t be possible if it goes to sleep or the server is stopped. It&#8217;s best suited for individual testing, but has limitations when used in a collaborative environment. For final testing, it is ideal to deploy the changes to the preview environment, where anyone can test the code by setting the <em>preview-id</em> in the call.</p>



<p>Header Propagation is mandatory in all the services present in the call path. If <em>preview-id</em> header is lost, calls will not be routed to the preview service, as its functionality is dependent on the header.</p>



<p>This blog was authored by <a href="https://in.linkedin.com/in/arpit-mishra-17b733192" target="_blank" rel="noreferrer noopener">Arpit Mishra</a>, in collaboration with <a href="https://in.linkedin.com/in/deepak-verma-2966a8115" target="_blank" rel="noreferrer noopener">Deepak Verma</a>, under the guidance of <a href="https://in.linkedin.com/in/ahmadumar" target="_blank" rel="noreferrer noopener">Umar Ahmad</a> .</p>


Embracing Shift-Left Testing with Local Preview

Eliminating Bottlenecks in Real-Time Data Streaming: A Zomato Ads Flink Journey


<h2 class="wp-block-heading"><strong>Introduction</strong></h2>



<p>At Zomato, we love food and we love data. And sometimes, those two passions collide in the creation of beautiful, portable, and secure PDFs. From invoices and receipts to detailed reports, PDFs are a cornerstone of our operations and a staple across the industry. However, our existing PDF generation process, powered by a PHP-based library, was starting to show its age. We encountered latency issues, especially during the peak hours and the situation worsened with complex, multi-page PDFs that included significant graphical content.</p>



<p>We needed a solution that could sustain our growing demands &#8211; something faster, more efficient, and scalable.</p>



<h2 class="wp-block-heading"><strong>The Perfect Blend: Chromium and Golang</strong></h2>



<p>In pursuit of a more tailored and efficient approach, we decided to develop our in-house PDF generation solution. We chose Go as our programming language, given its performance and our team&#8217;s familiarity with it.</p>



<p>For rendering, we leveraged <a href="https://github.com/go-rod/rod">Rod</a>, a go library for controlling headless Chromium. For PDF signing, we integrated Go PDF (<a href="https://github.com/digitorus/pdf">digitorus/pdf</a>), allowing us to generate and sign PDFs directly from HTML templates. This approach gave us full control over layout, styling, and security.</p>



<h2 class="wp-block-heading"><strong>Custom Certificate Signing</strong></h2>



<p>To ensure the authenticity and integrity of our PDFs, we implemented digital signing using a custom certificate. This adds a layer of security and trust to our documents, which is especially important for sensitive information like financial transactions.</p>



<h2 class="wp-block-heading"><strong>Optimizations: Squeezing Every Last Drop of Performance</strong></h2>



<p>We&#8217;re always looking for ways to improve performance, and Espresso is no exception. We implemented several optimizations to make our PDF generation process as fast and efficient as possible:</p>



<ul class="wp-block-list">
<li><strong>Pre-Paint PDF Generation:</strong> We generate the PDF before the paint step of the rendering process, saving valuable time.</li>



<li><strong>Chromium Flags:</strong> We fine-tuned various Chromium flags to further improve the start time of the application.</li>



<li><strong>DataURI Image Prefetching:</strong> We prefetch HTML images from the template and save them as dataURIs, reducing the rendering cost by eliminating the need for the browser to download images on page load.</li>



<li><strong>Streamlined Processing:</strong> By combining PDF generation and signing into a single application, we can work with streams instead of files, improving performance and reducing disk I/O.</li>
</ul>



<h2 class="wp-block-heading"><strong>Brewing PDFs for Everyone</strong></h2>



<p>We developed a small front-end application to accompany this system, allowing anyone at Zomato to create and customize PDF templates effortlessly. This tool empowers users to modify existing templates or design new ones from scratch, all while receiving real-time feedback on the appearance of their final PDF output.</p>


<div class="wp-block-image">
<figure class="aligncenter size-large"><img data-recalc-dims="1" loading="lazy" decoding="async" width="640" height="319" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=640%2C319&#038;ssl=1" alt="" class="wp-image-14539" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=1024%2C511&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=300%2C150&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=768%2C383&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=1536%2C767&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=2048%2C1022&amp;ssl=1 2048w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=1200%2C599&amp;ssl=1 1200w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?resize=1568%2C783&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2025/03/Untitled-design-1.png?w=1920&amp;ssl=1 1920w" sizes="auto, (max-width: 640px) 100vw, 640px" /></figure></div>


<h2 class="wp-block-heading"><strong>Open Source: Sharing the Espresso Love</strong></h2>



<p>Driven by our commitment to open source, we’re excited to announce that Espresso is now available for everyone. You can deploy it in your environment, tailor it to your specific requirements, or even actively contribute to its ongoing development. This includes the lib package, which you can utilise as a library within your Go applications to build and customize your PDF generation solutions.</p>



<p>We hope that Espresso will serve as a powerful asset for organizations and developers seeking a rapid, efficient, and dependable PDF generation solution.</p>



<p>You can find the repository on github: <a href="https://github.com/zomato/espresso">https://github.com/zomato/espresso</a>. We&#8217;re offering Espresso and the packages with a very permissive MIT license, allowing developers to utilise it as per their requirements/needs while retaining full ownership of the data.</p>



<h2 class="wp-block-heading"><strong>Benchmarks: The Proof is in the Pudding</strong></h2>



<p>Espresso isn&#8217;t just fast &#8211; it&#8217;s incredibly fast. Espresso has been successfully deployed across the majority of our PDF generation workflows, delivering consistently impressive performance metrics.</p>



<ul class="wp-block-list">
<li><strong>Rendering and Signing Times Under 200ms:</strong> Espresso can render and sign PDFs in under 200 milliseconds, making it ideal for high-traffic applications.</li>



<li><strong>Peak Load Handling:</strong> Espresso can handle peak loads of 120K requests per minute (RPM), ensuring your users never have to wait for their PDFs.</li>



<li><strong>Massive Signing Capability:</strong> We recently signed 1.6 million PDFs in just 19 minutes &#8211; approximately 1,400 PDFs per second.</li>
</ul>



<p>These numbers were witnessed when running espresso as an AWS ECS service with 4vcpu and 4G memory. This scaled to a hundred containers at peak load.</p>



<p>Espresso&#8217;s efficiency is truly remarkable. Compared to our prior system, it demands significantly less infrastructure, drastically reducing resource consumption. This translates directly into substantial cost savings, resulting in a 90% deduction in server costs for generating the same volume of PDFs.</p>



<h2 class="wp-block-heading"><strong>Conclusion: The Perfect Cup of PDF</strong></h2>



<p>Espresso is the culmination of our quest for the perfect PDF generation solution. It delivers simplicity, speed, efficiency and unwavering reliability, all within an open-source framework. We&#8217;re proud of what we&#8217;ve built and are excited to share it with the world. So go ahead, grab a cup of your favorite beverage, and give Espresso a try. We think you&#8217;ll find it&#8217;s the perfect blend of speed, simplicity, and power.</p>



<p><strong>And remember, much like a perfectly brewed coffee, the most effective solutions are often discovered in the simplest of places.</strong></p>
