Introduction

At Zomato, we develop and deliver fast, necessitating fast feedback loops. However, in a world of hundreds of microservices, testing the end-to-end flow can become time-consuming. To overcome this, we introduced Local Preview. This powerful development tool empowers developers to test their microservices directly on their local machines, eliminating the need to deploy to a shared environment. This tool is essential for accelerating the development process by providing immediate feedback on code changes and fostering better collaboration among developers working on different microservices.

Ideal Development Environment

An ideal development environment champions Local-First Development, where developers can test their entire code flow locally. It ensures:

1. Early Issue Detection – Testing locally enables developers to identify and resolve issues before deploying to shared environments, reducing the chances of integration failures and deployment delays.
2. Faster Feedback Loop – Immediate feedback on code changes speeds up development and debugging, eliminating time wasted waiting for CI/CD pipelines or remote environments.
3. Easier Code Debugging – We can leverage code debugging tools to debug code locally with ease, using breakpoints.

Problems with the Previous Workflow

At Zomato, we initiate the development process with coding and conducting initial testing on local machines. But the large number of microservices made it challenging to simulate every case locally. Several services have high dependencies on upstream services that are difficult to run locally. Moreover, local machines have inherent limitations on the number of services they can run simultaneously.

Once confident about the functionality of the code locally, the developers push changes to the remote repository, raise a pull request, and wait for reviews. Once the code is approved and merged, it will be deployed to the staging environment. The staging environment is a copy of the production environment (though not as scaled), with all services running. Only at this stage could we perform a comprehensive end-to-end (E2E) test and validate the changes on the UI.

If a bug was found during testing in the Staging Environment, the entire cycle had to be repeated:

The time taken for each cycle was significant, as each iteration involved multiple steps and dependencies. Despite its slowness, this workflow helped maintain high code quality and uptime, even while meeting aggressive product deadlines.

Solutions We Explored

`rsync` local code to remote

Our approach involved using rsync to synchronize the code from the local machine to a dedicated remote instance using our in-house command line tool ( zctl ). rsync copies the code present on the local machine to the remote machine, which is connected to the service mesh. This allows all flows to be tested on the remote machine while maintaining a faster feedback loop. We will deep dive into this in a future blog post.

Introducing Preview Environment

We launched the Preview Environment (which will be discussed in detail in a future blog post). It allows developers to deploy their code to an ephemeral environment that runs in parallel with staging, skipping the PR review process. However, even with the preview environment, the process still required us to push our code to the remote repository and wait for the application to be rebuilt and redeployed before testing could begin. This meant that while iteration time was significantly reduced, there was still substantial room for improvement.

To address this bottleneck, we developed Local Preview, which enables faster feedback by allowing developers to test changes locally without the overhead of remote builds and deployments.

Local Preview – Bringing the Power of Preview to Everyone’s Local Machine

Local Preview eliminates even more friction by enabling developers to test their microservices locally without deploying them to staging. This tool significantly speeds up the development process by providing immediate feedback on code changes. It is powered by a service deployed in our staging environment, called Local Proxy.

Local Proxy is responsible for routing calls from the staging environment to the local machine. It acts as a proxy between staging and local, using TCP Tunneling. The Local Proxy leverages components such as SSH Daemon and Envoy to manage connections and effectively route requests.

Here’s How it Works

1. The local Machine initiates a connection to the Local Proxy service to establish the reverse tunneling. Our zctl tool streamlines this process. It uses SSH to establish a secure TCP reverse tunnel between the local machine and the staging environment. This tunnel allows developers to test their services locally while maintaining a secure connection.
2. It relies on a unique HTTP header (preview-id) to distinguish requests intended for local services from those meant for standard staging services..To implement this, we are using the traffic-route policy of Kuma service mesh.
3. If a call contains the required preview-id, it is then routed to the Local Proxy.

type: TrafficRoute
mesh: dev
name: xyz-service
sources:
  - match:
      kuma.io/service: '*'
destinations:
  - match:
      kuma.io/service: xyz-service
conf:
  destination:
    kuma.io/service: xyz-service
    route_id: default
  http:
    - match:
        headers:
          preview-id:
            regex: .*xyz-service-29160.*
      destination:
        kuma.io/service: xyz-service
        route_id: xyz-service-29160

4. Local Proxy directs the call to the local machine via an SSH reverse tunnel. Envoy manages the routing of requests to the appropriate local service. It is configured to handle requests based on the preview-id, ensuring that each request reaches the correct destination.

5. Calls from the local service to the upstream microservices are routed to the Delegated Kuma-gateway via an internal VPN. The Kuma-gateway acts as an ingress gateway, routing requests based on headers. Staging datastores can also be connected from the local environment using the same internal VPN.
6. Local preview also has built-in failure recovery and monitoring systems. This is achieved by implementing two-way health checks and failure reports sent to both the developer’s machine and a central Grafana dashboard.

Conclusion: Fast Feedback, Smarter Development Cycles

Local Preview is a huge step in shifting left in our development process, making early testing faster, easier, and closer to the developer. It enables developers to test E2E flow on their local machine, drastically improving the feedback cycles. Since the local machine hosts the server, testing won’t be possible if it goes to sleep or the server is stopped. It’s best suited for individual testing, but has limitations when used in a collaborative environment. For final testing, it is ideal to deploy the changes to the preview environment, where anyone can test the code by setting the preview-id in the call.

Header Propagation is mandatory in all the services present in the call path. If preview-id header is lost, calls will not be routed to the preview service, as its functionality is dependent on the header.

This blog was authored by Arpit Mishra, in collaboration with Deepak Verma, under the guidance of Umar Ahmad .