At Zomato, we develop and deliver fast, necessitating fast feedback loops. However, in a world of hundreds of microservices, testing the end-to-end flow can become time-consuming. To overcome this, we introduced Local Preview. This powerful development tool empowers developers to test their microservices directly on their local machines, eliminating the need to deploy to a shared environment. This tool is essential for accelerating the development process by providing immediate feedback on code changes and fostering better collaboration among developers working on different microservices.
An ideal development environment champions Local-First Development, where developers can test their entire code flow locally. It ensures:
At Zomato, we initiate the development process with coding and conducting initial testing on local machines. But the large number of microservices made it challenging to simulate every case locally. Several services have high dependencies on upstream services that are difficult to run locally. Moreover, local machines have inherent limitations on the number of services they can run simultaneously.
Once confident about the functionality of the code locally, the developers push changes to the remote repository, raise a pull request, and wait for reviews. Once the code is approved and merged, it will be deployed to the staging environment. The staging environment is a copy of the production environment (though not as scaled), with all services running. Only at this stage could we perform a comprehensive end-to-end (E2E) test and validate the changes on the UI.
If a bug was found during testing in the Staging Environment, the entire cycle had to be repeated:
The time taken for each cycle was significant, as each iteration involved multiple steps and dependencies. Despite its slowness, this workflow helped maintain high code quality and uptime, even while meeting aggressive product deadlines.
`rsync` local code to remote
Our approach involved using rsync to synchronize the code from the local machine to a dedicated remote instance using our in-house command line tool ( zctl ). rsync copies the code present on the local machine to the remote machine, which is connected to the service mesh. This allows all flows to be tested on the remote machine while maintaining a faster feedback loop. We will deep dive into this in a future blog post.
Introducing Preview Environment
We launched the Preview Environment (which will be discussed in detail in a future blog post). It allows developers to deploy their code to an ephemeral environment that runs in parallel with staging, skipping the PR review process. However, even with the preview environment, the process still required us to push our code to the remote repository and wait for the application to be rebuilt and redeployed before testing could begin. This meant that while iteration time was significantly reduced, there was still substantial room for improvement.
To address this bottleneck, we developed Local Preview, which enables faster feedback by allowing developers to test changes locally without the overhead of remote builds and deployments.
Local Preview eliminates even more friction by enabling developers to test their microservices locally without deploying them to staging. This tool significantly speeds up the development process by providing immediate feedback on code changes. It is powered by a service deployed in our staging environment, called Local Proxy.
Local Proxy is responsible for routing calls from the staging environment to the local machine. It acts as a proxy between staging and local, using TCP Tunneling. The Local Proxy leverages components such as SSH Daemon and Envoy to manage connections and effectively route requests.
Here’s How it Works
type: TrafficRoute
mesh: dev
name: xyz-service
sources:
- match:
kuma.io/service: '*'
destinations:
- match:
kuma.io/service: xyz-service
conf:
destination:
kuma.io/service: xyz-service
route_id: default
http:
- match:
headers:
preview-id:
regex: .*xyz-service-29160.*
destination:
kuma.io/service: xyz-service
route_id: xyz-service-29160
4. Local Proxy directs the call to the local machine via an SSH reverse tunnel. Envoy manages the routing of requests to the appropriate local service. It is configured to handle requests based on the preview-id, ensuring that each request reaches the correct destination.
5. Calls from the local service to the upstream microservices are routed to the Delegated Kuma-gateway via an internal VPN. The Kuma-gateway acts as an ingress gateway, routing requests based on headers. Staging datastores can also be connected from the local environment using the same internal VPN.
6. Local preview also has built-in failure recovery and monitoring systems. This is achieved by implementing two-way health checks and failure reports sent to both the developer’s machine and a central Grafana dashboard.
Local Preview is a huge step in shifting left in our development process, making early testing faster, easier, and closer to the developer. It enables developers to test E2E flow on their local machine, drastically improving the feedback cycles. Since the local machine hosts the server, testing won’t be possible if it goes to sleep or the server is stopped. It’s best suited for individual testing, but has limitations when used in a collaborative environment. For final testing, it is ideal to deploy the changes to the preview environment, where anyone can test the code by setting the preview-id in the call.
Header Propagation is mandatory in all the services present in the call path. If preview-id header is lost, calls will not be routed to the preview service, as its functionality is dependent on the header.
This blog was authored by Arpit Mishra, in collaboration with Deepak Verma, under the guidance of Umar Ahmad .