Welcome to the third and final instalment of our three-part blog series! In parts<a href="https://www.zomato.com/blog/explained-how-we-handle-100million-daily-search-queries-pt1"> one</a> and <a href="https://www.zomato.com/blog/explained-how-we-handle-100million-daily-search-queries-pt2">two</a>, we unveiled the secrets behind Zomato&#8217;s powerful search engine, driven by Apache Solr, and shared our tactics for optimizing Solr’s performance. In this concluding chapter, we&#8217;ll take you behind the scenes of Zomato&#8217;s self-managed Solr clusters to show how we tackled the challenges arising from this rapid growth and transformed the scalability, reliability, and performance of Solr.



<h3 class="wp-block-heading">Hitting the infrastructural bottlenecks</h3>



Initially, we were running our Solr Clusters in Master-Slave Configuration. As Zomato&#8217;s growth continued at a breakneck speed, the index size started growing, and we started facing a new set of challenges:



<ul>
<li>To improve performance, we had to run the slave nodes on extremely large machines (120GB RAM) with a large JVM Heap (~100GB), but this resulted in poor tail latencies due to the longer&nbsp; <a href="https://www.oracle.com/technetwork/tutorials/tutorials-1873457.html">Stop The World</a> step in Garbage Collection.</li>



<li>Additionally, on every sync with the master, slaves had to open <a href="https://solr.apache.org/guide/8_7/index-replication.html#follower-replication">Searcher</a> on a large index, and <a href="https://solr.apache.org/guide/8_7/query-settings-in-solrconfig.html#caches">warming up</a> the caches took more time. This led to even worse tail latencies.</li>



<li>Furthermore, our business experiences daily traffic peaks during the typical lunch and dinner hours, necessitating rapid autoscaling. But it took ~10 minutes for a new replica to be ready for querying.</li>



<li>Lastly, manual handling was required for master failovers, which resulted in low data indexing reliability and fault tolerance.</li>
</ul>



<h3 class="wp-block-heading">Solving for scale: Moving to SolrCloud architecture</h3>



The logical solution to this problem was to divide the index into smaller segments and perform queries across multiple segments to address application requests. This approach led us to SolrCloud architecture, which allows managing large indexes in a more efficient and scalable manner, while still being self-managed. We observed several improvements:



<ul>
<li>We were able to run the cluster on smaller machines resulting in ~75% reduction in individual instances&#8217; CPU and Memory requirements.</li>



<li>We observed significant improvement in average and tail latencies.</li>



<li>Smaller index per shard also improved scale-up time by ~20%.</li>
</ul>



<h3 class="wp-block-heading">Improving the SolrCloud Performance</h3>



<h4 class="wp-block-heading">Custom sharding logic&nbsp;</h4>



To make sure that the migration to Solr Cloud is more performant, we developed a custom data-sharding strategy that aligns with Zomato’s hyperlocal business model, ensuring that data from a specific locality is stored within the same shard. As a result of this strategy, ~95% of queries were processed using a single shard, significantly reducing processing time and improving efficiency. The remaining queries were handled by multiple shards, providing an effective solution for handling a small subset of queries.&nbsp;



<h4 class="wp-block-heading">Turning off distributed search</h4>



At the time of splitting data into the shards, we ensured that we always knew which locality’s data was present in which shard. This made it possible to turn off the <a href="https://solr.apache.org/guide/8_4/distributed-requests.html">distributed search</a> by directing requests to the relevant shard and ensuring that the replica receiving the request would return the response by querying its local index only.



<pre class="wp-block-prismatic-blocks"><code class="language-">shards=shard1
shard.preference=local</code></pre>



By combining our understanding of customer behaviour with Solr&#8217;s internal request routing mechanisms, we successfully decreased the throughput on individual nodes by ~75%. Consequently, this reduction led to lower latency, decreased network traffic, and minimize costs, as the same traffic was served by a smaller number of nodes.



<h3 class="wp-block-heading">Solving for resiliency and availability</h3>



Following our successful migration from Solr Master-Slave to SolrCloud architecture, operations appeared to be running seamlessly. However, one day we started receiving alerts highlighting the absence of healthy hosts in one of the shards. This single-shard malfunction led to considerable downtime during peak business hours. At our operational scale, such downtimes result in substantial business losses and affect our customers, restaurant partners, and delivery partners alike.&nbsp;



<h4 class="wp-block-heading">The Downtime</h4>



Sharing a few details about our self-managed SolrCloud setup and the pre-conditions which triggered the downtime:



<ul>
<li>The Solr clusters were hosted on <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot instances</a> which means that running nodes can go down anytime.</li>



<li>We had configured to use only NRT (Near Real Time) replicas out of the <a href="https://solr.apache.org/guide/8_7/shards-and-indexing-data-in-solrcloud.html#types-of-replicas">three available options</a> i.e. NRT, TLOG and PULL replicas.</li>



<li>Whenever a new replica joins the cluster, a unique, monotonically increasing number (znode) is assigned by ZooKeeper. During the Leader Election process, the replica with the smallest value is given a chance to become the leader.</li>



<li>After a replica is allowed to become a leader, it verifies its index version proximity with the most recent version compared to the other replicas in the cluster. If its version is significantly outdated, it forfeits the chance of becoming the leader and goes into recovery, giving a chance to the next replica in the sequence.&nbsp;</li>



<li>After the leader is elected, any replica nodes significantly behind the new leader enter recovery mode to sync their index with the newly elected leader.</li>



<li>If any replica goes into recovery, it does not serve even the read traffic.</li>
</ul>



Now, what actually happened at the time of the incident:



<ul>
<li>The leader instance got reclaimed, and hence the cluster had no active leader.</li>



<li>A few minutes before the actual leader went down, a new replica joined the shard. Therefore, it had the most recent copy of the index.</li>



<li>Under these special circumstances, all the nodes in that shard went into recovery as their indexes were relatively older compared to the new replica and finally, the new replica was invited to become the leader.</li>



<li>The entire read traffic got redirected to the new leader. At the same time, all other replicas were trying to download GBs of data from the leader node. This led to a vicious cycle where the shard was not able to become healthy due to excess load on one node.</li>



<li>Finally, we had to cut all the traffic (single shard and cross-shard requests both) to allow it some time to become healthy.</li>
</ul>



<figure class="wp-block-image size-large is-resized"><img decoding="async" loading="lazy" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=702%2C304&#038;ssl=1" alt="Figure 1: Every engineer’s worst nightmare: No healthy hosts" class="wp-image-12872" width="702" height="304" title="" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=1024%2C443&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=300%2C130&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=768%2C333&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?w=1418&amp;ssl=1 1418w" sizes="(max-width: 702px) 100vw, 702px" data-recalc-dims="1" /></figure>



<h4 class="wp-block-heading">Switch to TLOG and Pull Replicas</h4>



At Zomato, we believe that “<a href="https://www.instagram.com/p/Cl08IrgvGTt/">Only the Paranoid Survive</a>”. After finishing the RCA of the incident, we decided that we need to solve a few things:



<ul>
<li>The Solr cluster must be highly available for processing the read queries by serving the read traffic during the Leader election or without any leader at all.</li>



<li>We should be able to quickly scale up the cluster without putting too much stress on the leader.</li>
</ul>



We decided to deep dive into Solr’s replication and leader election process. Each replica type has a different way of processing the updates from the leader, which decides the series of events during the Leader Election:



<ul>
<li>NRT:
<ol>
<li>Each replica receives the update from the leader and writes data to the transaction log (tlog).</li>



<li>Updates are periodically flushed from the tlog to the index and stored in the disc. This also means that the index files can be completely different (highlighted in Figure 2 as Idx1 and Idx2) in each replica.</li>



<li>NRT replica is eligible to become a leader during the leader election process.</li>



<li>In order to avoid any data loss, these replicas can go into recovery during the leader election.</li>
</ol>
</li>



<li>TLOG:
<ol>
<li>Each replica receives the update from the leader and writes data to the transaction log but does not write changes to the local index.</li>



<li>The index is periodically updated by fetching only the incremental changes from the leader. This also means that the index files will be the same as that of the leader.</li>



<li>TLOG replicas are also eligible to become a leader and just like NRT replicas, they can go into recovery during leader election to ensure data availability.</li>
</ol>
</li>



<li>PULL:
<ol>
<li>PULL replicas do not receive any updates directly from the leader and hence do not maintain a transaction log.</li>



<li>Similar to TLOG replicas, the index is periodically updated by fetching only the incremental changes from the leader.</li>



<li>PULL replicas are not eligible to become leader during the leader election.</li>
</ol>
</li>
</ul>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" width="471" height="521" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?resize=471%2C521&#038;ssl=1" alt="" class="wp-image-12874" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?w=471&amp;ssl=1 471w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?resize=271%2C300&amp;ssl=1 271w" sizes="(max-width: 471px) 100vw, 471px" data-recalc-dims="1" /></figure>



Keeping all of this in mind, we began trying out multiple combinations of replicas and recreated multiple scenarios and we observed that:



<ul>
<li>NRT and PULL replicas [Not Recommended]
<ol>
<li>During leader election, only the NRT replicas go into recovery. The cluster is able to serve read requests by PULL replicas at all times (even without a leader).</li>



<li>No benefit of Near Real Time searching even after having NRT replicas as PULL replicas reflect changes only after the NRT leader commits changes to the index.&nbsp;</li>



<li>Since each replica has its own copy of index files, in case of Leader Election,&nbsp; the follower replicas need to download the entire index from the new leader and discard their old index files. This leads to a lot of disk IO and network bandwidth wastage and puts the leader under stress.</li>
</ol>
</li>



<li>TLOG replicas
<ol>
<li>As mentioned earlier, the cluster goes into recovery for a short period of time during the Leader election process. Hence, the cluster will not be available for reads for that period.</li>
</ol>
</li>



<li>NRT and TLOG replicas [Not Recommended]
<ol>
<li>Behave the same way as NRT and PULL replicas combination hence the same limitations and advantages.</li>
</ol>
</li>



<li>TLOG and PULL replicas
<ol>
<li>During the leader election, only the TLOG replicas go into recovery and need to download only the incremental changes from the new leader.</li>



<li>Similar to the NRT and PULL combination, the cluster is able to serve read requests by PULL replicas even without a leader or during the leader election process.</li>



<li>The downside is that the cluster loses the capability of Near Real-Time Search.</li>
</ol>
</li>
</ul>



Finally, we went ahead with the combination of TLOG and PULL replicas as we can bear some delay in reflecting the updates but we need the Solr cluster to be highly available for querying.



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" width="736" height="766" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?resize=736%2C766&#038;ssl=1" alt="" class="wp-image-12875" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?w=736&amp;ssl=1 736w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?resize=288%2C300&amp;ssl=1 288w" sizes="(max-width: 736px) 100vw, 736px" data-recalc-dims="1" /></figure>



Extra Miles &#8211; seeding index data for faster scaling



Whenever a new node comes up, it has to download the entire index from the leader. This process has a serious limitation as it leads to bandwidth issues in case a lot of replicas need to be created at the same time.&nbsp;



We tried to look for solutions and sought help from the Solr community, but we eventually decided to build a custom solution:



<ol>
<li>Periodically take a snapshot of the index and save it in the AWS S3.</li>



<li>Whenever a new node comes up, it first downloads the latest snapshot from S3.</li>



<li>Configure the replica’s <a href="https://solr.apache.org/guide/8_7/datadir-and-directoryfactory-in-solrconfig.html">dataDir</a> to the path where the index is downloaded. This makes it possible to seed the index data for any new replica.</li>



<li>After this, the replica downloads only the incremental changes from the leader, allowing us to scale 100s of replicas at the same time.</li>
</ol>



This solved one more problem for us. Currently, Solr offers <a href="https://solr.apache.org/guide/8_7/making-and-restoring-backups.html">backup and restore</a> functionality for each collection, and it creates a single backup for the index data across all shards. Our solution enabled us to restore a specific shard by simply creating a TLOG replica using the seed data stored in S3. This effectively functions as a point-in-time recovery mechanism for individual shards, adding an extra layer of flexibility and resilience to our SolrCloud infrastructure.



<h3 class="wp-block-heading">In Conclusion</h3>



The spirit of being always paranoid, clubbed with a deeper understanding of our business and the underlying technology, continuously pushes us to build scalable, resilient and highly available solutions for customers. Our journey with Apache Solr has been an interesting one and full of learnings and we look forward to regularly sharing this with the community.



This blog was written by <a href="https://www.linkedin.com/in/nkmishra1997/">Nikhil Mishra</a> in collaboration with <a href="https://www.linkedin.com/in/saurav-singh-25317625/">Saurav Singh</a>.



<hr class="wp-block-separator has-css-opacity"/>



Sources:



<ul>
<li><a href="https://www.oracle.com/technetwork/tutorials/tutorials-1873457.html">Java Garbage Collection</a>, <a href="http://www.oracle.com" rel="nofollow">http://www.oracle.com</a></li>



<li><a href="https://solr.apache.org/guide/8_7/index-replication.html#follower-replication">Index Replication</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/query-settings-in-solrconfig.html#caches">Query Settings in SolrConfig</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_4/distributed-requests.html">Distributed Requests</a>, solr.apache.org</li>



<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot instances</a>, docs.aws.amazon.com</li>



<li><a href="https://solr.apache.org/guide/8_7/shards-and-indexing-data-in-solrcloud.html#types-of-replicas">Shards and Indexing Data in SolrCloud</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/docs/7_4_0/solr-core/org/apache/solr/cloud/LeaderElector.html#:~:text=Leader%20Election%20process,sequential%20node%2C%20etc.">Leader Election</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/datadir-and-directoryfactory-in-solrconfig.html">DataDir and DirectoryFactory in SolrConfig</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/making-and-restoring-backups.html">Making and Restoring Backups</a>, solr.apache.org</li>
</ul>

Explained: How Zomato Handles 100 Million Daily Search Queries! (Part Three)

In <a href="https://blog.zomato.com/explained-how-zomato-handles-100-million-daily-search-queries-p1">our previous blog post</a>, we talked about the challenges we faced due to the suboptimal design of our schema, which heavily relied on dynamic fields for fulfilling our requirements. This inturn, caused poor performance and high costs during peak times as we scaled up. While we implemented some quick fixes to address these issues, such as upgrading Solr version and utilizing doc_values for aggregate and sort fields, our dependency on dynamic fields remained a bottleneck when expanding the schema for new use cases.



Continuing our journey towards a more robust and scalable data indexing system, in this blog post, we will dive deeper into the changes we made to our schema and query paradigm. These changes not only helped us achieve greater resiliency, scalability, and cost-effectiveness but also enhanced our overall performance, ensuring a seamless user experience. So, let&#8217;s explore how we tackled these challenges and improved our search system.



<h3 class="wp-block-heading">Dynamic Fields &#8211; The Proposition</h3>



Dynamic fields offer a practical function by enabling the addition of various fields in a document, tailored to meet specific use cases. With this feature, we can create fields customized to the unique requirements of our data without having to modify the schema. For instance, we may need to store popularity values for different delivery areas of a Restaurant. To achieve this, we could use a dynamic field such as popularity_* to accomplish the task. During indexing, we can enrich the restaurant document with popularity values for its various delivery locations.



An example of a Restaurant with its popularity values at different delivery locations:



<pre class="wp-block-prismatic-blocks"><code class="language-javascript">{
 'id': 'res1',
 'entity': 'restaurant',
 'name': 'Dynamic Pizza Store',
 'location': '89.998900, 28.8389300',
 // popularity score for different delivery areas
 'popularity_area1': 0.93,
 'popularity_area2': 0.25,
 'popularity_areaN': 0.71
}</code></pre>



In the case of Zomato, where a single restaurant can deliver at multiple locations, the number of unique popularity_{area_id} fields can quickly accumulate, especially when considering all the restaurants in India. This scenario is just one of the many examples that can cause a substantial increase in the total number of unique fields. Furthermore, the system comprises of various document types, each with its own set of dynamic fields, which can exacerbate the issue, resulting in a field mapping explosion. This situation can lead to out-of-memory errors and make recovery difficult. Moreover, the high field count can also adversely affect full indexing speed. Using dynamic fields, even if it’s one of the most conventional design patterns, for unbounded fields, is not a scalable or sustainable solution in the long run.



<blockquote class="wp-block-quote">
An experiment to revisit the Restaurant popularity score which needs to be A/B tested in production will introduce another dynamic field like popularity_v2_* or extend the existing field, in either approach such experiments will increase the unique field counts by hundreds of thousands.
</blockquote>



<h3 class="wp-block-heading">Usecase &amp; Alternative Model</h3>



Filtering, sorting, or aggregating documents, such as restaurants or dishes, based on various fields like popularity, location, rating, and cost, is a practical use case. While these fields are mostly global, they may not always be sufficient for every use case. Zomato, being a hyperlocal heavy business, requires more granular values for entities to be relevant to specific use cases. For example, a dish like Chole Bhature may be more popular in Delhi than in Kolkata. Moreover, the popularity or demand for a dish may vary significantly within a city. For instance Sushi may be more popular in the Golf Course area of Gurugram as against the lanes of Old Gurugram. In such cases, the global properties of an entity might not be enough, and more granular values are needed to find the most relevant results. Hence having granular and localized information available during indexing is crucial for enhancing search relevance and speed. For instance, if a restaurant has popularity scores for various delivery locations, incorporating that information at index time would be beneficial. Although Dynamic Fields appear to be the logical choice to index such information alongside other properties, the unbounded or high cardinality of such properties can lead to a mapping explosion.&nbsp;



An alternative model for the above example could be as follows:



<pre class="wp-block-prismatic-blocks"><code class="language-javascript">{
 'id': 'res1',
 'entity': 'restaurant',
 'name': 'Dynamic Pizza Store',
 'location': '89.998900, 28.8389300',
 // popularity score as map
 'popularity': {
 'area1': 0.93,
 'area2': 0.25,
 'areaN': 0.71
 }
}</code></pre>



The above structure assumes that popularity can be indexed as a Map data type and that the map contains the popularity value of different areas. This approach seems more concise and intuitive and avoids dynamic field popularity_*. However, Solr does not support such a field type out of the box. Although we could store a custom field type or serialized map data, it would not efficiently filter, sort, or aggregate data on such granular properties. We cannot use the data indexed in the above format to filter restaurants serving in areaX and order them by their popularity in that area. This would have been possible, if Solr magically supported the query pattern to filter on popularity.areaX != 0 and sort by popularity.areaX, unfortunately that&#8217;s not the case.



<h3 class="wp-block-heading">Nested Documents Rationale</h3>



Even though Solr cannot index and query a field as a Map, it does support nested documents and facilitates efficient indexing and <a href="https://solr.apache.org/guide/8_7/searching-nested-documents.html#searching-nested-documents">retrieval</a> of them, according to the Solr <a href="https://solr.apache.org/guide/8_7/indexing-nested-documents.html">documentation</a>:



<blockquote class="wp-block-quote">
Nested documents in Solr can be used to bind a blog post (parent document) with comments (child documents) — or as a way to model major product lines as parent documents, with multiple types of child documents representing individual SKUs (with unique sizes/colours) and supporting documentation (either directly nested under the products or under individual SKUs.
</blockquote>



In the nested document model, the above example can be represented as follows:



<pre class="wp-block-prismatic-blocks"><code class="language-javascript">{
 'id': 'res1',
 'entity': 'restaurant',
 'name': 'Dynamic Pizza Store',
 'location': '89.998900, 28.8389300',
 // popularity score as the list of nested objects
 'popularity': [
 {
 'id': 'res1_area1',
 'entity': 'popularity',
 'value': 0.93
 },
 {
 'id': 'res1_area2',
 'entity': 'popularity',
 'value': 0.25
 },
 {
 'id': 'res1_areaN',
 'entity': 'popularity',
 'value': 0.71
 }
 ]
}</code></pre>



Further research revealed that the Nested document model is NOT a construct at the Lucene level. Solr has gone beyond the simple document structure that Lucene recognizes by abstracting this concept. Solr uses the following fields to maintain relationships between the document entities:



<ul>
<li>_root_ : automatically populated in all the docs (root, parent, child) of a hierarchy, contains the value of root document ID</li>



<li>_nest_path_ : automatically populated in non-root docs, contains the path of the document in the hierarchy</li>



<li>_nest_parent_ : automatically populated for the docs with its immediate parent ID if exists</li>
</ul>



In order to ensure that the fields mentioned above are populated, the schema must have them defined in a specific manner, and it is not optional:



<pre class="wp-block-prismatic-blocks"><code class="language-">&lt;field name="_root_" stored="false" type="string" indexed="true"/>
&lt;fieldType name="_nest_path_" class="solr.NestPathField" />
&lt;field name="_nest_path_" type="_nest_path_"/>
&lt;field name="_nest_parent_" stored="true" indexed="true" type="string"/></code></pre>



By including the fields mentioned above in the documents, Solr can determine the relationships between the documents, which allows for the hierarchical indexing, updating, or deleting of a group of documents at the same time. For instance, it can index an entire Restaurant document with all of its Dishes and area Popularity documents together. Indexing the relationships between documents typically leads to faster queries than performing an equivalent <a href="https://solr.apache.org/guide/8_7/other-parsers.html#join-query-parser">join</a> at query time because the relationships are already stored in the index, which eliminates the need to calculate them. As a result, Solr can respond to queries more quickly. Since the concept of Nested documents is an abstraction in Solr, all child documents are ultimately standalone documents. This has some nuances for the schema definition.



<ul>
<li>A field can be configured in one way only, no matter what sort of documents are using it</li>



<li>It may be infeasible to use required restrictions on the fields, since&nbsp; not all documents will have all the fields</li>
</ul>



<h3 class="wp-block-heading">What all changed?</h3>



The implementation of a nested structure has resulted in a single document (like Restaurant) being divided into multiple documents (like Restaurants and its Popularity and Dishes), forming a hierarchical relationship that allows for the entire hierarchy to be accessed at once. Multiple small documents are used to replace dynamic fields, resolving the issue of mapping explosion. While this approach leads to a significant increase in the number of individual Lucene-level documents, we have not encountered any problems dealing with this tradeoff. Although there is a slight increase in index size, the indexing speed has improved significantly due to the absence of the mapping explosion. To efficiently query this new structure, we migrated our queries to the Block Join Query Parser (<a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-query-parsers">BJQ</a>), which can effectively search the nested structure using relationships.



<h3 class="wp-block-heading">Block Join Query Paradigm</h3>



To ensure efficient searching within the nested document construct, queries must incorporate the relationship between the parent and child documents. By utilizing these relationships, various access patterns can be established, of which there are two primary ones.



<ul>
<li>Child Query &#8211; A <a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-children-query-parser">query</a> that matches some parent documents and returns the children of those documents</li>
</ul>



<pre class="wp-block-prismatic-blocks"><code class="language-">q={!child of=&lt;blockMask>}&lt;someParents></code></pre>



<ul>
<li>Parent Query &#8211; A<a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-parent-query-parser"> query</a> that matches child documents and returns their parents</li>
</ul>



<pre class="wp-block-prismatic-blocks"><code class="language-">q={!parent which=&lt;blockMask>}&lt;someChildren></code></pre>



<ul>
<li>blockMask here is to identify the set of document which should be treated as parents</li>
</ul>



To summarize, Block Join Queries (BJQ) can be used to query nested documents by filtering either the parent documents based on the attributes of their children or the child documents based on the attributes of their parents. In the specific use case of a Restaurant with Dishes and Popularity as children, BJQ can be used to find Restaurants (parents) that serve a certain Dish (child) using the Parent Query Parser. It can also be used to find all the Dishes being served by Restaurants in a particular area by filtering Restaurants with a popularity score in that area. The possibilities for using BJQ are immense and can be leveraged for a variety of use cases. More information can be found in the official <a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-query-parsers">Solr BJQ</a> documentation.



<h3 class="wp-block-heading">End Note</h3>



We faced limitations in adding more dynamic fields, which led to delays in implementing some business use cases. To overcome these limitations, we changed the schema paradigm and migrated most of our use cases to the Nested Document model. We also changed our queries to efficiently query the new structure using BJQ. While the Nested Document model does increase the index size slightly, we experienced faster indexing speeds due to the reduced unique field counts. Overall, these changes were successful in addressing our challenges with dynamic fields.



<h3 class="wp-block-heading">Way Ahead</h3>



The migration to a nested structure effectively resolved the immediate issue and ensured future compatibility with various use cases. However, as the number of restaurants on the platform continues to grow and more delivery locations are added, there is an increasing number of child documents associated with each parent. While this is not a problem from a query perspective, as the relationship between parents and children is established during index time, the size of the index has grown over time. As a result, maintaining the entire index on a single machine in a Master-Slave setup is no longer a viable option.&nbsp;



In the next blog post, the last in this series, we will delve into our decision to transition to a Solr Cloud setup and the performance and production issues associated with it.



Keep coming back for more!



This blog was written by <a href="https://www.linkedin.com/in/saurav-singh-25317625/">Saurav Singh</a>.



<hr class="wp-block-separator has-css-opacity"/>



Sources:



<ul>
<li><a href="https://solr.apache.org/guide/8_7/searching-nested-documents.html#searching-nested-documents">Searching Nested Documents</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/indexing-nested-documents.html">Nested Document indexing</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/other-parsers.html#join-query-parser">Join Query Parser</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-query-parsers">BJQ</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-children-query-parser">Child Query Parser</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/other-parsers.html#block-join-parent-query-parser">Parent Query Parser</a>, solr.apache.org</li>
</ul>

Explained: How Zomato Handles 100 Million Daily Search Queries! (Part Two)

Whether you are craving a piping hot pizza or a scrumptious biryani, customers can find their desired dishes effortlessly with just a few clicks. But have you ever wondered what serves as a backbone to all Search features available on the Zomato app and website?



Providing the correct and specific search results to our customers becomes the most basic criterion to ensure a good user experience. With the ability to handle an unlimited number of inputs, our search system is responsible for powering 100+ millions of search queries every day. On Zomato, Solr-Lucene is the base of all search functions and solves some of the use cases most important to us &#8211; restaurant and reviews search, cuisine search, sorting and grouping of restaurants and cuisines based on rating, search score, and much more.



<h3 class="wp-block-heading">So, why do we choose Solr for such use cases?</h3>



Solr is an open-source, enterprise-level search platform built on Apache Lucene, which is a highly popular and widely used library for search-related applications. It has gained trust and popularity among various well-known technology companies, which is a testament to its reliability, scalability, and robustness. This popularity has led to a thriving community of developers and users, making it easier for developers to find support and resources when needed. Additionally, the well-known Elasticsearch engine is also based on the Lucene library.



<h3 class="wp-block-heading">Scale &#8211; Performance &amp; Cost!</h3>



Initially, everything functioned seamlessly, but as we experienced substantial growth in recent years, some of our assumptions failed at scale. One Solr node could not manage the high traffic volume, and the servers resulted in frequent Out of Memory (OOM) errors as we increased the traffic. The OOM issue became a bottleneck and required a larger cluster to handle peak traffic, leading to a substantial increase in server cost. This emphasizes the need for thoughtful consideration of cost implications when making technology decisions, particularly for high-traffic applications like Zomato.



<h3 class="wp-block-heading">OOM, Why?</h3>



The Out of Memory (OOM) issue on our Solr setup was caused by several factors. Solr runs as a Java process, which has a Garbage Collector (GC) that runs at a certain frequency to clean up non-referenced objects and free up memory for further use. However, in this case, GC was struggling to reclaim memory from the Old Gen Heap space, causing the space to grow until it reached an OOM state, resulting in the process being killed. We also observed that the Heap shrinks back to the normal size after it syncs data from the Master Machine.&nbsp;



<blockquote class="wp-block-quote">
We run some of the Solr clusters in a Master-Slave setup, where the Master machine handles indexing (write) and the Slave machines sync data from the Master at a certain frequency. The Slave machines manage the Query (read) traffic behind a load balancer.
</blockquote>



Despite tweaking the JVM parameters and memory settings, the instances continued to go OOM even with constant traffic. <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/applications_remote.html">JVM monitoring</a> and Heap analysis led to the discovery that the size of the caches maintained by Solr was the root cause of the issue, which was confirmed through the Solr Admin Dashboard as well. One of these caches, the Field Cache, was growing without limits, and it was uncovered that the Field Cache cannot be configured to limit its maximum size or count in general.



<figure class="wp-block-image aligncenter"><img decoding="async" src="https://lh4.googleusercontent.com/s4GMAiBkxG2RsKYBkx_mIOsEIKnGRID5GOm0w7ETvyL0TOWSgR-xLtwvNYJH9LtWsHz1XXSyJUWz11CeWYh8E3aCfSe1fRyWR2NE9GVeburtUkGiqEXANXvGZuIc43HpU1LQ4JvZh4NkqlDiQiyN0C0" alt=""/></figure>



<h3 class="wp-block-heading">What is Field Cache?</h3>



Field cache is a crucial component of the Solr search engine, which enables fast and efficient sorting, grouping, and faceting on fields. Solr uses the Field Cache to store the field values of all the filtered documents so that it can quickly access them when needed. This is necessary because when fields are scanned, the process of iterating through all the documents and loading each document&#8217;s fields can be slow and resource-intensive, resulting in multiple disk seeks.



Solr gradually un-inverts and puts the data into the Field Cache at search time, which creates an in-memory column-oriented view of documents that makes these types of queries much faster. However, the Field Cache is not configurable and does not support auto-warming, which means if the cache is purged (after syncing with the master), the aggregation query performance will be impacted.



<pre class="wp-block-code"><code><code>From Doc level stored info
{
 'DocX': {'A':1, 'B':2, 'C':3},
 'DocY': {'A':2, 'B':3, 'C':4},
 'DocZ': {'A':4, 'B':3, 'C':2}
}</code>
</code></pre>



<pre class="wp-block-code"><code><code>To Field level cached info - Field Cache
{
 'A': {'DocX':1, 'DocY':2, 'DocZ':4},
 'B': {'DocX':2, 'DocY':3, 'DocZ':3}
}</code></code></pre>



Another issue with Field Cache is that if there is no limit on the number of such fields, the cache can grow indefinitely over time, making it even more resource-intensive. The <a href="https://solr.apache.org/guide/7_6/dynamic-fields.html">Dynamic Field</a> in Solr is one such construct that theoretically makes the overall number of fields unlimited across all the documents, which can be dangerous for applications using Solr if they have aggregation queries on such fields.



Elasticsearch, on the other hand, limits the total number of fields to 1000 in its default setting, which it calls &#8220;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings:~:text=Consider%20a%20situation%20where%20every%20new%20document%20inserted%20introduces%20new%20fields%2C%20such%20as%20with%20dynamic%20mapping.%20Each%20new%20field%20is%20added%20to%20the%20index%20mapping%2C%20which%20can%20become%20a%20problem%20as%20the%20mapping%20grows.">mapping explosion</a>&#8220;. Unfortunately, Dynamic Fields resulted in a higher number of fields, exceeding far beyond 1000, in some of our use cases.



<h3 class="wp-block-heading">Possible fixes &amp; nuances</h3>



In our setup, Slave instances pull data from the Master instance at a specific frequency. If there is no change in the index version on the Master instance, the Slave will not pull anything and the caches won&#8217;t be purged as there is no data update. To ensure that there is always an update on the Master machine, we started indexing a Mock document at regular intervals that is shorter than the sync interval. This ensures that the Slave gets an updated index version and eventually ends up purging its caches at regular intervals.



While this approach was a lifesaver, it ended up impacting the response time and causing higher CPU usage due to cache auto-warming for other caches and the complete rebuilding of the Field Cache at query time. The constant purging of the cache saved the Slaves from OOM errors, but it drastically limited the throughput of a single Slave node. As too many requests increase the memory usage, it results in OOM well before the Slave can even sync data from the Master. Moreover, syncing too frequently would not only harm the response time but would also consume a lot more CPU and memory resources.



<h3 class="wp-block-heading">The Right Fix &#8211; DocValues</h3>



The main objective was to increase throughput while keeping memory usage to a minimum and eliminate the Out of Memory (OOM) error. The biggest challenge was the growing Field Cache caused by sorting/grouping queries and multiple dynamic fields. The solution was to adopt <a href="https://solr.apache.org/guide/6_6/docvalues.html#why-docvalues">DocValues</a>, a way of storing field values that is more efficient for sorting and faceting than traditional indexing. DocValues use a column-oriented approach with a document-to-value mapping built at index time, which reduces memory requirements and makes lookups faster. Like other index files, DocValues are <a href="https://github.com/apache/lucene/blob/54fba99cb131c7414caf2c21b52aa990ac17db3e/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java#L56">loaded</a> into memory using <a href="https://lucene.apache.org/core/7_6_0/core/org/apache/lucene/store/MMapDirectory.html">MMapDirectory</a>, which relies on the operating system to load the relevant data into RAM instead of the JVM, thereby avoiding OOM issues. Enabling DocValues is straightforward, simply use docValues=&#8220;true&#8221;



<pre class="wp-block-code"><code>&lt;field name="field_name" type="string" indexed="false" stored="false" docValues="true" /&gt;</code></pre>



<h3 class="wp-block-heading">DocValues &#8211; Solved Problems</h3>



DocValues create un-inverted data at index time. With un-inverted data, queries are more efficient and faster, reducing the overhead of un-inversion at runtime. This has also eliminated the problem of the ever-growing FieldValue cache, making the system more scalable and resilient to increasing traffic. The results of this optimization have been remarkable, with a whopping 10x increase in throughput per Slave node. This was rigorously load tested in a real-world production scenario and worked flawlessly. The reduced cluster size also resulted in significant cost savings across multiple Solr clusters used for various use cases, cutting the costs by approximately 80%, close to ₹30L per month.



<h3 class="wp-block-heading">DocValues &#8211; Issues</h3>



There were some issues with DocValues when dealing with Dynamic Fields, particularly with data sparseness and slow indexing processes. Dynamic fields, when defined as DocValues, can result in sluggish segment merge during indexing and optimization processes if they are present in a limited number of documents. The indexing process became time-consuming because of the inefficient handling of Dynamic Field Explosions. This was due to the DocValues producers and consumers that used iterators to travel through every single document for each DocValue field, regardless of its presence in a single document or all documents. This issue was reported in improvement ticket <a href="https://issues.apache.org/jira/browse/LUCENE-7253">LUCENE-7253</a> which later got fixed in the <a href="https://lucene.apache.org/core/7_0_0/changes/Changes.html#v7.0.0.improvements:~:text=LUCENE%2D7489%3A%20Better%20storage%20of%20sparse%20doc%2Dvalues%20fields%20with%20the%20default%20codec.">Lucene-7.0.0</a> major version upgrade, where the iterator only traveled through documents with a non-empty value for a given field. There were further issues specific to our use cases for which we forked the Solr-Lucene repo and added fixes, and updated unit test cases for them.



Based on these improvements, we decided to upgrade our cluster from v6.x to v7.6.0, and the results were fantastic. The indexing speed was at-par, latency was reduced, cache hits improved, resiliency was drastically improved, and the overall cost was lower compared to the previous state. Further upgrades to some of our clusters to v8.7.0 have also shown the same positive results, and the improvements continue to work flawlessly.



<h3 class="wp-block-heading">Afterword</h3>



When it comes to optimizing Solr performance, every use case is unique and requires experimenting with various configurations, including JVM and cache settings. Choosing the right schema to index data can also have a significant impact on performance, as it helps to improve query speed, reduce index size, and minimize memory usage during runtime. Another crucial aspect is adjusting Solr cache parameters, which can greatly improve search latencies and minimize memory requirements. Writing efficient queries is also crucial for maximizing cache hits, but that falls outside the scope of this discussion. When dealing with faceting (aggregations) and sorting queries, using DocValues is the best option as it reduces memory footprint and minimizes the risk of memory leaks. Finally, it&#8217;s important to note that Solr performs best with a relatively static index, so it&#8217;s best to avoid frequent syncing with the master node, depending on the use case.



<h3 class="wp-block-heading">More to come</h3>



DocValues and other fixes addressed the issue, but the underlying problem of Mapping Explosion remained. In future articles, we will explore how to overcome Dynamic Field explosion in inappropriate scenarios and the reasons behind our migration to Solr Cloud architecture for some clusters.



This blog was written by <a href="https://www.linkedin.com/in/saurav-singh-25317625/">Saurav Singh</a>.



<hr class="wp-block-separator has-css-opacity"/>



Sources:



<ul id="block-dc06f231-be1a-4da1-96e4-394b68f5cd17">
<li><a href="https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/applications_remote.html">JVM Monitoring</a>, docs.oracle.com</li>



<li><a href="https://solr.apache.org/guide/7_6/dynamic-fields.html">Dynamic Field</a>, solr.apache.org</li>



<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings:~:text=Consider%20a%20situation%20where%20every%20new%20document%20inserted%20introduces%20new%20fields%2C%20such%20as%20with%20dynamic%20mapping.%20Each%20new%20field%20is%20added%20to%20the%20index%20mapping%2C%20which%20can%20become%20a%20problem%20as%20the%20mapping%20grows.">Mapping explosion</a>, elastic.co</li>



<li><a href="https://solr.apache.org/guide/6_6/docvalues.html#why-docvalues">DocValues</a>, solr.apache.org</li>



<li><a href="https://github.com/apache/lucene/blob/54fba99cb131c7414caf2c21b52aa990ac17db3e/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java#L56">DocValues loading</a>, github.com</li>



<li><a href="https://lucene.apache.org/core/7_6_0/core/org/apache/lucene/store/MMapDirectory.html">MMapDirectory</a>, lucene.apache.org</li>



<li><a href="https://issues.apache.org/jira/browse/LUCENE-7253">LUCENE-7253</a>, issues.apache.org</li>



<li><a href="https://lucene.apache.org/core/7_0_0/changes/Changes.html#v7.0.0.improvements:~:text=LUCENE%2D7489%3A%20Better%20storage%20of%20sparse%20doc%2Dvalues%20fields%20with%20the%20default%20codec.">Lucene-7.0.0</a>, lucene.apache.org</li>
</ul>

Explained: How Zomato Handles 100 Million Daily Search Queries! (Part One)

What do you say when someone asks you why are you participating in a Hackathon? Saying that you&#8217;ve come to a hackathon for (free) food is not the ideal response, and more so to a CEO.&nbsp;



It was a Saturday morning, and I was in the Zomato headquarters for a hackathon. Our team was working in one of the many glass meeting rooms when a serious-looking man in a black hoodie walked in and introduced himself, saying, &#8220;Hi, I&#8217;m Deepi&#8221;.&nbsp;



Hearing this, one of my teammates said, &#8220;I got it&#8221;. I, on the other hand, was super confused. I thought, maybe, I heard &#8216;DP&#8217;, which is an acronym for &#8216;delivery partner&#8217; at Zomato. I thought it was an internal joke I was unaware of. So, I just smiled and let it pass.&nbsp;



We began talking about the project, and after sharing and receiving a few suggestions, the man in black asked, &#8220;Yahan kya karne aaye ho?&#8221;, which loosely translates to &#8220;what are you doing here?&#8221; I mentioned the first (honest) thing that came to my mind, &#8220;khaana khaane&#8221;, translating to &#8220;to have food&#8221;. And, we all had a hearty laugh.&nbsp;



Shortly after he left, my teammate, who was smiling earlier, nudged me and said it was the founder you were speaking with. It took me a good ten minutes to let that sink in.&nbsp;



Well, this is one of the many unusual things that happened last weekend at the headquarters. Let&#8217;s start from the beginning.&nbsp;&nbsp;



A few weeks back, I was browsing through my LinkedIn feed and came across a post on HackaNoodle – a hackathon powered by Zomato &amp; Blinkit, with a special focus on women and the trans community. A big fan of hackathons/coding competitions in general, I quickly opened the link to learn more about it and, soon after, registered my name.&nbsp;



Since HackaNoodle was scheduled for November, we all had a few days to form our teams. We made a team of four – <a href="https://www.linkedin.com/in/ritikap/">Ritika Patel</a>, <a href="https://www.linkedin.com/in/palak-bansal-b2379a145/">Palak Bansal</a>, <a href="https://www.linkedin.com/in/akshit-sehgal-062b901b1/">Akshit Sehgal</a>, and me (<a href="https://www.linkedin.com/in/ng11213/">Nihal Gupta</a>). We started meeting online on Zoom calls to brainstorm what we wanted to build. With Diwali and festive preparations, the days went by quickly. Finally, the much-awaited weekend was here.



<h2 class="“text-font-family wp-block-heading">The D-day</h2>



On November 5th, we all were standing outside a large building in Gurugram with a huge, red board on top reading &#8216;Zomato&#8217;. It did not feel like anything big until we stepped in. But oh, how it all changed as soon as I entered the office. I could feel the energy all around me – hundreds of coders, a lot of hustle, and the crew welcoming each of us. However, one thing that (pleasantly) surprised me was the number of women present there. We had 40+ teams, and all of them were led by women.&nbsp;



At the gates, we all got our goodie bags. Like a child, I took my bag aside, found the nearest table, sat there and started unpacking everything one by one. There was a cool black hoodie with my name written on it (which I wore then and there), a laptop pad, a water bottle, stickers (special thanks to the person who included cool stickers in the goodie bag, I love pasting stickers on my laptop), and a few munchies.&nbsp;



<figure class="wp-block-image aligncenter size-full"><img decoding="async" loading="lazy" width="600" height="400" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_5860-1-600x400-1.png?resize=600%2C400&#038;ssl=1" alt="" class="wp-image-11710" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_5860-1-600x400-1.png?w=600&amp;ssl=1 600w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_5860-1-600x400-1.png?resize=300%2C200&amp;ssl=1 300w" sizes="(max-width: 600px) 100vw, 600px" data-recalc-dims="1" /></figure>



Once everyone settled in, the engineering team started briefing us on HackaNoodle – what it was, the theme, the judging criteria, and more. Here, we got to know how Zomato was founded because of a pizza craving Deepinder Goyal had 14 years back (now that I recall this, my answer does not sound dumb – food can take you places!). The story made us realise that we seldom remember how everything starts with a basic idea – be it a product or an organisation.&nbsp;&nbsp;



And we had 24 hours to think of that idea and build it from scratch.&nbsp;&nbsp;&nbsp;



<h2 class="“text-font-family wp-block-heading">And it began…</h2>



The marathon started, and all teams got a room each to eat, sleep, code, and repeat!



We hurried to discuss what we should build. Since the theme was &#8220;Build India&#8217;s Next Unicorn&#8221;, we had many horizons to explore. The challenge was to pick out the best.&nbsp;



In between our brainstorming sessions, the Zomato team came for check-ins. They mentored and brainstormed with us and gave us real-time suggestions. It felt great to work with such brilliant minds.



<h2 class="“text-font-family wp-block-heading">Code, code, and more code</h2>



Once we finalised the idea, the rest of the day was packed with coding, caffeine, fixing bugs, and, yes… more caffeine.&nbsp;



We continuously tested our code, scratched our heads, and pulled our hair out when the code was not working. But in the end – we made it work!



<h2 class="“text-font-family wp-block-heading">Food scenes</h2>



What kept us working all those long hours without snapping at each other was food. We were in Zomato&#8217;s office, so no doubt the food arrangements had to be grand! Everything we needed – unlimited beverages, noodles, and munchies – was within an arm&#8217;s distance. A great variety of dishes were served during every meal.



It felt like I was Charlie, and this was my chocolate factory.&nbsp;



<figure class="wp-block-image aligncenter size-full"><img decoding="async" loading="lazy" width="600" height="400" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_6134-1-600x400-1.png?resize=600%2C400&#038;ssl=1" alt="" class="wp-image-11711" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_6134-1-600x400-1.png?w=600&amp;ssl=1 600w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/DSC_6134-1-600x400-1.png?resize=300%2C200&amp;ssl=1 300w" sizes="(max-width: 600px) 100vw, 600px" data-recalc-dims="1" /></figure>



<h2 class="“text-font-family wp-block-heading">The showtime</h2>



The next morning, we had to showcase what we had built. On one side were the top leaders of Zomato and Blinkit, and on the other, a bunch of college students giving their 100% to make India&#8217;s next big unicorn.&nbsp;



If you&#8217;ve ever been to a hackathon, you know how excruciating the pitching process can be. No matter how great your idea is, it&#8217;s not worth it if you can&#8217;t sell it. We built a personal workout training application using AI that can track your body while you are performing any exercise or yoga. Our AI would provide feedback, and real-time analytics like posture correction, calories burnt, and activity time. In a nutshell, we were providing people with their own personal trainer, who is available at all times.&nbsp;







<figure class="wp-block-image aligncenter size-full"><img decoding="async" loading="lazy" width="600" height="337" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/exercise-AI-demo-1-600x337-1.png?resize=600%2C337&#038;ssl=1" alt="" class="wp-image-11712" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/exercise-AI-demo-1-600x337-1.png?w=600&amp;ssl=1 600w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/02/exercise-AI-demo-1-600x337-1.png?resize=300%2C169&amp;ssl=1 300w" sizes="(max-width: 600px) 100vw, 600px" data-recalc-dims="1" /></figure>



Before selling it to others, we sold it to ourselves first. We practised our pitch again and again. And by the time we had the actual presentation, we felt super-prepared.&nbsp;&nbsp;



<h2 class="“text-font-family wp-block-heading">The results</h2>



After what seemed like an eternity, we got the results. Gunjan Patidar (Chief Technology Officer, Zomato) announced that we made it to the top six. We now had to pitch the product to the audience, and they would vote to finalise the winners.&nbsp;



Here are the top three ideas that garnered the most votes (and for all the right reasons) –



<ul>
<li>A tool for advanced detection of dyslexic patients, built by Team &#8216;The Noob Army&#8217;</li>



<li>Generative art videos based on audio clips, created by Team &#8216;Dining Philosophers&#8217;</li>



<li>An e-commerce platform for differently-abled people, developed by Team &#8216;Agile Coders&#8217;</li>
</ul>



It was exciting to see these ideas unfold and witness the great power of tech – how it can solve so many problems when coupled with the right intent. Although we didn&#8217;t win the hackathon, I consider myself extremely fortunate because we got a chance to brainstorm with top leaders and engineers, which was one heck of an experience. We built a useful product, secured a place in the top six, had loads of fun, met enthusiastic coders like us, had delicious food, and learnt a lot.&nbsp;



Can&#8217;t thank Zomato and Blinkit enough for organising such a fantastic and memorable event!&nbsp;



This blog is written by <a href="https://www.linkedin.com/in/ng11213/">Nihal Gupta</a> – a B.Tech (CSE) student at Maharaja Surajmal Institute of Technology (MSIT) who came to the Zomato headquarters to participate in HackaNoodle&#8217;22 – a coding competition hosted by Zomato and Blinkit.&nbsp;



____



This is a personal blog. Any views or opinions expressed in this blog are personal and do not represent the views of the organisation or institutions that the writer of the blog may or may not be associated with in a professional or personal capacity unless explicitly stated.



All team names mentioned above are decided by the participants themselves and do not form any legal entities. Additionally, they do not represent Zomato or its group companies.&nbsp;