Vinifera is an open-source monitoring tool that detects and responds to accidental or critical sensitive leaks. It monitors public contributions made by developers across their repositories, forks, and gists, and notifies if any sensitive information is leaked. Started as a tool to monitor Github activity, Vinifera has now evolved to support multiple platforms like Gitlab, Slack and Asana.



Since we started using Vinifera in 2019, we have been able to prevent many accidental leaks on public platforms. We also <a href="https://github.com/zomato/vinifera">open-sourced major components of Vinifera</a> with the hope that it can help other companies in strengthening their security posture.



Addressing Security Vulnerabilities: Vinifera&#8217;s Solution for Data Exposures and Proactive Monitoring Sensitive data exposures are common yet frequently overlooked security vulnerabilities that are faced by many organizations. This vulnerability class is hard to avoid due to multiple points of exposure and is easy to exploit, making it a sweet target for researchers.



From an outsider&#8217;s perspective, getting data from internal sources is hard, but it is easy to monitor and wait for someone else to make that information public. The severity of such leaks can range from minor to critical. For example, the leakage of an internal database schema, while not posing an immediate security risk, could potentially assist someone in connecting relevant information and uncovering potential vulnerabilities. The leak of third-party API keys and credentials is not only concerning, but it also exposes the associated data to potential attacks &#8211; providing attackers with an opportunity to infiltrate the network further. In order to address these security vulnerabilities, the following is done —



<ol>
<li>We host an internal <a href="https://privatebin.info/">private-bin</a> and <a href="https://github.com/gchq/CyberChef/">Cyberchef</a> instance to facilitate easy sharing of internal data and provide essential capabilities for developers in terms of data transformation. While this approach offers some level of support, it does not entirely resolve the problem.</li>



<li>Vinifera provides continuous monitoring capabilities that enable swift identification and response to leaks. By tracking and scanning registered sources such as users, repositories and gists, Vinifera promptly detects changes or anomalies. This proactive approach enhances data security by allowing for immediate action and risk mitigation towards data leak.</li>
</ol>



Vinifera&#8217;s Development Journey Vinifera was built using a standard Rails app. The development process involved the following key components —



<ol>
<li>Sidekiq: <a href="https://github.com/sidekiq/sidekiq">Sidekiq</a> is used for efficient job lifecycle management, enabling tasks such as scheduling and retries</li>



<li>Gitleaks Integration: <a href="https://github.com/gitleaks/gitleaks">Gitleaks&#8217;</a> Docker image is used for scanning with communication facilitated through the Docker API</li>



<li>Data Storage and Cache: <a href="https://www.postgresql.org/">Postgres</a> was chosen as the persistent data store, while Redis served as the application cache, ensuring reliable performance</li>



<li>Slack Integration: Vinifera provides real-time monitoring and event checking through Slack</li>



<li>On-Call Team Collaboration: Pager events are triggered to notify the on-call team and respond to any security incidents.</li>
</ol>



<figure class="wp-block-image size-large"><img decoding="async" loading="lazy" width="640" height="747" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=640%2C747&#038;ssl=1" alt="" class="wp-image-13096" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=877%2C1024&amp;ssl=1 877w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=257%2C300&amp;ssl=1 257w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=768%2C897&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=1315%2C1536&amp;ssl=1 1315w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?resize=1568%2C1831&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image10-1.png?w=1712&amp;ssl=1 1712w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /><figcaption class="wp-element-caption">Vinifera’s architecture</figcaption></figure>



 Vinifera&#8217;s Operations Vinifera&#8217;s operations can be broadly categorized into three key phases —



<ol>
<li>Discovery: During the discovery phase, Vinifera scans and identifies all assets, actively searching for new targets to monitor. Once identified, these targets are registered for ongoing monitoring and polling.</li>
</ol>



<img decoding="async" loading="lazy" src="https://lh6.googleusercontent.com/RgxcKQsNKSC1xKf3MVndUE8lqsSSAs8-QWJOYN4UF8umqZ1X7Y0xXpSqsEC9LsF4M9dUIzcl96qnGmrvucfFyHKX2lHxi7QbUqnfGTLmu_5ajvD62qXTP6boeVeX_cI1xc8bKBM5ft_XplKD5IbLuRBasmmwzJl4" width="522" height="249">



<ol start="2">
<li>Lifecycle: After onboarding a new employee into our GitHub organization, we initiate a registration process where they are identified as targets. Subsequently, we conduct comprehensive scans on their public repositories and Gist &#8211; treating them as “assets”. To efficiently track changes, revisions to each asset are stored in a manner that allows quick identification of modifications.</li>
</ol>


<div class="wp-block-image">
<figure class="aligncenter is-resized"><img decoding="async" loading="lazy" src="https://lh4.googleusercontent.com/QRkNGfs8sY8fgVstPG03SWZldQU9-Zlu7LiGP1XGupUoQBXJc9TStHZ4jwO97wPM8ThcrjGRTI_AqtPCR3JrO7NXevy1JPjXLkZ1ZbEgqmG2GEYcfb5v1yg0uvBKvU5FDev-bd9r1ojNV9mr8FAOg4la9mzzk8BH" alt="" style="width:514px;height:268px" width="514" height="268"/></figure></div>


<ol start="3">
<li>Scanning: Scanning is triggered via Sidekiq, and all scans are performed within isolated Docker containers, ensuring secure and reliable execution. The scanning process varies based on the type of scan required &#8211; whether it is a repository scan or an analysis of raw files. Vinifera further leverages Datadog to monitor relevant statistics and ensure the overall health of the service.</li>
</ol>



<figure class="wp-block-image size-large"><img decoding="async" loading="lazy" width="640" height="275" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=640%2C275&#038;ssl=1" alt="" class="wp-image-13105" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=1024%2C440&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=300%2C129&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=768%2C330&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=1536%2C659&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?resize=1568%2C673&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?w=1999&amp;ssl=1 1999w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/image1.png?w=1920&amp;ssl=1 1920w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



An Overview of the Vinifera System After a successful execution of the &#8220;target_monitor&#8221; Sidekiq job, it is re-enqueued for the subsequent scan, with the enqueue information stored in the database. This enables fine-tuning and control over the timing of the next scan.



In the event of failures, Vinifera carefully analyzes the type of errors encountered. If the failures are attributed to issues with GitHub or the Docker Box, appropriate measures are taken to handle and throttle jobs, providing the system with a window to avoid further errors. Scalability Enhancements: Boosting Stability and Performance To enhance the stability and scalability of Vinifera, we implemented several measures —



<ol>
<li>Auto throttling of jobs based on error types: When relying on third-party services or internal services with limited capacity, it is necessary to have checks to avoid overwhelming the service. We introduced auto throttling of jobs based on error types, dynamically adjusting limits for specific periods. This approach helps avoid disruption caused by exponential retries, ensuring smooth operation of the service.</li>
</ol>


<div class="wp-block-image">
<figure class="aligncenter"><img decoding="async" src="https://lh3.googleusercontent.com/UXE6hZS9vZDNVo7hySAHty-xMYfkp1ZYII9Pk_NXGmp6yLvTdOhkp5ZZelmbLCzKGxoV9b_3bBwg-LVMeMg_Je2jooxtF_eC9XZMSLDPckdjCMOyByBdr1he3FKrDHa-Wx2Ip7eOJFxGQGFc0uDknYU" alt=""/></figure></div>


<figure data-carousel-extra='{"blog_id":1,"permalink":"https:\/\/zomatoblog.com\/introducing-vinifera\/"}' class="wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex">
<figure class="wp-block-image size-large"><img decoding="async" loading="lazy" width="640" height="295" data-id="13101" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=640%2C295&#038;ssl=1" alt="" class="wp-image-13101" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=1024%2C472&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=300%2C138&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=768%2C354&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=1536%2C709&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=2048%2C945&amp;ssl=1 2048w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?resize=1568%2C723&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.10.57-PM.png?w=1920&amp;ssl=1 1920w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>
</figure>



<ol start="2">
<li>Caching GitHub API responses to avoid rate limiting: To mitigate rate-limiting issues, we leveraged caching of GitHub API responses. We employed the <a href="https://github.com/sourcelevel/faraday-http-cache">faraday-http-cache</a> library, utilizing response headers like Etag, Last-Modified and Age to cache the responses in Redis. This however, comes with a small impact on storage since responses are now stored in Redis memory. To ensure efficient storage management, Vinifera employs a selective caching approach. Heavy responses and resources that are not needed for future access are intentionally excluded from the caching process.</li>
</ol>



<ol start="3">
<li>Optimized scanning of big forks and external projects: Scanning large forks, such as those of substantial open-source projects, posed resource utilization challenges. To address this, we directly determined and scanned the commit patch instead of cloning the entire fork. One of the main causes that spiked resource usage on our Docker daemon was scanning big forks. This arose when a developer forked a large open-source project for contribution, and we scanned the entire repo to look for violations. Some forks were enormous (for example, Linux kernel repo, odoo, android AOSP source code) and sometimes caused the cluster to choke. In some cases, the maximum memory used for scanning reached almost 1GB and ~99% of the total&nbsp;CPU. By leveraging the user activity GitHub API, we could quickly identify commits pushed by developers to collaborative projects or forks, extracting the relevant patch for direct scanning.</li>
</ol>



<ol start="4">
<li>Running scans on a dedicated docker daemon: Separating scanning operations to a dedicated machine greatly improved performance and prevented resource congestion. By isolating the scanning process on a separate Docker daemon, we were able to limit the blast radius in case of any issues. This approach ensured the continuity of discovery operations in an uninterrupted manner, in spite of  the dedicated Docker machine experiencing errors. Additionally, we optimized resource allocation by restricting scans to a single CPU core and a fraction of CPU weight, preventing resource exhaustion. <img decoding="async" src="https://lh4.googleusercontent.com/6WIV4V0fRBRaBdP-yy3F4_DWlTk1wturzDsDMdkUdlwd3Jvf11FdSw6ZK6MCo2AW404w86cMA5dulWyuEmaGt5rR2_NK9ibVDvPsJdO_NUkp0JWJqStGCVplxNwriOmUSSD1A5rcXe7vENqHdjFsJCU" style=""></li>
</ol>



Expanding Platform Support: Vinifera’s Multi-Service Compatibility Vinifera&#8217;s journey began as a monitoring tool for GitHub sources, but it naturally evolved to support GitLab and other services as well. Depending on the platform, Vinifera employs different monitoring approaches to ensure comprehensive coverage —



<ol>
<li>Asset Monitoring Approach: Services like GitHub and GitLab are monitored using an asset monitoring approach. This involves registering assets and continuously looking for any changes or updates.</li>



<li>Query-Based Monitoring: Services like Slack, Asana, and Intelx are monitored using a query-based approach. Vinifera periodically polls search results against fixed queries, detecting any changes or relevant information.</li>
</ol>



Currently, Vinifera supports scanning for the following platforms —



<ul>
<li>GitHub</li>



<li>GitLab</li>



<li>Slack</li>



<li>Asana</li>



<li>Intelx</li>
</ul>



<a href="https://intelx.io/">Intelx</a>, specifically, serves as a valuable source for monitoring information exposures across various assets. It functions as a search engine that collects and indexes information from multiple public sources. Vinifera leverages Intelx to identify and neutralize any potential exposure risks proactively.



<figure class="wp-block-image aligncenter size-large"><img decoding="async" loading="lazy" width="640" height="233" style='width:100%' src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=640%2C233&#038;ssl=1" alt="" class="wp-image-13121" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=1024%2C373&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=300%2C109&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=768%2C279&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=1536%2C559&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=2048%2C745&amp;ssl=1 2048w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?resize=1568%2C570&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.35.39-PM-3.png?w=1920&amp;ssl=1 1920w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /><figcaption class="wp-element-caption">A snapshot showing information captured from Intelx for our assets </figcaption></figure>


<div class="wp-block-image">
<figure class="aligncenter size-large is-resized"><img decoding="async" loading="lazy" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=640%2C229&#038;ssl=1" alt="" class="wp-image-13113" style="width:834px;height:299px" width="640" height="229" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=1024%2C367&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=300%2C108&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=768%2C275&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=1536%2C551&amp;ssl=1 1536w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=2048%2C734&amp;ssl=1 2048w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?resize=1568%2C562&amp;ssl=1 1568w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?w=1280&amp;ssl=1 1280w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/09/Screenshot-2023-09-13-at-9.36.50-PM.png?w=1920&amp;ssl=1 1920w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /><figcaption class="wp-element-caption">A snapshot showing information captured from Intelx for one of our queries</figcaption></figure></div>


Distributed Security Alerting We built a robust alerting and acknowledgment system – DiSeAl (Distributed Security Alerting) to distribute the security alerting across the entire organization and integrated it with Vinifera. Whenever a new resource is registered in Vinifera, our internal lookup logic identifies the corresponding owner within our organization. Subsequently, an alert is sent to the owner, ensuring they are promptly notified of any new public assets associated with their account discovered on Vinifera. This proactive approach empowers users to stay vigilant and report any accidental exposures, aiding in identifying and mitigating potential risks


<div class="wp-block-image">
<figure class="aligncenter is-resized"><img decoding="async" loading="lazy" src="https://lh3.googleusercontent.com/grYNj4M8DpnFA5yC9iunlyDPFYfW4I8T-nu33llB0eQhLTcXU54O880nDvy7OA1iGnmTVkc50733_alKr-slNGSvp_ia7-ZcC3xfqZWWAWLy6nF1LCX2CpyBNA6K306GT_7j2wah7CamiYbremRzGNqkPvK94aVR" alt="" style="width:488px;height:415px" width="488" height="415"/></figure></div>


Conclusion Vinifera serves as a powerful monitoring tool, proactively detecting and mitigating accidental leaks across multiple platforms. Vinifera strengthens the security posture with continuous monitoring, robust alerting, and multi-tenant support. With Vinifera, we can take proactive measures to identify and mitigate potential risks, bolster data protection, and prevent leaks.By open-sourcing Vinifera, Zomato aims to contribute to the broader security community, inviting others to strengthen their security postures and swiftly identify and mitigate potential leaks. If you are passionate about solving security problems at scale, we would love to hear from you at <a href="mailto:infosechiring@zomato.com">infosechiring@zomato.com</a>.

Introducing Vinifera: A guide on how it prevents and mitigates accidental data leaks at scale

Building Kimchi for hack-a-noodle 2022 – Zomato’s on-the-fly iOS UI rendering engine

At Zomato, our internal microservices and monolithic applications collectively produce a significant amount of logs each day. With a maximum production rate of 150 million logs per minute, this leads to the production of over 50 TB of uncompressed logs per day.&nbsp;



In the past, we stored all of these logs on self-hosted Elasticsearch. However, as the traffic on our platform and our services grew over time, our logs increased exponentially. Given this increase, managing the clusters became difficult from an operational standpoint. Given the nature of the business, we also had to over-provision our clusters for the variable traffic patterns. While the costs kept rising, we were unable to provide the experience we had originally hoped to.&nbsp;



<h2>Clicking with Clickhouse – Building LogStore</h2>



In the early days, we explored migrating a few sets of logs to S3 stored in <a href="https://cwiki.apache.org/confluence/display/hive/languagemanual+orc">ORC format</a> using apache-spark &#8211; which would be queryable through Trino. While it definitely brought down the cost substantially, querying such huge logs and keeping query runtimes generous under 10s wasn’t possible. We had an ingestion lag of 5-10 minutes since we were writing large files to improve query latency. With the objective of keeping the end-to-end query latency under 1s and ingestion lag of less than a minute, we explored various options which could potentially replace the ELK stack.



Clickhouse emerged to be the best candidate for our use case, and we decided to move forward with it. It could handle terabytes of data while keeping the latency low. Also, Clickhouse is horizontally scalable, with a shared-nothing architecture. This further reduced the operational overhead. 



<img decoding="async" loading="lazy" width="923" height="584" src="https://lh5.googleusercontent.com/SgNbcOSsqTH_md4a_35pJ-LO_D6tKwLU9-P-ppN5KUZffplrMHcfnpw8QdR0YZVd1P5NRUj_F2LK2X5QGncMIpS87plK51tZE8wH-_LTWKHuAabbiIU6PO7SdqoC0w9SikKy26M4xAJaJcF9c-8v5Dg"> 



<h2>Behind the curtain: The wizardry of our design decisions</h2>



We used various design choices to achieve high performance from the setup while keeping the costs low.



Scale Description: As previously mentioned, the data size, at times, was more than 50 TB daily, with events peaking at 150M/ minute. The requirement for querying all these logs within seconds of their production was paramount. LogStore received about 12,000 queries/day with a time range from 15 minutes to 7 days. To cater to this, the Clickhouse cluster ran 10 AWS EC2 nodes of M6g.16xlarge. The current cluster was scaled up to cater to burst queries in case of any production incident.



Inserts: Instead of relying on Clickhouse Kafka plugins, we opted for custom Golang workers. These workers efficiently batched the requests and performed insertions into Clickhouse, significantly reducing the overhead on Clickhouse. To optimize costs, we ran these workers on spot instances, resulting in additional savings.&nbsp;



To optimize performance, the inserts were batched to minimize I/O and CPU cycles. We grouped up to 20,000 messages per table, ensuring a maximum lag of 5 seconds. This approach significantly reduced the number of inserts at Clickhouse.



For faster data ingestion, we leveraged the native format to insert data into Clickhouse, achieving a speed boost of approximately 1.8 times compared to using HTTP. This native format was also less I/O intensive, resulting in more efficient processing by Clickhouse.



To distribute the workload evenly, the workers followed a round-robin strategy, inserting the batched logs into any available Clickhouse node. If a node was unavailable or under heavy load, the request was seamlessly redirected to another functioning node. This ensured a robust and reliable data ingestion process.



Schema Design: One of the most significant challenges was designing a schema that could accommodate the semi-structured nature of logs. This was achieved&nbsp; by creating a semi-structured table design, where the logs&#8217; common columns are kept at the top level, and all unmatched columns are dumped into a map&lt;String, String&gt; column.&nbsp;



This map exploded to the top level in the API, making it easy for clients to access the data. The top-level fields can use special codecs per column to improve the compression ratio. Also, we use LowCardinality String wherever possible to achieve a very high compression ratio for columns.



<pre class="wp-block-prismatic-blocks"><code class="language-">create table if not exists foo_service (
	ts 	Datetime,
	env 	Lowcardinailty(String),
	container_id	Lowcardinailty(String),
	trace_id 	FixedString,
	msg 	String,
	offset 	Uint64 codec(DoubleDelta,zstd)
	_others 	Map&lt;LowCardinality(String), String&gt;
	INDEX foo_service_msg_index
	TYPE tokenbf_v1(212062,3,0)
	GRANULARITY 1
)
Engine = MergeTree
Partition by toYYYYMMDD(ts)
order by (env, trace_id, ts)</code></pre>



Custom SDK: The schema problem also needed to be fixed at the root level by urging the developers to use structured logging, instead of dumping everything into the message column and then finding a needle in the haystack. We built an SDK which helps in structuring the logs. This SDK standardized some top-level fields &#8211; like caller, log production time, log production stream, etc. so that it was easier to manage and clean up logs. 



<pre class="wp-block-prismatic-blocks"><code class="language-"> log.With(
 log.Int("user_id", userID),
 log.String("url", url),
 log.Err(err),
 ).Info("could not make http request")</code></pre>



So that the user can search the logs with user_id.equals(1) instead of message like ‘%user_id = 1 %’



Frontend: We built a custom dashboard for developers to apply filters and view logs. The dashboard’s web vitals score is at par with industry standards with an <a href="https://web.dev/fcp/">FCP score</a> of 0.95 seconds and an <a href="https://web.dev/lcp/">LCP score</a> of 1.90 seconds. To achieve this, we used a lazy loading technique which defers loading of non-critical resources at page load time. We also built a filter application and date range picker flow which is open for any customizations in the future according to developer needs. Since log data is huge (approx 3K records / 1MB), we cannot render all the data on the page at once. We applied DOM virtualization concepts to render only a required number of records that are visible in the viewport. <img decoding="async" loading="lazy" width="2048" height="1186" src="https://lh6.googleusercontent.com/pbUMbkDkxvqCrvJD22Y-yG77_6H2CSgWKQ0hSqni_HcHYNYacDfZDog7TBLwI8XuY3wArb_kZwQJx4ATJjxZQlZ8JBLOzOF7Zydlei07tGiBg-k_vaA3BiQpzDKfTtJKBGmf7PHJ3y2UtMEVSlhTJSM"> <img decoding="async" loading="lazy" width="2048" height="1192" src="https://lh3.googleusercontent.com/pbUVr2Vk_-MCaMM9TfH5eSBdrEkgu355uI4GIvQ44N_frzhplNaAxgvzQFCfCCgodnIw7MsgOtn-yOFk__C0dk18cIJgDfEhNtu_e_TXYD5cZqrCaMxTtLtG4xa1AO8mNR7N3Af8sOYnfacgUQW3bHw">



<figure class="wp-block-image"><img decoding="async" src="https://lh5.googleusercontent.com/vtMxji3V-ux49PVBVFMZXTg8CIu9g0H0JhiZyhTO55tNQYqcD7kywCpzXUdQ9rt2tVGTcQJwFPZN4PQdzIJLJ5PkT2iFNZQMLNUax4JeRmvUB4-T02zLAo5MFLYIDxH6bpGjMSQieFcCYMzij869CoI" alt=""/></figure>



Replication: We opted out of replication, and consequently Zookeeper, to simplify operations since the data is already persistent on S3 and the attached EBS is being backed up. In case of an unmerry event, the new node gets attached to the EBS and is up and running in no time.



<figure class="wp-block-image"><img decoding="async" src="https://lh3.googleusercontent.com/ETZ4WzGDgzcbXfpwMvlqoJb42SJlE0uetcqLlNLuWnLH7dLhWUNaf5iaRZoyNsDZS3GqFCL2smdpDb3OkKCbXvRR_gd1OtA07XYwlhU4qzoGfkBVQl-sF-TvPe_Pdas5Kk50XKe-v52DtoFKpxPU_GM" alt=""/></figure>



Data Tiering: We manage the data lifecycle by moving older data to the cold tier from the hot tier after every 24 hours using TTL and delete the data after 3 months . For our workload, which was mostly CPU and I/O bound, it made sense to use AWS Graviton C6g instances. To cope with IO-intensive workloads, each node was attached to multiple disks with gp3 disks with 4000 IOPS and 1000 MB/s.



<figure class="wp-block-image"><img decoding="async" src="https://lh6.googleusercontent.com/-q42plmXWjIfx61IuQbqQKWMkfCgEzid0eZ-eAzMdhW167bYFdCnZdg7VC8F5-qOBz83udtg_NYtPOKXURxv1roj4CATYQuRgeUyNRqAxH4MXFdU4vrCXXLwcdxu7pAYo3QxC1U5NQQYhnxFZmALcMk" alt=""/></figure>



Secondary Indexes: By utilizing the tokenbf_v1 index, the speed of like queries improved as many parts were bypassed if the index didn&#8217;t match. This was particularly beneficial for queries that involved scanning large amounts of data, which would otherwise be prohibitively costly to execute on their own and surpass scan limits.



The inverted index was recently added as an experimental feature in Clickhouse, but it led to frequent memory issues and performance degradation in our POC. We could switch to inverted indexes once it matures.



<figure class="wp-block-image"><img decoding="async" src="https://lh3.googleusercontent.com/cjynB46IhpVJUqRNxxj0Ki08qZC3bAH0JRqYWiHGg8nqrRp5vcXAoPytiRJaHjvFQ-I2wwqK_QrD2sAuEOrEKpbS0SQ3yB3mwWguT7Dd5s3hjBxTvFPH6WHqnl8R-DSDU54ZVX4m2YeHwhNFbWoP__M" alt="" title="Chart"/></figure>



<figure class="wp-block-image"><img decoding="async" src="https://lh5.googleusercontent.com/EfigoKhNwBtIeCouF2EgT0Zp2puB-9a93EDVE7fO2zATdJvOmH4DCG03vRP6T1C5v8H0dIKlR5RQf_MyEutkdPGnHYkBCF_rXJ9MbPGiYIfs_D2dlP3evJipYaSS3X2qLJNE1G-oCkYVoqguaKlehw" alt=""/></figure>



Deciding the Size of Bloom Filter: “A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.” False positive matches are possible, but false negatives are not – in other words, a query returns either &#8220;possibly in set&#8221; or &#8220;definitely not in set&#8220;.&nbsp; We used bloom filter in the tokenized message column to verify if the search query was present, effectively filtering out non-matching granules, thus speeding up the search results. For each block of 8000 records with an average of 50 words per record, n ( number of elements in a filter ) = 400K, p (probability of false positives ) as 0.1 and 3 hash functions i.e. k. The size of filter per block tends to be 235 Kib. This helps in speedy search results which can handle all the records.



n = ceil(m / (-k / log(1 &#8211; exp(log(p) / k))))



p = pow(1 &#8211; exp(-k / (m / n)), k)



m = ceil((n * log(p)) / log(1 / pow(2, log(2))));



k = round((m / n) * log(2));



Ensuring resiliency of the system: The system implemented query throttling mechanisms at both the user and system levels to tackle the problem of performance degradation caused by an overwhelming number of queries.&nbsp;



In instances where the system was inundated with a significant number of unoptimized queries, it selectively terminated the queries that were consuming excessive resources or taking an unreasonably long time to execute, prioritizing other queries and ensuring the system&#8217;s continued availability to most users. Additionally, the read queries&#8217; niceness was enhanced to prioritize writes in case of any unforeseen issues.



<figure class="wp-block-image"><img decoding="async" src="https://lh4.googleusercontent.com/qBhmw-s3trruN-U2ubjvMlcjrw_HIML6Qw4Qg38sD2lwMYYO05xL0AuzsUdPCYORrtS_82ZGXaPZNz51jc4o2lCzfJwDmBCsyg8f_m66ZoLJeyoYo5Qn6JnwqEFkf48MPuBRXs-wg0ORMpqOsHRq9Q" alt=""/></figure>



Monitoring: ClickHouse server has embedded instruments for self-state monitoring. We scraped clickhouse metrics via prometheus, and then visualized and set alerts over it via grafana.



<figure class="wp-block-image"><img decoding="async" src="https://lh6.googleusercontent.com/N4RA73kX9W_NNR8E9IHqr2xohyJfBk2ZMQOY3pVYpoS00hO8P_lWBZpyJAnWP0aHSC6yAVLok4fTWkiAWbAqsVEYcwQFAInsEAej03l-OKRwb6TVZWRCvY8vJIAJqfthgzWc2rONcuqV-t_Ygig7nvw" alt=""/></figure>



We closely monitored various system health metrics such as CPU, Memory and Network, as well as specific Clickhouse metrics like delayed or rejected inserts, insert time, and query time. This helped us ensure that the system is both reliable and performing optimally.



In addition, we measured 95th percentile query run time at the table level, enabling us to evaluate performance and identify patterns in query behavior. Based on these insights, we could take steps such as adding indexes or sort keys, improving logging patterns, and more, to improve overall system performance.



<figure class="wp-block-image"><img decoding="async" src="https://lh3.googleusercontent.com/zDX4i55qcp3U8VdcPlK1cmbyYwvdNRDKld8jxDXq-j83Mk5u3GOk7IlmaONhh0lBpKFsb4gBXbpQeHE7gyJpeGEGx7wAbHkINgDHkl1RfsI5Kl0_dOgPxV_hTZobB97ux7p7xTHac3soEsTaNG6kVPg" alt=""/></figure>



Security: To ensure that only authorized individuals can access and query the logs, we employed features such as Google Authentication, Table-level access control, and Query Auditing. These measures helped enhance security and prevent unauthorized access to sensitive information.



<h2>Impact&nbsp;</h2>



Migrating to ClickHouse had a significant impact on our systems. Here&#8217;s how ClickHouse proved to be a game-changer:



Reliability: Real-time Data Ingestion – Ingestion lag of less than 5 seconds ensured real-time data availability, minimizing discrepancies and supporting timely decision-making.



Performance: Lightning-Fast Queries – P99 query time of 10 seconds accelerated data processing, delivering actionable insights promptly. For example, querying the largest index with the right filters for 7 days takes less than 20 seconds.



Customizations: Tailored Solutions – Our in-house solution allowed us to customize and adapt the platform to our specific requirements. This flexibility surpasses the limitations we previously faced, such as losing visibility due to paid access logs.



Auditing and Cost Savings – LogStore’s auditing capabilities and cost-effectiveness has the potential to save more than a million dollars per year compared to our previous setup, without compromising on system reliability and security.



What&#8217;s the compass pointing towards?



With new services emerging every day and evolving requirements, our project&#8217;s future plans include:



<ul style="text-align:justify"><li>Enhancing full-text search capabilities through the incorporation of inverted indexes.</li><li>Providing LogQL query support to power advanced use cases.</li><li>Adding support for similar logs and patterns to categorize errors and their frequency.</li><li>Building an automated system to fine-tune indexes and improve search experience using query patterns.</li><li>Migrating to <a href="https://opentelemetry.io/docs/specs/otel/logs">Open-Telemetry</a> logging standard</li></ul>



This blog was written by <a href="https://www.linkedin.com/in/anmol-virmani-0344028a/">Anmol Virmani</a> and <a href="https://www.linkedin.com/in/palashg/">Palash Goel</a> in collaboration with <a href="https://www.linkedin.com/in/bhavyarustgi10/">Bhavya Rustgi</a> and <a href="https://www.linkedin.com/in/sameer-saxena-87229283/">Sameer Saxena</a>, under the guidance of <a href="https://www.linkedin.com/in/rathorehimanshu/">Himanshu Rathore</a> and <a href="https://www.linkedin.com/in/ahmadumar/">Umar Ahmad</a>.

Building a cost-effective logging platform using Clickhouse for petabyte scale

Welcome to the third and final instalment of our three-part blog series! In parts<a href="https://www.zomato.com/blog/explained-how-we-handle-100million-daily-search-queries-pt1"> one</a> and <a href="https://www.zomato.com/blog/explained-how-we-handle-100million-daily-search-queries-pt2">two</a>, we unveiled the secrets behind Zomato&#8217;s powerful search engine, driven by Apache Solr, and shared our tactics for optimizing Solr’s performance. In this concluding chapter, we&#8217;ll take you behind the scenes of Zomato&#8217;s self-managed Solr clusters to show how we tackled the challenges arising from this rapid growth and transformed the scalability, reliability, and performance of Solr.



<h3 class="wp-block-heading">Hitting the infrastructural bottlenecks</h3>



Initially, we were running our Solr Clusters in Master-Slave Configuration. As Zomato&#8217;s growth continued at a breakneck speed, the index size started growing, and we started facing a new set of challenges:



<ul>
<li>To improve performance, we had to run the slave nodes on extremely large machines (120GB RAM) with a large JVM Heap (~100GB), but this resulted in poor tail latencies due to the longer&nbsp; <a href="https://www.oracle.com/technetwork/tutorials/tutorials-1873457.html">Stop The World</a> step in Garbage Collection.</li>



<li>Additionally, on every sync with the master, slaves had to open <a href="https://solr.apache.org/guide/8_7/index-replication.html#follower-replication">Searcher</a> on a large index, and <a href="https://solr.apache.org/guide/8_7/query-settings-in-solrconfig.html#caches">warming up</a> the caches took more time. This led to even worse tail latencies.</li>



<li>Furthermore, our business experiences daily traffic peaks during the typical lunch and dinner hours, necessitating rapid autoscaling. But it took ~10 minutes for a new replica to be ready for querying.</li>



<li>Lastly, manual handling was required for master failovers, which resulted in low data indexing reliability and fault tolerance.</li>
</ul>



<h3 class="wp-block-heading">Solving for scale: Moving to SolrCloud architecture</h3>



The logical solution to this problem was to divide the index into smaller segments and perform queries across multiple segments to address application requests. This approach led us to SolrCloud architecture, which allows managing large indexes in a more efficient and scalable manner, while still being self-managed. We observed several improvements:



<ul>
<li>We were able to run the cluster on smaller machines resulting in ~75% reduction in individual instances&#8217; CPU and Memory requirements.</li>



<li>We observed significant improvement in average and tail latencies.</li>



<li>Smaller index per shard also improved scale-up time by ~20%.</li>
</ul>



<h3 class="wp-block-heading">Improving the SolrCloud Performance</h3>



<h4 class="wp-block-heading">Custom sharding logic&nbsp;</h4>



To make sure that the migration to Solr Cloud is more performant, we developed a custom data-sharding strategy that aligns with Zomato’s hyperlocal business model, ensuring that data from a specific locality is stored within the same shard. As a result of this strategy, ~95% of queries were processed using a single shard, significantly reducing processing time and improving efficiency. The remaining queries were handled by multiple shards, providing an effective solution for handling a small subset of queries.&nbsp;



<h4 class="wp-block-heading">Turning off distributed search</h4>



At the time of splitting data into the shards, we ensured that we always knew which locality’s data was present in which shard. This made it possible to turn off the <a href="https://solr.apache.org/guide/8_4/distributed-requests.html">distributed search</a> by directing requests to the relevant shard and ensuring that the replica receiving the request would return the response by querying its local index only.



<pre class="wp-block-prismatic-blocks"><code class="language-">shards=shard1
shard.preference=local</code></pre>



By combining our understanding of customer behaviour with Solr&#8217;s internal request routing mechanisms, we successfully decreased the throughput on individual nodes by ~75%. Consequently, this reduction led to lower latency, decreased network traffic, and minimize costs, as the same traffic was served by a smaller number of nodes.



<h3 class="wp-block-heading">Solving for resiliency and availability</h3>



Following our successful migration from Solr Master-Slave to SolrCloud architecture, operations appeared to be running seamlessly. However, one day we started receiving alerts highlighting the absence of healthy hosts in one of the shards. This single-shard malfunction led to considerable downtime during peak business hours. At our operational scale, such downtimes result in substantial business losses and affect our customers, restaurant partners, and delivery partners alike.&nbsp;



<h4 class="wp-block-heading">The Downtime</h4>



Sharing a few details about our self-managed SolrCloud setup and the pre-conditions which triggered the downtime:



<ul>
<li>The Solr clusters were hosted on <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot instances</a> which means that running nodes can go down anytime.</li>



<li>We had configured to use only NRT (Near Real Time) replicas out of the <a href="https://solr.apache.org/guide/8_7/shards-and-indexing-data-in-solrcloud.html#types-of-replicas">three available options</a> i.e. NRT, TLOG and PULL replicas.</li>



<li>Whenever a new replica joins the cluster, a unique, monotonically increasing number (znode) is assigned by ZooKeeper. During the Leader Election process, the replica with the smallest value is given a chance to become the leader.</li>



<li>After a replica is allowed to become a leader, it verifies its index version proximity with the most recent version compared to the other replicas in the cluster. If its version is significantly outdated, it forfeits the chance of becoming the leader and goes into recovery, giving a chance to the next replica in the sequence.&nbsp;</li>



<li>After the leader is elected, any replica nodes significantly behind the new leader enter recovery mode to sync their index with the newly elected leader.</li>



<li>If any replica goes into recovery, it does not serve even the read traffic.</li>
</ul>



Now, what actually happened at the time of the incident:



<ul>
<li>The leader instance got reclaimed, and hence the cluster had no active leader.</li>



<li>A few minutes before the actual leader went down, a new replica joined the shard. Therefore, it had the most recent copy of the index.</li>



<li>Under these special circumstances, all the nodes in that shard went into recovery as their indexes were relatively older compared to the new replica and finally, the new replica was invited to become the leader.</li>



<li>The entire read traffic got redirected to the new leader. At the same time, all other replicas were trying to download GBs of data from the leader node. This led to a vicious cycle where the shard was not able to become healthy due to excess load on one node.</li>



<li>Finally, we had to cut all the traffic (single shard and cross-shard requests both) to allow it some time to become healthy.</li>
</ul>



<figure class="wp-block-image size-large is-resized"><img decoding="async" loading="lazy" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=640%2C277&#038;ssl=1" alt="Figure 1: Every engineer’s worst nightmare: No healthy hosts" class="wp-image-12872" width="640" height="277" title="" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=1024%2C443&amp;ssl=1 1024w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=300%2C130&amp;ssl=1 300w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?resize=768%2C333&amp;ssl=1 768w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?w=1418&amp;ssl=1 1418w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-1-1.png?w=1280&amp;ssl=1 1280w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



<h4 class="wp-block-heading">Switch to TLOG and Pull Replicas</h4>



At Zomato, we believe that “<a href="https://www.instagram.com/p/Cl08IrgvGTt/">Only the Paranoid Survive</a>”. After finishing the RCA of the incident, we decided that we need to solve a few things:



<ul>
<li>The Solr cluster must be highly available for processing the read queries by serving the read traffic during the Leader election or without any leader at all.</li>



<li>We should be able to quickly scale up the cluster without putting too much stress on the leader.</li>
</ul>



We decided to deep dive into Solr’s replication and leader election process. Each replica type has a different way of processing the updates from the leader, which decides the series of events during the Leader Election:



<ul>
<li>NRT:
<ol>
<li>Each replica receives the update from the leader and writes data to the transaction log (tlog).</li>



<li>Updates are periodically flushed from the tlog to the index and stored in the disc. This also means that the index files can be completely different (highlighted in Figure 2 as Idx1 and Idx2) in each replica.</li>



<li>NRT replica is eligible to become a leader during the leader election process.</li>



<li>In order to avoid any data loss, these replicas can go into recovery during the leader election.</li>
</ol>
</li>



<li>TLOG:
<ol>
<li>Each replica receives the update from the leader and writes data to the transaction log but does not write changes to the local index.</li>



<li>The index is periodically updated by fetching only the incremental changes from the leader. This also means that the index files will be the same as that of the leader.</li>



<li>TLOG replicas are also eligible to become a leader and just like NRT replicas, they can go into recovery during leader election to ensure data availability.</li>
</ol>
</li>



<li>PULL:
<ol>
<li>PULL replicas do not receive any updates directly from the leader and hence do not maintain a transaction log.</li>



<li>Similar to TLOG replicas, the index is periodically updated by fetching only the incremental changes from the leader.</li>



<li>PULL replicas are not eligible to become leader during the leader election.</li>
</ol>
</li>
</ul>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" width="471" height="521" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?resize=471%2C521&#038;ssl=1" alt="" class="wp-image-12874" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?w=471&amp;ssl=1 471w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image.png?resize=271%2C300&amp;ssl=1 271w" sizes="(max-width: 471px) 100vw, 471px" data-recalc-dims="1" /></figure>



Keeping all of this in mind, we began trying out multiple combinations of replicas and recreated multiple scenarios and we observed that:



<ul>
<li>NRT and PULL replicas [Not Recommended]
<ol>
<li>During leader election, only the NRT replicas go into recovery. The cluster is able to serve read requests by PULL replicas at all times (even without a leader).</li>



<li>No benefit of Near Real Time searching even after having NRT replicas as PULL replicas reflect changes only after the NRT leader commits changes to the index.&nbsp;</li>



<li>Since each replica has its own copy of index files, in case of Leader Election,&nbsp; the follower replicas need to download the entire index from the new leader and discard their old index files. This leads to a lot of disk IO and network bandwidth wastage and puts the leader under stress.</li>
</ol>
</li>



<li>TLOG replicas
<ol>
<li>As mentioned earlier, the cluster goes into recovery for a short period of time during the Leader election process. Hence, the cluster will not be available for reads for that period.</li>
</ol>
</li>



<li>NRT and TLOG replicas [Not Recommended]
<ol>
<li>Behave the same way as NRT and PULL replicas combination hence the same limitations and advantages.</li>
</ol>
</li>



<li>TLOG and PULL replicas
<ol>
<li>During the leader election, only the TLOG replicas go into recovery and need to download only the incremental changes from the new leader.</li>



<li>Similar to the NRT and PULL combination, the cluster is able to serve read requests by PULL replicas even without a leader or during the leader election process.</li>



<li>The downside is that the cluster loses the capability of Near Real-Time Search.</li>
</ol>
</li>
</ul>



Finally, we went ahead with the combination of TLOG and PULL replicas as we can bear some delay in reflecting the updates but we need the Solr cluster to be highly available for querying.



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" width="640" height="666" src="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?resize=640%2C666&#038;ssl=1" alt="" class="wp-image-12875" srcset="https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?w=736&amp;ssl=1 736w, https://i0.wp.com/zomatoblog.com/wp-content/uploads/2023/04/image-2.png?resize=288%2C300&amp;ssl=1 288w" sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" /></figure>



Extra Miles &#8211; seeding index data for faster scaling



Whenever a new node comes up, it has to download the entire index from the leader. This process has a serious limitation as it leads to bandwidth issues in case a lot of replicas need to be created at the same time.&nbsp;



We tried to look for solutions and sought help from the Solr community, but we eventually decided to build a custom solution:



<ol>
<li>Periodically take a snapshot of the index and save it in the AWS S3.</li>



<li>Whenever a new node comes up, it first downloads the latest snapshot from S3.</li>



<li>Configure the replica’s <a href="https://solr.apache.org/guide/8_7/datadir-and-directoryfactory-in-solrconfig.html">dataDir</a> to the path where the index is downloaded. This makes it possible to seed the index data for any new replica.</li>



<li>After this, the replica downloads only the incremental changes from the leader, allowing us to scale 100s of replicas at the same time.</li>
</ol>



This solved one more problem for us. Currently, Solr offers <a href="https://solr.apache.org/guide/8_7/making-and-restoring-backups.html">backup and restore</a> functionality for each collection, and it creates a single backup for the index data across all shards. Our solution enabled us to restore a specific shard by simply creating a TLOG replica using the seed data stored in S3. This effectively functions as a point-in-time recovery mechanism for individual shards, adding an extra layer of flexibility and resilience to our SolrCloud infrastructure.



<h3 class="wp-block-heading">In Conclusion</h3>



The spirit of being always paranoid, clubbed with a deeper understanding of our business and the underlying technology, continuously pushes us to build scalable, resilient and highly available solutions for customers. Our journey with Apache Solr has been an interesting one and full of learnings and we look forward to regularly sharing this with the community.



This blog was written by <a href="https://www.linkedin.com/in/nkmishra1997/">Nikhil Mishra</a> in collaboration with <a href="https://www.linkedin.com/in/saurav-singh-25317625/">Saurav Singh</a>.



<hr class="wp-block-separator has-css-opacity"/>



Sources:



<ul>
<li><a href="https://www.oracle.com/technetwork/tutorials/tutorials-1873457.html">Java Garbage Collection</a>, <a href="http://www.oracle.com" rel="nofollow">http://www.oracle.com</a></li>



<li><a href="https://solr.apache.org/guide/8_7/index-replication.html#follower-replication">Index Replication</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/query-settings-in-solrconfig.html#caches">Query Settings in SolrConfig</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_4/distributed-requests.html">Distributed Requests</a>, solr.apache.org</li>



<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot instances</a>, docs.aws.amazon.com</li>



<li><a href="https://solr.apache.org/guide/8_7/shards-and-indexing-data-in-solrcloud.html#types-of-replicas">Shards and Indexing Data in SolrCloud</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/docs/7_4_0/solr-core/org/apache/solr/cloud/LeaderElector.html#:~:text=Leader%20Election%20process,sequential%20node%2C%20etc.">Leader Election</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/datadir-and-directoryfactory-in-solrconfig.html">DataDir and DirectoryFactory in SolrConfig</a>, solr.apache.org</li>



<li><a href="https://solr.apache.org/guide/8_7/making-and-restoring-backups.html">Making and Restoring Backups</a>, solr.apache.org</li>
</ul>