Cutting down the carbon footprint of Contextual Intelligence with LLMs and Efficient Crawling

In our increasingly digital world, the demand for data has skyrocketed. Every day, billions of web pages are crawled, indexed, and analyzed to provide the contextual intelligence that powers search engines, recommendation systems, and countless AI-driven applications. While these technologies offer tremendous benefits, they also come with a significant environmental cost — CO2 emissions.

However, there’s good news. By optimizing the way we harness contextual intelligence, particularly through the use of Large Language Models (LLMs) and efficient crawling techniques, we can significantly reduce the carbon footprint of these operations. 

The Environmental Impact of Data Processing

Before diving into solutions, it’s essential to understand the scale of the problem. Data centers, which are the backbone of our digital infrastructure, consume vast amounts of energy. According to some estimates, data centers contribute to about 2% of global CO2 emissions, a figure comparable to the airline industry. A significant portion of this energy is used for powering servers that handle data processing tasks, including web crawling and running AI models like LLMs.

The Role of Large Language Models (LLMs)

Large Language Models, such as GPT-4, or NinaData’s Buying Intent AI, are at the heart of many modern AI systems. They have the ability to understand and generate human-like text, making them invaluable for tasks such as contextual analysis, natural language processing (NLP), and content generation. However, training and running these models is resource-intensive.

Optimizing LLMs for Efficiency

The key to reducing the environmental impact of LLMs lies in optimization. Here are several strategies:

1. Model Pruning: This involves trimming down a large model by removing redundant parameters that don’t significantly impact performance. Pruned models require less computational power to run, which translates into lower energy consumption.

2. Distillation: Model distillation is a process where a large model (teacher) is used to train a smaller, more efficient model (student). The smaller model retains much of the capability of the larger one but requires fewer resources to operate.

3. Efficient Inference Techniques: Implementing optimized algorithms for model inference can reduce the amount of computation required to generate outputs, thereby cutting down energy usage.

4. Hardware Acceleration: Leveraging specialized hardware like GPUs and TPUs, which are designed for parallel processing, can make LLM operations more energy-efficient.

The Importance of Efficient Crawling

Web crawling is the process of systematically browsing the internet to index and retrieve data. Traditional crawling methods can be wasteful, as they often involve downloading massive amounts of data without considering relevance or redundancy. This not only leads to excessive storage and processing needs but also increases CO2 emissions due to the energy required.

To optimize crawling one can practice:

1. Focused Crawling: Instead of indexing the entire web, focused crawling targets specific areas of interest, significantly reducing the amount of data that needs to be processed.

2. Incremental Crawling: This method only updates indexes with new or changed content, rather than re-crawling entire websites. This reduces unnecessary data processing.

3. Predictive Crawling: By using machine learning models to predict which pages are likely to be updated or are of high value, crawlers can prioritize these pages, further cutting down on energy use.

4. Caching and Deduplication: Storing frequently accessed data locally (caching) and eliminating duplicate data retrievals (deduplication) can reduce the load on servers and minimize redundant energy consumption.

The Synergy Between LLMs and Efficient Crawling

When LLMs and efficient crawling are combined, the potential for CO2 reduction is significant. For instance:

Contextual Prioritization: LLMs can be used to analyze and predict the relevance of web content before it is crawled, allowing crawlers to focus only on high-value data. This cuts down on unnecessary crawling and data processing.

Enhanced Content Understanding: By employing LLMs, crawlers can better understand the context of web content, enabling more precise and efficient data indexing.

Real-Time Adaptation: Efficient crawling can provide real-time updates to LLMs, ensuring they work with the most current and relevant data without redundant processing.

How we have reduced our carbon footprint at NinaData

Ninadata leverages the power of LLMs like GPT-3.5 Turbo and LLaMA 3 to deliver contextual intelligence in the Adtech space. However, our commitment goes beyond performance—we aim to reduce our environmental impact. By implementing advanced optimization techniques and efficient crawling strategies, Ninadata ensures that its operations are both effective and eco-friendly. This dual approach, utilizing cutting-edge models alongside sustainable practices, positions NinaData as a leader in sustainable AdTech innovation.

To begin with, our crawling is extremely focused, as it operates at URL level. There is no wastage of resources, compared to crawling everything from a site. And we certainly only crawl from sites that have relevant URLs.

So how do we know that the URLs we crawl are relevant? We leverage a number of programmatic supply side sources as a pre-vetting stage. This provides us with in the order of billions of URLs for classification every day, guaranteeing that whatever we crawl and analyze is fresh and being bidded on. 

And how is our crawling and analysis efficient? Our crawling is extremely efficient since we do it at URL level. We also only re-analyze changed content and utilize a layer which selects sites likely to have relevant content based on contextual analysis. If the content is unlikely to have changed since last visit according to a quick scan, we avoid fruitless re-analysis. We also have real-time adaptation through the use of caching and deduplication. If data indicates a site only changes its content slowly, we are less likely to check for the staleness of the data and vice versa.

We also only store the analysis of the URLs for both copyright/privacy protection and efficiency reasons. We utilize specialized, efficient hardware in all of our model building, running on AWS which can reduce carbon emissions by up to 93% compared to on-premise processing. Our constant improvements to our models along with new classifiers are also deployed in a manner where URLs get re-analyzed only on demand. Models are pruned and efficient inference techniques are a basic requirement.

The following table highlights recent benchmarks for various LLMs, showcasing their energy consumption and the effectiveness of different optimization techniques in reducing their carbon footprint. NinaData actively keeps abreast of the latest optimization techniques for the models it uses at any particular time.

Model	Hardware Used	Inference Energy Consumption	Optimization Technique	Energy Savings
LLaMA 65B	NVIDIA A100 GPUs	High	Power Limit Optimization (Zeus)	~15%
GPT-3 (13B)	NVIDIA V100 GPUs	Moderate	Pipeline Parallelism (Perseus)	~10-12%
GPT-4	NVIDIA A100 GPUs	High	Hybrid Parallelism	~18%
Mistral	NVIDIA A100 GPUs	Moderate	Model Pruning	~12%
Claude	NVIDIA H100 GPUs	Low	Efficient Sharding	~14%
LLaMA 7B	NVIDIA A100 GPUs	Low	Efficient Sharding	~8-10%

Here are the sources from which the information in the benchmark table was derived:

LLaMA 65B and GPT-4: Information on energy consumption and optimization techniques, particularly power limit optimization and hybrid parallelism, can be found in recent studies on large language models and their energy efficiency. A detailed discussion of these techniques is available on the ML.Energy leaderboard and the PyTorch blog on Deep Learning Energy Measurement and Optimization.

Mistral: Details on model pruning and its impact on energy savings for the Mistral model are covered in energy optimization reports, which can be explored further in the ML.Energy Colosseum and Zeus’ documentation.

Claude: Information about efficient sharding techniques and their application to the Claude model, along with energy savings, can be found in similar reports on AI energy optimization at Zeus and ML.Energy.

In summary, we have stopped doing any unnecessary crawling. We focus, at page level, only on the content that is highly likely to be relevant, and we do a final check before submitting a URL for analysis, which is then done in a self-adapting, CO2 emission minimizing manner. This results in significant reduction to our carbon footprint. 

Conclusion: A Greener Future for AI and Data Processing

As we continue to rely on AI and data-driven technologies, it’s crucial that we consider their environmental impact. By optimizing the use of LLMs and employing efficient crawling techniques, we can significantly reduce CO2 emissions associated with data processing. These advancements not only contribute to a greener planet but also make our digital systems more efficient and sustainable.

The journey toward a more eco-friendly digital future is ongoing, and every optimization counts. With a concerted effort, we can leverage the power of contextual intelligence while minimizing its environmental footprint, paving the way for a sustainable digital landscape.