Volumes of data can explode, and with that, so do costs. While proper observability becomes even more critical, very intelligent observability is what is needed, especially for managing highly distributed data and removing expensive data redundancy.  

During Grafana Labs’ annual user conference, GrafanaCon in Barcelona, Grafana discussed how it is relying on Kafka for its Loki logs project for observability needs at scale. It also says much work has been done to make queries faster and more precise.  

This work for the open source project, of course, reflects significant code commits that both Grafana and the user community have made. 

While ReveCom has not yet verified these results, Grafana says the new Loki version can deliver up to 20 times less data required for scanning and 10 times faster performance with aggregated queries. 

A significant architectural change is the introduction of Kafka as the new orchestration mechanism for distributed Loki systems. While small, single-machine instances of Loki will not need Kafka, larger distributed installations will require it to achieve greater scale –  and a 30% reduction in cost, Trevor Whitney, senior software engineer at Grafana Labs, said. 

“So there is, I think the trade-off will be worthwhile,” Whitney said. Grafana Labs has been relying on the new architecture with Kafka for its internal use operations environment and new production cells. Grafana Labs has also developed some tools to safely migrate to run this system, which are available in the open source repository.  

“If you are running a distributed system, then you will need Kafka, because that is going to be our new orchestration mechanism for a distributed system,” Whitney said. 

The transition towards using Kafka for distributed logs and analysis will support the integration of new database tools and statistical probability charts into highly distributed models.  

“It’s just a good idea to use Kafka for the distributed types of logs you’re looking for, the analysis of it, et cetera,” Jen Villa, Grafana Labs’ senior director of product, told me during the conference. “I was amazed by how far the project has come: You previously just had basic, rudimentary logs, and this is quite a jump.” 

For improved search optimization, Loki draws from the mathematical Bloom filter premise, by which a Bloom filter is used to rule out a subset of search results that are not part of the target data. This approach “trims the haystack” of massive log data, Villa said. “The result is that you will still have some false positives, but you certainly won’t have any false negatives,” Villa said. 

While the Loki project has undergone tremendous changes since its creation, cardinality changes have remained, due mainly to the aforementioned explosion in data to manage. Loki’s current storage format, called chunks, is primarily row-based and often contains 97% log lines, making it difficult to target specific metadata without downloading everything. To solve this, the new version of Loki relies on data objects, a new columnar storage format.  

Loki is now configured to store and process fewer, larger files and their timestamps and the metadata separately, which can be downloaded and processed independently. This is thanks to the capability to introduce data objects into a new columnar storage format specifically for logs.  

For faster query execution times, Loki’s log lines with data objects are now stored in different dimensions in their own columns. This allows for more selective reading of the columns. The “chunks” of pertinent data can be pinpointed more easily.  “The philosophy was, read everything and then filter in the new query engine, filter early, scan selectively, and we only process the dimensions needed for that specific query, which, of course, results in faster queries over less data,” Whitney said during his talk.  

What does this all mean for the end user besides the purported cost-savings and efficiency gains? There is certainly less often costly redundancy and duplication involved with log data, noted Oren Lion, director of software engineering, productivity engineering, for TeleTracking, a healthcare operations platform provider. This improvement translates into faster queries against Loki when visualizing logs. “Loki will run more efficiently and have increased performance, which could also translate into lower cost,” Lion said. “So, as a consumer, of course, I do like to see faster queries with Loki when visualizing logs and cost improvement with Loki.” 

Analyst Take 

Loki has demonstrated some real improvements. These are not just nice-to-have additional features, solving problems that people didn’t have. The cardinality improvements in this new version can offer a reduction in log data and accuracy, and enable pinpointing the logs and discerning the logs that are required to solve problems and for analysis. This is a huge plus for this new release. Additionally, by relying on Kafka, it now offers much better support for highly distributed systems at scale — a capability that has been highly requested by the open source user community. It remains true to its open source history, as it offers a very enticing way to use it for free.