Get started with Parseable, an open source log storage and observability platform

Log data is one of the fastest-growing segments across data storage. It's also one of the most complicated spaces. There are several products and solutions with overlapping use cases and confusing marketing.

This article looks at Parseable, a log storage and observability platform. Parseable is geared towards a better user experience, with an easy-to-deploy and use interface and a simple, cloud-native architecture. I'll also show how to set up Parseable with FluentBit to store logs.

What is Parseable?

Parseable is a free and open source log storage and observability platform. Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.

Some core concepts for building Parseable are:

Indexing free

Traditionally, text search engines like Elastic have doubled as log storage platforms. This makes sense because log data must be searched to be really useful. But indexing comes at a high cost. It is CPU intensive and slows down ingestion. Also, the index data generated by these systems are of the same order of storage as the raw log data. This doubles the storage cost and increases complexity. Parseable changes this. With columnar data formats (parquet), it is possible to compress and query the log data efficiently without indexing it.

Ownership of both data and content

With parquet as the storage format and stored in standard object storage buckets, users own their log data and have complete access to the actual content. This means users can easily use analysis tools like Spark, Presto, or TensorFlow to extract more value from the data. This feature is extremely powerful, opening up new avenues of data analysis.

Fluid schema

Logs are generally semi-structured by nature, and they're ever-evolving. For example, a developer may start with a log schema like this:

{
  "Status": "Ready",
  "Application": "Example"
}

But as more information is collected, the log schema may evolve to:

{
  "Status": "Ready",
  "Application": {
    "UserID": "3187F492-8449-4486-A2A0-015AE34F1D09",
    "Name": "Example"
  }
}

Engineering and SRE teams regularly face schema-related issues. Parseable solves this with a fluid schema approach that lets users change the schema on the fly.

Simple ingestion

The current ingestion mechanism for logging platforms is quite convoluted, with several available protocols and connectors. Parseable aims to make log ingestion as easy as possible. The result is you can use HTTP POST calls to send logs to Parseable. No complicated SDKs are required.

What if you want to use a logging agent like FluentBit, Vector, LogStash, or others? Almost all the major log collectors support HTTP, so Parseable is already compatible with your favorite log collection agent.

Get started

You can use a Docker image to try out Parseable. This image shows Parseable in demo mode, using publicly-accessible object storage.

$ cat << EOF > parseable-env
P_S3_URL=https://minio.parseable.io:9000
P_S3_ACCESS_KEY=minioadmin
P_S3_SECRET_KEY=minioadmin
P_S3_REGION=us-east-1
P_S3_BUCKET=parseable
P_LOCAL_STORAGE=/data
P_USERNAME=parseable
P_PASSWORD=parseable
EOF

$ mkdir -p /tmp/data

$ docker run \
  -p 8000:8000 \
  --env-file parseable-env \
  -v /tmp/data:/data \
  parseable/parseable:latest

Log in to the Parseable UI using the credentials passed here (that's parseable and parseable.) The demo already contains some data because Parseable is pointing to the publicly-open bucket.

Make sure to change the bucket and credentials to your object storage instance before sending any data to Parseable.

Refer to the documentation to understand how Parseable works and how to ingest logs.

Set up FluentBit to send logs to Parseable

You can use a Docker compose file to configure both Parseable and FluentBit, making it easier to set up and tear down as needed.

First, save this file as fluent-bit.conf in a directory. The file is the configuration used to send data to Parseable.

[SERVICE]
  Flush 5
  Daemon Off
  Log_Level debug

[INPUT]
  Name dummy
  Tag dummy

[OUTPUT]
  Name http
  Match *
  Host parseable
  http_User parseable
  http_Passwd parseable
  format json
  Port 8000
  Header X-P-META-meta1 value1
  Header X-P-TAG-tag1 value1
  URI /api/v1/logstream/fluentbit1
  Json_date_key timestamp
  Json_date_format iso8601

Now save the following file as docker-compose.yaml in the same directory as above:

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - parseable

  parseable:
    image: parseable/parseable
    ports:
      - "8000:8000"
    environment:
      - P_S3_URL=https://minio.parseable.io:9000
      - P_S3_ACCESS_KEY=minioadmin
      - P_S3_SECRET_KEY=minioadmin
      - P_S3_REGION=us-east-1
      - P_S3_BUCKET=parseable
      - P_LOCAL_STORAGE=/tmp/data
      - P_USERNAME=parseable
      - P_PASSWORD=parseable

The docker-compose.yaml refers to the fluent-bit.conf file and passes it to the FluentBit container as the configuration file.

Parseable is deployed with the default configuration (as in the above Docker setup). You can observe the data FluentBit container sent to Parseable in the Parseable Console running at http://localhost:8000.

Wrap up

In this article, you've taken your first look at Parseable, the open source log storage and analysis platform built in Rust. A single Docker command gets you started with Parseable so you can experience the UI and establish FluentBit as a data source. If you think this looks too easy, then it's probably time to try Parseable!