Introduction

Raijin is a columnar database that supports semi-structured data of varied forms, like logs. With its hybrid columnar data storage and block-level data compression it is designed for high ingestion rates and low data storage requirements. Perhaps the most exciting characteristic of the Raijin database engine is that it achieves its query performance without indexes.

Flexible schema

Most database engines that natively use SQL follow the traditional relational database design that requires each table to have a fixed schema. This is where Raijin takes a radically different approach. Although you can create tables with a schema that defines a specific set of columns, and their associated data types, this is not strictly enforced. Essentially, Raijin provides the flexibility to follow the proposed schema if it fits, but allows each new record to include new columns—​or fields as we often refer to them—​as needed.

Likewise, if some of the previously defined columns in the schema are suddenly not present in a new batch of logs that is being ingested, Raijin will refrain from including them. The other fields that are actually present will be the only ones inserted, thus reducing overheard and storage requirements.

Another feature of this flexibility is how Raijin reacts to a column that is defined as a single data type, but is receiving data of other data types. If for some reason your log source delivers a STRING value, or perhaps a DATETIME value, for a column that has been defined as INT, no problem. That field will be considered to have a VARIANT data type.

With Raijin, there is only overhead for columns—​or fields—​that are actually used. Exceptions to the desired schema do not cause ingestion failures. Nor does any data throw an error or get discarded just because your log source recorded it as a different data type than was expected.

Other features

Table partitioning

Create partitions to enhance your database performance

Public API

Accepts JSON data via an HTTP POST for ingesting log data

Full-featured SQL

Even includes an option to insert JSON data: INSERT INTO events {"hostname":"srv-04","eventtime":"2022-06-24 12:20:12Z"};

Plugins

Integrate with Apache Superset and Grafana

Especially suited for log data

Given the time series nature of event logs and their structure that often varies, Raijin’s features make it well-suited for consuming log data. However, this is not a coincidence. The impetus for creating Raijin was the need for a high-performance database engine tailor-made for log data that would also integrate easily and efficiently with NXLog agents.