Introduction

Most database engines that natively use SQL follow the traditional relational database design that requires each table to have a fixed schema. This is where Raijin takes a radically different approach. Although you can create tables with a schema that defines a specific set of columns, and their associated data types, this is not strictly enforced. Essentially, Raijin provides the flexibility to follow the proposed schema if it fits, but allows each new record to include new columns—​or fields as we often refer to them—​as needed.

Likewise, if some of the previously defined columns in the schema are suddenly not present in a new batch of logs that is being ingested, Raijin will refrain from including them. The other fields that are actually present will be the only ones inserted, thus reducing overheard and storage requirements.

Another feature of this flexibility is how Raijin reacts to a column that is defined as a single data type, but is receiving data of other data types. If for some reason your log source delivers a STRING value, or perhaps a TIMESTAMP value, for a column that has been defined as INT, no problem. That field will be considered to have a VARIANT data type.

With Raijin, there is only overhead for columns—​or fields—​that are actually used. Exceptions to the desired schema do not cause ingestion failures. Nor does any data throw an error or get discarded just because your log source recorded it as a different data type than was expected.

Given the time series nature of event logs and their structure that often varies, Raijin’s features make it well-suited for consuming log data. However, this is not a coincidence. The impetus for creating Raijin was the need for a high-performance database engine tailor-made for log data that would also integrate easily and efficiently with NXLog agents.