Ledgers, here in the broadest sense, are historical, append-only lists of things. The tools and modules in this collection allow you to track the state of a ledger of arbitrary size by computing the ledger's hash using a cryptographic accumulator that links the ledger's hash at an earlier state to that of later one. Additionally, this accumulator allows you to generate (publish) compact, opaque, cryptographic proofs asserting that a ledger's hash at one state is derived from its hash at an earlier state (when the list had fewer objects). Underlying this capability is the skip ledger data structure (the repo's namesake).
A collection of sub-modules build on the capabilities of the skip ledger data structure. In particular, the base module offers a compact, flexible packaging (file format) for differentially disclosing and proving the contents of any parts (single or multiple items) from the ledger. The format is designed in a way to be both self-verifying and to allow you to merge information from multiple files into a single file-so long as the files share the same lineage (i.e. their hashes prove that they are from the same historical ledger).
If a Merkle tree is a type of binary tree, then a skip ledger is a type of skip list. With a skip ledger the hash of each of row is dependent both on the hash of the previous rows and the input hash (the hash of the source object). When viewed back-to-front, that is from a given row to row number 1, the hash of any given row in a skip ledger can be seen as the root of a binary hash tree structure. So the way this hashing is organized, the hash of row number n is both the hash of the ledger's state when it had (or has) exactly n rows, and the hash of the state of a crytographic accumulator (analogous to the root of a merkle tree).
At its bottom abstraction a skip ledger is just a list of append-only SHA-256 hashes: the individual hashes may represent any objects: rows from a view in a relational database, or maybe the hashes of successive time-slices from a video-feed-it doesn't matter. Skip ledgers can be annotated with crumtrails (witness records) so that the minimum age of their rows can be established.
The Morsel file format (.mrsl) is also defined here. The format is designed for packaging
compact proofs that a given
source object's hash at a given row number, for example, indeed matches that recorded in
the skip ledger. The format makes few assumptions about what the source objects should
be (they could be just a blob of bytes), however it does introduce a simple object model designed
to aid the hashing of structured data (eg from SQL), or semi-structured data
(eg tokenized text). It models each source row much like SQL, only with fewer types. The source hashing
model allows for individual column values to be redacted. It also supports salting each
individual cell value (the default) in order resist rainbow attacks and frequency analysis of
the hash-obfuscated data.
This module uses a relational (SQL) database as the backing storage for a skip ledger and its crumtrail annotations. It also assumes the source rows (representing the ledger being tracked) come from a table (or more likely a joined view) in a relational database. Hashes for column values are computed on the fly depending on their SQL type. The column values in each row is defined by an arbitrary SQL query that takes a single parameter, the row number, as its argument. The query typically involves using the SQL ROW_NUMBER() function.
Morsel files allow auxilliary assets to be packaged alongside the data in the rows they prove. The first such asset was introduced with the optional JSON meta file describing what the ledger and its columns are about. This module defines a JSON template for generating PDF reports from the data inside a morsel. This JSON, along with any other assets it uses (e.g. image logos) are packaged in the "assets" section of the morsel.
This is a command line tool for tracking and managing an SQL-backed ledger. Each ledger instance is defined by a configuration file. You can periodically update the state of the ledger (as new rows are added as a result of some business activity), verify old rows haven't changed, as well as generate / publish morsel files from the ledger. The ledger's configuration file may optionally reference either (or both) the location of the ledger's meta file and the ledger's PDF report template: if either is present (and well formed), then these are also embedded in the morsel files sldg generates.
Since morsel files (.mrsl) are packaged in binary format, we need a tool for reading
and manipulating them. The command line tool mrsl verifies, displays and extracts information
from morsels: for example, the contents of rows at select rows numbers from the ledger, their age, and so on.
It also knows how to compare 2 .mrsl files and determine whether they come from ledgers
that have the same lineage. When this is so, you can use mrsl to merge 2 or more .mrsl
files into a new one; conversely, you can also remove or redact information from an existing
.mrsl file.
If a morsel file contains an embedded report template, you can generate PDF files from it. mrsl can display what arguments, if any, the template takes. (The arguments determine which source rows are included in the PDF.)
Example ledger configuration files, morsel file output, and in turn, PDF generated from a morsel are provided here.
© 2020-2022 crums.io