10a: Introduction to Kappa Architecture

There seems to be a pattern emerging in the sort of code we're writing:

1. write all data to append-only logs
2. iterate over the data in each log to form an intermediary "view" of that log data (e.g. all chat messages)
3. refresh this "view" of the data whenever new log entries get added

The problem

If we want to build more advanced applications, such as games, map editors, and forums, we don't want to just print data as soon as you get it. We want to have 'views' of the data. You can think of these 'views' sort of like SQL tables.

Kappa Architecture

These are actually the foundational principles of "kappa architecture", a way of structuring data where:

• the data in the append-only log(s) is the true data source
• cheap "views" of the data are created and maintained, by iterating over existing log entries, and updating incrementally as new data gets added to the logs

This pattern turns out to be very powerful. Suddenly views are very cheap to produce: if you want to change the format or structure of a view, you can simply delete it and regenerate it from the log data.

You can create views for all sorts of things: sorting chat messages by timestamp, getting the current/latest name of a player, or efficient spatial indexes that let you query for all players / objects in an area.

Further reading: http://kappa-architecture.com

Exercise

kappa-core is a module that models a kappa architecture over multifeed and hypercore. kappa-core lets you read/write to append-only logs (just like multifeed), but also define views over that log data.

This is how you create a kappa-core instance and write to a new feed:

The kappa-core and multifeed databases are compatible, so we can use the same data we had before. Start a new file, kappa-chat.js, and enter the following:

const kappa = require('kappa-core')

const core = kappa('./multichat', {valueEncoding: 'json'})
core.writer('local', function (err, feed) {
    // `feed` is just a hypercore instance, just like the first few exercises.
})

Try to print the feed's key field. Use key.toString('hex') to see it as a hexidecimal string. Is it different or the same each time you run this?

This core.feed is essentially the same as multi.writer and initialises a local writable feed. You'll still have access to feed.key and feed.discoveryKey from here.

Once you solve this exercise continue to exercise 10b