JC

How does Waka work? - Part 1

Matt keeps asking me how Waka works, so I’ve decided to write a guide. In Part 1, we go over how the server does things. I’ll try make an effort to keep this guide updated - contact me if it’s not.

The Server Stack #

Before you even try to start Waka, you’ll need:

Handy Tools #

Third Party API’s #

Configuration #

/config.js #

This is app wide config. Defaults should be fine, but change the public & private ports if required. Make sure you don’t expose the private port to the web! You can also disable auto updates / worker management of cities by adjusting the autoupdate entry.

/server-master/db/config-master.js, config-slave.js #

These are where the app looks for database connection strings. Make your SQL Server install allows SQL Server Authentication, and TCP/IP Network Protocol is enabled. The user set in config-master.js must be allowed to Create Databases, and the user set in config-slave.js must be allowed to access and modify the created database. You will need an already created database in config-master.js, but this is not needed for the slave.

By default, workers in the workers table use the dbconfig of slave, but you can set this to whatever if you want to use config-whatever.js (if you need to distribute over multiple instances of SQL Server).

Environmental Variables #

Starting #

Yay! You’ve configured Waka. Run node index.js to start, and it’ll create the database and start on the port you specified.

If you don’t want it to start downloading data from Auckland automatically, make sure you disable the auto updates in /config.js.

Architecture #

Waka has a master process that manages a collection of worker processes, each encapsulating a particular region & version and providing the API. Each worker process uses its own database which keeps things smaller, faster, and is easier to delete when the data has expired. There’s mapping functionality which sends all requests to a particular region to a particular version. If you’re using the auto importers you shouldn’t have to worry about the mappings, but it is possible to override.

Worker spawning is done through node’s child_process.fork, and all communication thereafter is done through a standard REST API - using both the public and private methods on both the master and the worker. Requests are proxied through to the mapped worker from the master. Check out all the router files for further details.

Static rendering and hosting of assets is done through yet another process. For static rendering, it uses the standard public REST API, and transforms the JSON to HTML. Files in /dist are all hosted under / when a route cannot be found. If a file is not found in /dist, the static server returns a default or 404 page, and lets the client side JavaScript take over the rest.

Server-Master Private API #

If you’re not using an auto importer or just need to emergency fix things, you can do so with the private API. It’s not secured, so you have to make sure the port isn’t exposed.

For most of these requests, just pop a JSON object with the prefix (region) and version in the request body.

Update: You can now just head to http://localhost:8001 and there’s a nice UI for running these request.

Server-Worker & Server-Static Public API #

This is the public API. You can generate them with npm run document or just head to https://waka.app/docs/index.html.

SQL Migrations #

Because we create the database from scratch every time a new worker is created, we don’t have to worry about migrations! Simply delete the worker and try again if you’re having any issues. The SQL files which create the tables and stored procedures are located in /server-worker/db/procs

I’m not sure how to do the migrations for the master database yet, but the scripts that create it on first run are in /server-master/db/procs

Tests #

No unit tests yet 😢. There’s a couple of tests in the Auckland auto importer that does a quick check before changing the live mapping, but that’s it. There’s definitely room for improvement here.

Other Things #

Here’s some things you might want to do when modifying or using Waka as a dev.

Adding a New City #

Now what? #

Hopefully you download the code and have a play - fix a few issues, or even just file some 😃 https://github.com/consindo/waka/issues

In Part 2, we’ll go over the client-side development.