When I’m out in public, I look at my watch a lot. It’s not because I’m nervous, or because I’m obsessed with the time. It’s because I’m checking on my neural networks.
At any given moment, I probably have four different machines crunching through ML tasks (e.g. training neural networks or downloading data). To keep tabs on all these machines, I use my own logging system called StatusHub. With StatusHub, I can use my phone, my watch, my tablet, or my laptop to see logs from every job across all of my machines. On my watch, I see a scrollable list that looks like this:
On my phone or laptop, I can see the same log through a web UI:
I can even view the log through the command-line, but I won’t bore you with a picture of that one.
Pushing log messages
You push log messages to a StatusHub server with the
sh-log command. Without
sh-log, I might train a neural network like so:
$ go run *.go 2017/07/03 18:32:57 done 4002029568 updates: cost=0.108497 2017/07/03 18:32:58 done 4002168832 updates: cost=0.114127 2017/07/03 18:32:59 done 4002308096 updates: cost=0.109726 ...
As you can see, the program already produces log messages, but annoyingly they only go to standard error. To push the messages to StatusHub, we can simply use
$ sh-log TrainEmbedding go run *.go 2017/07/03 18:32:57 done 4002029568 updates: cost=0.108497 2017/07/03 18:32:58 done 4002168832 updates: cost=0.114127 2017/07/03 18:32:59 done 4002308096 updates: cost=0.109726 ...
In the above example,
go run *.go and echoes the standard output/error to a StatusHub server (which is configured via environment variables). The first argument to
sh-log is the service name, which helps to distinguish between different jobs in the log. If you look back at the screenshots from the beginning of this post, the service names should stand out right away.
sh-log command also plays nicely with UNIX pipes. If you don’t provide a command for
sh-log to run, it reads directly from standard input. For example, this is how I log information about my GPU:
$ nvidia-smi | head -n 9 | tail -n 1 | cut -b 3-77 | sed -e 's/\s\s*/ /g' | sh-log GPU 23% 35C P8 10W / 250W | 63MiB / 12186MiB | 0% Default
The simplest way to view a log is via the StatusHub web UI or through the Android Wear application. However, StatusHub also ships with some commands for reading and manipulating logs.
To dump the log for a given service, there is the
$ sh-dump tweetdump ... -rw-r--r-- 1 alex staff 12141949085 Jul 3 19:09 tweets.csv -rw-r--r-- 1 alex staff 12142001648 Jul 3 19:09 tweets.csv -rw-r--r-- 1 alex staff 12142061169 Jul 3 19:10 tweets.csv -rw-r--r-- 1 alex staff 12142116283 Jul 3 19:10 tweets.csv
You can also use the
sh-stream command to view the output of a service in real time, for example:
$ sh-stream tweetdump -rw-r--r-- 1 alex staff 12141949085 Jul 3 19:09 tweets.csv -rw-r--r-- 1 alex staff 12142001648 Jul 3 19:09 tweets.csv -rw-r--r-- 1 alex staff 12142061169 Jul 3 19:10 tweets.csv -rw-r--r-- 1 alex staff 12142116283 Jul 3 19:10 tweets.csv ...
My favorite tool, though, is
sh-avg, you can compute the averages of numerical fields over the last several log messages. For example, to average the results from the “TrainEmbedding” service:
$ sh-avg TrainEmbedding size 10: cost=0.108642 size 20: cost=0.108811 size 50: cost=0.108578
You can also specify a particular average size (i.e. the number of log records to average):
$ sh-avg TrainEmbedding 32 size 32: cost=0.108722
If you want to be able to quickly see averages from your phone or smartwatch, you can setup a job to log the averages of another job:
$ while (true) do sh-log EmbedAvg sh-avg TrainEmbedding 30; sleep 30; done
As you can see, StatusHub allows you to be a command-line ninja with magical logging powers.
Once you have basic building blocks like
sh-stream, the possibilities are boundless.
With a pipe-based IRC client like ii, you can push chat logs to StatusHub in one terminal command. This makes it easy to keep tabs on IRC activity, even from devices without an IRC client (e.g. a smartwatch).
You could also pipe
ii in order to send log messages to someone on IRC. This might not seem useful, but it actually could be. For example, say you want to be notified when a process finishes running. You could run this in one terminal:
$ ./some_long_task; sh-log Notify echo 'Task done!'
And then in some other terminal, perhaps on some other machine, run something like this:
$ sh-stream Notify | send-irc-messages
Using StatusHub yourself
The StatusHub repository has official installation instructions, but I thought I’d give a gist here as well. There are really three parts to a successful StatusHub installation:
sh-serverprocess running on some internet-accessible machine. All of your devices should be able to connect to this machine over HTTP/HTTPS, either through a reverse proxy or via port forwarding.
- A set of jobs that do logging via the
sh-logcommand. To have
sh-loggo to the correct server, you will need to set some environment variables.
- One or more devices from which you will consume logs. These devices simply need a browser, but you can also install the StatusHub commands and setup your environment variables accordingly.