Keeping Tabs On All My Neural Networks

Posted on 07/04/2017

When I’m out in public, I look at my watch a lot. It’s not because I’m nervous, or because I’m obsessed with the time. It’s because I’m checking on my neural networks.

At any given moment, I probably have four different machines crunching through ML tasks (e.g. training neural networks or downloading data). To keep tabs on all these machines, I use my own logging system called StatusHub. With StatusHub, I can use my phone, my watch, my tablet, or my laptop to see logs from every job across all of my machines. On my watch, I see a scrollable list that looks like this:

On my phone or laptop, I can see the same log through a web UI:

I can even view the log through the command-line, but I won’t bore you with a picture of that one.

Pushing log messages

You push log messages to a StatusHub server with the sh-log command. Without sh-log, I might train a neural network like so:

$ go run *.go
2017/07/03 18:32:57 done 4002029568 updates: cost=0.108497
2017/07/03 18:32:58 done 4002168832 updates: cost=0.114127
2017/07/03 18:32:59 done 4002308096 updates: cost=0.109726
...

As you can see, the program already produces log messages, but annoyingly they only go to standard error. To push the messages to StatusHub, we can simply use sh-log:

$ sh-log TrainEmbedding go run *.go
2017/07/03 18:32:57 done 4002029568 updates: cost=0.108497
2017/07/03 18:32:58 done 4002168832 updates: cost=0.114127
2017/07/03 18:32:59 done 4002308096 updates: cost=0.109726
...

In the above example, sh-log executes go run *.go and echoes the standard output/error to a StatusHub server (which is configured via environment variables). The first argument to sh-log is the service name, which helps to distinguish between different jobs in the log. If you look back at the screenshots from the beginning of this post, the service names should stand out right away.

The sh-log command also plays nicely with UNIX pipes. If you don’t provide a command for sh-log to run, it reads directly from standard input. For example, this is how I log information about my GPU:

$ nvidia-smi | head -n 9 | tail -n 1 | cut -b 3-77 | sed -e 's/\s\s*/ /g' | sh-log GPU
23% 35C P8 10W / 250W | 63MiB / 12186MiB | 0% Default

Viewing logs

The simplest way to view a log is via the StatusHub web UI or through the Android Wear application. However, StatusHub also ships with some commands for reading and manipulating logs.

To dump the log for a given service, there is the sh-dump command:

$ sh-dump tweetdump
...
-rw-r--r--  1 alex  staff  12141949085 Jul  3 19:09 tweets.csv
-rw-r--r--  1 alex  staff  12142001648 Jul  3 19:09 tweets.csv
-rw-r--r--  1 alex  staff  12142061169 Jul  3 19:10 tweets.csv
-rw-r--r--  1 alex  staff  12142116283 Jul  3 19:10 tweets.csv

You can also use the sh-stream command to view the output of a service in real time, for example:

$ sh-stream tweetdump
-rw-r--r--  1 alex  staff  12141949085 Jul  3 19:09 tweets.csv
-rw-r--r--  1 alex  staff  12142001648 Jul  3 19:09 tweets.csv
-rw-r--r--  1 alex  staff  12142061169 Jul  3 19:10 tweets.csv
-rw-r--r--  1 alex  staff  12142116283 Jul  3 19:10 tweets.csv
...

My favorite tool, though, is sh-avg. Using sh-avg, you can compute the averages of numerical fields over the last several log messages. For example, to average the results from the “TrainEmbedding” service:

$ sh-avg TrainEmbedding
size 10: cost=0.108642
size 20: cost=0.108811
size 50: cost=0.108578

You can also specify a particular average size (i.e. the number of log records to average):

$ sh-avg TrainEmbedding 32
size 32: cost=0.108722

If you want to be able to quickly see averages from your phone or smartwatch, you can setup a job to log the averages of another job:

$ while (true) do sh-log EmbedAvg sh-avg TrainEmbedding 30; sleep 30; done

As you can see, StatusHub allows you to be a command-line ninja with magical logging powers.

Going crazy

Once you have basic building blocks like sh-log and sh-stream, the possibilities are boundless.

With a pipe-based IRC client like ii, you can push chat logs to StatusHub in one terminal command. This makes it easy to keep tabs on IRC activity, even from devices without an IRC client (e.g. a smartwatch).

You could also pipe sh-stream into ii in order to send log messages to someone on IRC. This might not seem useful, but it actually could be. For example, say you want to be notified when a process finishes running. You could run this in one terminal:

$ ./some_long_task; sh-log Notify echo 'Task done!'

And then in some other terminal, perhaps on some other machine, run something like this:

$ sh-stream Notify | send-irc-messages

Using StatusHub yourself

The StatusHub repository has official installation instructions, but I thought I’d give a gist here as well. There are really three parts to a successful StatusHub installation:

  1. An sh-server process running on some internet-accessible machine. All of your devices should be able to connect to this machine over HTTP/HTTPS, either through a reverse proxy or via port forwarding.
  2. A set of jobs that do logging via the sh-log command. To have sh-log go to the correct server, you will need to set some environment variables.
  3. One or more devices from which you will consume logs. These devices simply need a browser, but you can also install the StatusHub commands and setup your environment variables accordingly.