Integrating Apache Traffic Server

Overview

We will configure Apache Traffic Server (ATS) to use Backtrace's Invoker as a crash log helper. The workflow upon ATS crash will be:

On startup, ATS will launch invoker, which will monitor the traffic_server process.
On traffic_server crash, invoker will launch ptrace via a script which provides metadata.
This script will, in turn, submit the crash file that ptrace generates to the coroner object store.

Prerequisites

Apache Traffic Server (tested with version 6.2.0)
Coroner Server running and accessible from the ATS machine. (See Coroner Server installation)
Coroner Client package on the ATS machine. (See Coroner Client Installation)
Backtrace Ptrace package installed on the ATS machine. (See Ptrace Installation)
Backtrace Invoker package installed on the ATS machine:$ sudo apt-get install backtrace-invoker
(Optional but recommended) - Install the morgue command-line queryingtool: (See: Morgue Installation)

Set Up Helper Scripts

invoker.sh

#!/bin/sh
exec /opt/backtrace/bin/invoker -t "/home/ats/backtrace.sh %p" $@

backtrace.sh

This will call ptrace with any metadata you wish to gather, then submits it to coroner. Here, we're extracting the version number via traffic_ctl, but feel free to add any additional metadata that you need.

#!/bin/sh

## Change the following as needed
PROJECT=ats
TOKEN=ats
DM=/export/ats                                                           
CF=/etc/coronerd/coroner.cf

PATH=/opt/backtrace/bin:/usr/bin:/bin:$PATH

# Add commands to gather any metadata that you wish to pass to the ptrace invocation
VERSION=`/opt/ts/bin/traffic_ctl metric get proxy.node.version.manager.short|cut -d ' ' -f 2`

mkdir -p ${DM}/                                                                 
ptrace --kv="version:$VERSION" $1 -O ${DM}/ats
if test "$?" == "0"; then                                                       
   coroner -c $CF put -u ${PROJECT} ${TOKEN} ${DM}/*.btt                                    
fi

Make sure that your traffic_server processes can execute these scripts, and has permission to write to the destination folder in backtrace.sh (referred to by the variable DM)

$ chmod +x invoker.sh
$ chmod +x backtrace.sh

Configure ATS Crash Log Helper

Add or edit the following line in <ATS root>/etc/trafficserver/records.config:

CONFIG proxy.config.crash_log_helper STRING /home/ats/invoker.sh

Verify invoker Launch

$ sudo <ATS root>/bin/trafficserver restart

$ ps aux | grep invoker
nobody   11616  0.0  0.1   4336  1396 ?        T    16:43   0:00 /opt/backtrace/bin/invoker -t /home/ats/backtrace.sh %p --syslog --wait --host x86_64-unknown-linux-gnu

Test

Let's send a SIGABRT to traffic server...

$ sudo kill -SIGABRT <PID of traffic_server>

...and verify via the morgue tool that the number of crashes for that project has increased.


>morgue list myproj
...
<omitted>
...
     Occurrences: 1 (100.00%)

Troubleshooting

Launch backtrace.sh manually

Try launching backtrace.sh manually to make sure there are no errors:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If there are any errors, continue with the troubleshooting steps below to verify the correct installation of Backtrace components and proper permissions.

Verify ptrace installation

Running /opt/backtrace/bin/ptrace <pid> on a process you own should generate a .btt file:

$ ps
  PID TTY          TIME CMD
 2291 pts/7    00:00:00 bash
11709 pts/7    00:00:00 ps

$ /opt/backtrace/bin/ptrace 2291
/home/test/bash.2291.1461078102.btt

$ ls *.btt
bash.2291.1461078102.btt

If not, verify your ptrace installation: Ptrace Installation

Verify coroner installation

Make sure that you are able to send snapshots to the server via coroner:

$ coroner -c put project1 project1_token bash.2291.1461078102.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If not, verify your coroner client installation: Coroner Client Installation

Launch backtrace.sh manually

Try launching backtrace.sh manually to make sure there are no errors:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

Make sure ATS has permission to run invoker.sh

$ chmod +x /path/to/invoker.sh

Make sure backtrace.sh is being invoked and generating the .btt file

If you comment out the last 3 lines of backtrace.sh, you can verify that the invoker is running the script and generating the .btt file in the desired location.