Integrating Apache Traffic Server

Integrating Apache Traffic Server

Overview

We will configure Apache Traffic Server (ATS) to use Backtrace's Invoker as a crash log helper. The workflow upon ATS crash will be:

1. On startup, ATS will launch invoker, which will monitor the traffic_server process.
2. On traffic_server crash, invoker will launch ptrace via a script which provides metadata.
3. This script will, in turn, submit the crash file that ptrace generates to the coroner object store.

Prerequisites

• Apache Traffic Server (tested with version 6.2.0)
• Coroner Server running and accessible from the ATS machine. (See Coroner Server installation)
• Coroner Client package on the ATS machine. (See Coroner Client Installation)
• Backtrace Ptrace package installed on the ATS machine. (See Ptrace Installation)
• Backtrace Invoker package installed on the ATS machine:$ sudo apt-get install backtrace-invoker
• (Optional but recommended) - Install the morgue command-line queryingtool:  (See: Morgue Installation)

Set Up Helper Scripts

invoker.sh

#!/bin/sh
exec /opt/backtrace/bin/invoker -t "/home/ats/backtrace.sh %p" $@

backtrace.sh

This will call ptrace with any metadata you wish to gather, then submits it to coroner. Here, we're extracting the version number via traffic_ctl, but feel free to add any additional metadata that you need.

#!/bin/sh

## Change the following as needed
PROJECT=ats
TOKEN=ats
DM=/export/ats                                                           
CF=/etc/coronerd/coroner.cf

PATH=/opt/backtrace/bin:/usr/bin:/bin:$PATH

# Add commands to gather any metadata that you wish to pass to the ptrace invocation
VERSION=`/opt/ts/bin/traffic_ctl metric get proxy.node.version.manager.short|cut -d ' ' -f 2`

mkdir -p ${DM}/                                                                 
ptrace --kv="version:$VERSION" $1 -O ${DM}/ats
if test "$?" == "0"; then                                                       
   coroner -c $CF put -u ${PROJECT} ${TOKEN} ${DM}/*.btt                                    
fi 

Make sure that your traffic_server processes can execute these scripts, and has permission to write to the destination folder in backtrace.sh (referred to by the variable DM)

$ chmod +x invoker.sh
$ chmod +x backtrace.sh

Configure ATS Crash Log Helper

Add or edit the following line in <ATS root>/etc/trafficserver/records.config:

CONFIG proxy.config.crash_log_helper STRING /home/ats/invoker.sh

Verify invoker Launch

$ sudo <ATS root>/bin/trafficserver restart

$ ps aux | grep invoker
nobody   11616  0.0  0.1   4336  1396 ?        T    16:43   0:00 /opt/backtrace/bin/invoker -t /home/ats/backtrace.sh %p --syslog --wait --host x86_64-unknown-linux-gnu

Test

Let's send a SIGABRT to traffic server...

$ sudo kill -SIGABRT <PID of traffic_server>

...and verify via the morgue tool that the number of crashes for that project has increased.

>morgue list myproj
...
<omitted>
...
     Occurrences: 1 (100.00%)

Troubleshooting

Launch backtrace.sh manually

Try launching backtrace.sh manually to make sure there are no errors:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If there are any errors, continue with the troubleshooting steps below to verify the correct installation of Backtrace components and proper permissions.

Verify ptrace installation

Running /opt/backtrace/bin/ptrace <pid> on a process you own should generate a .btt file:

$ ps
  PID TTY          TIME CMD
 2291 pts/7    00:00:00 bash
11709 pts/7    00:00:00 ps

$ /opt/backtrace/bin/ptrace 2291
/home/test/bash.2291.1461078102.btt

$ ls *.btt
bash.2291.1461078102.btt

If not, verify your ptrace installation: Ptrace Installation

Verify coroner installation

Make sure that you are able to send snapshots to the server via coroner:

$ coroner -c put project1 project1_token bash.2291.1461078102.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If not, verify your coroner client installation: Coroner Client Installation

Launch backtrace.sh manually

Try launching backtrace.sh manually to make sure there are no errors:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

Make sure ATS has permission to run invoker.sh

$ chmod +x /path/to/invoker.sh

Make sure backtrace.sh is being invoked and generating the .btt file

If you comment out the last 3 lines of backtrace.sh, you can verify that the invoker is running the script and generating the .btt file in the desired location.