Integrating Apache Traffic Server
Overview
We will configure Apache Traffic Server (ATS) to use Backtrace's Invoker as a crash log helper. The workflow upon ATS crash will be:
On startup, ATS will launch invoker, which will monitor the traffic_server process.
On traffic_server crash, invoker will launch ptrace via a script which provides metadata.
This script will, in turn, submit the crash file that ptrace generates to the coroner object store.
Prerequisites
Apache Traffic Server (tested with version 6.2.0)
Coroner Server running and accessible from the ATS machine. (See Coroner Server installation)
Coroner Client package on the ATS machine. (See Coroner Client Installation)
Backtrace Ptrace package installed on the ATS machine. (See Ptrace Installation)
Backtrace Invoker package installed on the ATS machine:$ sudo apt-get install backtrace-invoker
(Optional but recommended) - Install the morgue command-line queryingtool: (See: Morgue Installation)
Set Up Helper Scripts
invoker.sh
#!/bin/sh
exec /opt/backtrace/bin/invoker -t "/home/ats/backtrace.sh %p" $@
backtrace.sh
This will call ptrace
with any metadata you wish to gather, then submits it to coroner
. Here, we're extracting the version number via traffic_ctl
, but feel free to add any additional metadata that you need.
#!/bin/sh
## Change the following as needed
PROJECT=ats
TOKEN=ats
DM=/export/ats
CF=/etc/coronerd/coroner.cf
PATH=/opt/backtrace/bin:/usr/bin:/bin:$PATH
# Add commands to gather any metadata that you wish to pass to the ptrace invocation
VERSION=`/opt/ts/bin/traffic_ctl metric get proxy.node.version.manager.short|cut -d ' ' -f 2`
mkdir -p ${DM}/
ptrace --kv="version:$VERSION" $1 -O ${DM}/ats
if test "$?" == "0"; then
coroner -c $CF put -u ${PROJECT} ${TOKEN} ${DM}/*.btt
fi
Make sure that your traffic_server
processes can execute these scripts, and has permission to write to the destination folder in backtrace.sh
(referred to by the variable DM
)
$ chmod +x invoker.sh
$ chmod +x backtrace.sh
Configure ATS Crash Log Helper
Add or edit the following line in <ATS root>/etc/trafficserver/records.config
:
CONFIG proxy.config.crash_log_helper STRING /home/ats/invoker.sh
Verify invoker Launch
$ sudo <ATS root>/bin/trafficserver restart
$ ps aux | grep invoker
nobody 11616 0.0 0.1 4336 1396 ? T 16:43 0:00 /opt/backtrace/bin/invoker -t /home/ats/backtrace.sh %p --syslog --wait --host x86_64-unknown-linux-gnu
Test
Let's send a SIGABRT to traffic server...
$ sudo kill -SIGABRT <PID of traffic_server>
...and verify via the morgue tool that the number of crashes for that project has increased.
>morgue list myproj
...
<omitted>
...
Occurrences: 1 (100.00%)
Troubleshooting
Launch backtrace.sh manually
Try launching backtrace.sh
manually to make sure there are no errors:
$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
If there are any errors, continue with the troubleshooting steps below to verify the correct installation of Backtrace components and proper permissions.
Verify ptrace installation
Running /opt/backtrace/bin/ptrace <pid>
on a process you own should generate a .btt file:
$ ps
PID TTY TIME CMD
2291 pts/7 00:00:00 bash
11709 pts/7 00:00:00 ps
$ /opt/backtrace/bin/ptrace 2291
/home/test/bash.2291.1461078102.btt
$ ls *.btt
bash.2291.1461078102.btt
If not, verify your ptrace installation: Ptrace Installation
Verify coroner installation
Make sure that you are able to send snapshots to the server via coroner:
$ coroner -c put project1 project1_token bash.2291.1461078102.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
If not, verify your coroner client installation: Coroner Client Installation
Launch backtrace.sh manually
Try launching backtrace.sh
manually to make sure there are no errors:
$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9
Make sure ATS has permission to run invoker.sh
$ chmod +x /path/to/invoker.sh
Make sure backtrace.sh is being invoked and generating the .btt file
If you comment out the last 3 lines of backtrace.sh
, you can verify that the invoker
is running the script and generating the .btt file in the desired location.