coresnap consists of two primary components. The
archive tool and
archive is responsible for writing out core files to disk and communicating with the
coresnapd journaling mechanism.
archive is only relevant to Linux targets and is used to handle writes and invocations from the
QUICK START GUIDE
coresnap invokes the Backtrace snapshot tool on core files automatically and then sends the resulting snapshot to the object store using the object store client. Configuration of coresnap consists of installing the package, supplying a
coroner configuration for
coresnapto pass to the object store client and then enabling the service.
Please note that some of the command snippets require administrator privileges.
Make sure that the Backtrace package repository is installed in order to refer to the Backtrace packages. For more information, refer to the documentation portal.
$ yum install backtrace-coresnap
$ apt-get install backtrace-coresnap
$ apt-get install backtrace-coresnap
Please refer to the
coroner documentation for coroner-specific configuration options. Your site administrator or team lead probably already have one for your environment. All you have to do is place your coroner configuration file in
/usr/local/etc/coroner.cf. This path may be changed by modifying
If you wish to modify the Coresnap configuration file, copy
/etc/coresnap/coresnap.conf and then modify your new configuration file. Refer to the configuration file for configuration options.
In order to disable the
apport crash reporting service, the following commands can be used.
$ service apport stop
$ echo "enabled=0" > /etc/default/apport
It is also possible simply uninstall the
apport service with the following commands.
$ apt-get purge apport
$ systemctl disable abrtd.service
The final step is to enable coresnap. You want to disable any crash-reporting solution that is currently enabled on your platform.
$ /etc/init.d/coresnapd start
The status of
coresnapd should report that it is up:
On Ubuntu systems that are using
Upstart, the following line enables Coresnap on boot.
$ echo "enabled=1" > /etc/default/coresnapd
On RHEL systems that are still using
init.d then the following sequence of commands can be used to enable Coresnap on boot.
/sbin/chkconfig --add coresnapd
/sbin/chkconfig --list coresnapd
/sbin/chkconfig coresnapd on
$ systemctl start coresnapd
The status of
coresnapd should report that it is up:
$ systemctl status coresnapd
● coresnapd.service - Backtrace coredump aggregation service
Loaded: loaded (/lib/systemd/system/coresnapd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2016-04-10 17:02:47 EDT; 34min ago
Process: 15863 ExecStart=/opt/backtrace/sbin/coresnapd $CORESNAPD_OPTS (code=exited, status=0/SUCCESS)
Main PID: 15864 (coresnapd)
Apr 10 17:02:47 broadwell systemd: Starting Backtrace coredump aggregation service...
At this point, enable
coresnapd with the following command.
$ systemctl enable coresnapd
Currently, coresnapd can be run on a FreeBSD host and handle its kernel core files, but not userland core files. To configure a FreeBSD host, install the coresnap package, then run:
# sysrc coresnapd_enable=YES
# sysrc local_startup="$(sysrc -n local_startup) /opt/backtrace/etc/rc.d"
/usr/local/etc/coresnap/coresnap.conf and change
snapshot.command to read:
snapshot.command = /opt/backtrace/bin/ptrace -o%S --kv=coresnap.object:%B --resource=%a --load= --map-path=%a --core %c %a/boot/kernel/kernel
Now start coresnapd:
# service coresnapd start
Check its status:
$ service coresnapd status
coresnapd is running as pid 613.
Companion scripts are used for FreeBSD kernel coredump processing. To process kernel core archives, the server currently requires
kernel.sh to be run as a cron job, which can be done in
* * * * * coresnap /opt/backtrace/coresnap/sbin/kernel.sh >/dev/null 2>&1
Note this script must be run as the same user as coresnapd.
To configure FreeBSD hosts to package up kernel core archives and send them to another system running
- Configure the target host for this script:
- # sysrc backtrace_kernel_target=coresnap@localhost
- NB: The username must be specified, and must be the same as the user used to run coresnapd on the target.
- Either set backtrace_kernel.sh to:
- Run on startup: # sysrc backtrace_kernel_enable=YES
- Or run it as a cron job:
* * * * * root /usr/local/etc/rc.d/backtrace_kernel.sh onestart > /dev/null 2>&1
To test the setup, simply force a crash:
# sysctl debug.kdb.panic=1 Prior to executing this command, the machine should have already been configured to save kernel core files, by configuring a swap device large enough to store one (8GB should do), and running:
# sysrc dumpdev=AUTO
Frequently Asked Questions
How do I see what coresnapd is doing?
Coresnap logs error messages in
syslog. Refer to
syslog to diagnose any potential issues. If there are problems in executing sub-processes, you may run
coresnapd by hand with
/opt/backtrace/sbin/coresnapd -f. This specifies foreground mode and forces command output mode to inherit. Command output is displayed on your terminal output in this case.
Which project are snapshots submitted to?
By default, all dumps are submitted to a "blackhole" project using the "blackhole" authentication token. Please ensure that these exist in
coronerd and your
coroner configuration file if this default is used. Otherwise, format rules are typically used for dynamic selection of projects.
is a daemon that is responsible for reliable archival and processing of generated core dumps. It is a process management tool that invokes executables and monitors the resultant processes. These processes can be passed format strings that expand to the set of files being monitored. A file belongs to one of five stages: staging, pending, sending, committed and failed. A file can also have an additional asset file. The state of these files is observable by the state of the filesystem at the path specified by the
dump.root directive in the coresnap configuration file.
coresnap relies on
rename(2) semantics for journaling state of processing for a dump. This process consists of the following stages.
- STAGING: These are core files that are written out by the coresnap archival tool. Upon successful commit of data to disk, file is atomically renamed into the pending directory.
- PENDING: These are core files that are pending processing by coresnapd during the snapshot phase. If the process invoked during the snapshot phase exits with a 0-status then it is expected the process has created an output file in the sending directory. coresnapd takes action upon completion of the phase based on the options specified by the configuration file.
- SENDING: These are files that are pending processing by coresnapd during the send phase. coresnapdtakes action upon completion of the phase based on the options specified by the configuration file.
- COMMITTED: This stage consists of two directories. One for the output of the pending stage and one for the input of the sending stage. Depending on the configuration of coresnapd, these directories are either empty or contain successfully committed objects.
- FAILED: This stage consists of one directory. One for the output of the pending stage and one for the input of the sending stage.
Below is an example of the contents of the default coresnap archive directory.
The asset file is persisted until both the snapshot and coredump are unlinked.
archive tool routes standard input to a uniquely-allocated file in the archive specified by the
dump.rootdirective. Files have the permissions specified by
dump.mode and are owned by the user and group specified by
dump.group respectively. If for some reason, the archive fails to generate a file then any intermediate files are unlinked. The
archive tool also generated a TAR archive in the asset directory that contains environment assets of interest such as
/proc/meminfo and more. The
archive tool attempts to write out standard input as a sparse file. Many process dumps contain plenty of zero-filled pages so this may yield significant space-savings. The archive tool aborts if it detects that an invocation causes the mount point of
dump.root to violate the free block percentage specified by
dump.quota. This protection mechanism is oblivious to concurrent invocations of the archive tool.
The default coresnap configuration file is located in
/opt/backtrace/etc/coresnap/coresnap.conf. coresnap tools search for the following configuration files in this order:
/usr/local/etc/coresnap/coresnap.conf and then
/opt/backtrace/etc/coresnap/coresnap.conf. After installation of the package, a user may wish to copy the default configuration file to a higher-priority path. It is not recommended that a user replaces or modifies the default coresnap configuration file.
The various configuration options are documented inline the default configuration file in