If you are submitting minidump files then you will need to ensure that debug symbols have been uploaded to Backtrace in order to have accurate classification, deduplication and callstack rendering.

Generating Symbols

Windows

In Visual Studio, your DEBUG build will generate .pdb symbol files. If you wish to generate symbol files in your RELEASE build or in a custom configuration, make sure that "Generate Debug Info" is turned on in your project's Configuration Properties.

Linux

Backtrace supports .pdb  ELF and .sym file formats, 

Mac and iOS

Backtrace supports the  dSYM format. 

Symbol Formats and Upload Methods

Regardless of your upload method, Backtrace provides a great of observability into symbol state. Navigate to your project configuration page and click on Symbols to see a record of all symbols.

Backtrace supports .pdb .sym (Breakpad symbol files)  ELF and dSYM files and symbol archives (compressed or uncompressed archives containing the files). These symbol files can be uploaded manually or hooked up into your build system so they are automatically uploaded.

NOTE: The table above does not include .elf and dSYM files, which are uploadable as an archive via Web Browser, curl, and HTTP.

You may also specify a tag query string parameter to any of your symbol uploads in order to group symbols for ease of management. A tag is simply a group of symbols, like a folder on your filesystem. You may want to have different tags for different versions of your application or different platforms. If no tag is specified, symbols are placed into the anon tag.

The recommended symbol upload format is a symbol archive. This is a .tar.gz or .zip file containing one or more .sym ,.pdb , ELF  or dSYM files. There are no restrictions on the layout of the files but you must ensure no relative paths are used and that files have the correct basename. For example, debug information for Editor.exe must be in a file called Editor.pdb or Editor.sym. Socorro symbol archives work with Backtrace without further configuration.

NOTE: Make sure that your symbols files (.pdb .sym, etc) are uploaded before their corresponding .exe and .dll files, or included in the same archive, to ensure proper processing. Note that .exe and .dll files are only required for 64-bit executables.

Symbol Management

Breakpad and Socorro

Backtrace is completely compatible with existing Breakpad and Socorro users. Simply upload the .sym files through an HTTP POST or the sym_upload tool. Ensure that you have an Access Token in order to upload to your instance. You must provide a token (referring to your symbol access token) and a format query string parameter with the value symbols.

Access tokens used for symbol uploads are different than error submission tokens. When uploading symbols, make sure you're using a symbol access token, which can be found under Project Settings | Symbols | Access Tokens.

Below is an example invocation of a symbol upload using the sym_upload tool.

sym_upload null_read_av.sym 'http://yourcompany.sp.backtrace.io:6097/post?format=symbols&token=57f2126dcef18bb0d2af35ec1d813f3775ee8228d1d886de522b2aedceff8b87'

HTTP API

In order to build automation around symbol upload, such as integration into a build and release process, you'll want to interface directly with the HTTP API provided by Backtrace.

Simply issue an HTTP POST to <your instance>/post?format=symbols&token=<access token>. An access token can be created by navigating to Projects > Your Project > Symbols > Manage Access Tokens and clicking on the Create a new access token button.

Below is an example of a curl invocation to submit a symbol archive.

curl --data-binary @symbols.tar 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=5ae2136d4ef181b0d2afa5ef1d81ff377eea8228d1d883d4552621ed1eff8b87'

Example curl Commands for HTTP Symbol Upload

Web Browser

Navigate to your project configuration page and click on Symbols in order to manage symbols through your web browser. You are able to manually upload .pdb or .sym and compressed archives of .sym, .pdb , ELF or dSYM files directly in your web browser. Follow the on-screen instructions for more details.

Troubleshooting

If you are blocked on uploading symbols from the command line, try uploading directly from your web browser. Instructions are available above.

Invalid Token

If you receive an "invalid token" error in the response when uploading symbols via HTTP, check to make sure that you're using a symbol token and not a project token. You can create a symbol token on the Project Settings page by clicking the Symbols section, then the Manage Access Tokens tab in the middle of the page.

Missing Symbols

Sometimes, symbols can get missed during the symbol upload process. A list of missing symbols for a dump is available in the web debugger annotations pane. For more information about how to acquire this list, refer to the Web Debugger guide.

After uploading missing symbols, you can Reprocess Objects to have the dumps with missing symbols reprocessed. Note that the grouping of dumps may change after missing symbols are uploaded. To reprocess objects, click on the Reprocess objects button in the project configuration page (either by clicking on Configure Project in the top right menu or clicking on the relevant project in theConfigure Organization screen). The Reprocess objects button is in the top right corner of the configuration page.

Locating Missing Symbols

The first step in locating missing symbol files is to determine the name of the symbol file and its debug identifier. There are two ways to do this via the Debugger UI:

  1. When viewing an error that's missing symbols, Backtrace will add a Missing Symbols branch to the Annotations section at the lower-right part of the screen. When you expand an entry number, you can see the path to the corresponding executable, the name of the symbol file, the debug identifier, the exe or module (shown as "path"), and version information:

   2. If an error is missing symbols, you can also see this in the callstack. In an affected frame, you will see a triangular warning symbol along with a memory address where the function signature would normally be. If you mouse over one of these frames, you'll see a yellow warning missing symbols pop-up. The debug identifier and the name of the symbol file are in the heading:

If you have the morgue  tool installed, you can also check the status of symbol uploads and obtain lists of missing symbols by using the morgue symbol  commands.  See https://github.com/backtrace-labs/backtrace-morgue for more information.

You now have the information you need to check your in-house symbol archives for the missing symbol file. How to identify the symbol file depends on what kind of symbol files you're using:

.sym files - If you're using Breakpad .sym files, this is quite simple: The first line of Breakpad sym file is the MODULE record which lists the pdb name and debug identifier. For exampleMODULE windows x86_64 D0489F894E07424AAB5E626FF8C943DD1 advapi32.pdb

.pdb files - (More Likely) If you're using .pdb files, you need to extract the debug identifier from .pdb files using one of the following methods, and find the matching one:

  • dia2dump - From the Microsoft MSDN site
  • Since .exe and .pdb files are normally stored in the same location, you can use the Windows build tool dumpbin.exe to dump the executable's headers, which includes the matching symbols' name and debug identifier. For example, the following command suggests symbol bcrypt.pdb with id 5C82DF990DA04C46A2B22ABB82D6B66A1.
> dumpbin.exe /headers bcrypt.dll

  Debug Directories

        Time Type       Size      RVA  Pointer
    -------- ------ -------- -------- --------
    584A7C7E cv           23 00021A80    20C80    Format: RSDS, {5C82DF99-0DA0-4C46-A2B2-2ABB82D6B66A}, 1, bcrypt.pdb

On Windows 32-bit executable objects all of the relevant debug information is stored in the corresponding PDB file and Backtrace will ignore the executable and library files for 32-bit applications.  64-bit executable objects the executable, libraries, and PDB files must be uploaded as some unwinding information required for symbolification is contained exclusively in the executable objects.

.dsym files - MacOS .dsym files are supported.

.elf files - ELF files are supported.

Once you locate and upload missing symbols, make sure to Reprocess Objects, as described in the previous section on missing symbols.

Mismatched module name

In addition to not having uploaded the proper symbol file, another cause for mixups is having a mismatched module name.  When viewing missing symbol information, Backtrace will show the name of the module name (executable or dll/library file) that it's expecting that symbol file to match.  This is "path" in the missing symbols section of annotations in the Debugger, or shown as "Library" when looking at the tooltip of a frame that's shown as missing symbols.

If the name of the module doesn't match, Backtrace will show the symbol is missing, so make sure your symbol files reflect the correct module name, and if uploading exe/dll files, make sure these have the correct names.

Inaccurate callstack

Windows

On Windows, 64-bit applications store some unwinding information exclusively in the executable object (.exe or .dll file). For this reason, we advise to include the executable code of your application and library during symbol upload. These files can be uploaded as stand-alone files, but you must ensure that the file base name matches the base name of the .pdb file. For example, the debug information for Editor.exe is expected to be in Editor.pdb. It is important that the name of the executable is Editor.exe in this context to pair with Editor.pdb. If the executable was uploaded as Word.exe, then you are unable to pair with Editor.pdb. It is recommended that a symbol archive is used.

Visual Studio

If you are using Visual Studio, you'll need to ensure that symbols are being generated correctly. Additional details are available in the Crashpad Visual Studio document.

Did this answer your question?