ardupilot/Tools/debug

Hardware Debugging with ArduPilot

This directory contains files that are useful for setting up to debug ArduPilot using either a black magic probe or a stlink-v2 with openocd.

This assumes you are debugging a ChibiOS based firmware on a STM32 board.

Debugging with a Black Magic Probe

If you have a black magic probe (see https://1bitsquared.com/products/black-magic-probe) then first make sure it has the latest firmware. See the wiki here for details: https://github.com/blacksphere/blackmagic/wiki

Next, copy the file gdb-black-magic.init to the ArduPilot source directory, in the same directory where you will be starting the debugger. Rename the file to ".gdbinit"

Now either edit the .gdbinit to give the path to the serial port for your black magic probe, or install the provided udev rules file so that the probe will be loaded as /dev/ttyBmpGdb

Now make sure you have the right version of arm-none-eabi-gdb installed. We recommend version 10-2020-q4-major, which is available here: https://firmware.ardupilot.org/Tools/STM32-tools/

Now build ArduPilot with the --debug configure option. You may also like to include the --enable-asserts. Enabling asserts will slow down the firmware quite a lot, but will help catch ChibiOS API usage bugs.

For example:

./waf configure --board Pixhawk1 --debug --enable-asserts

Now build and install your firmware:

./waf copter --upload

After it is loaded you can attach with gdb like this:

arm-none-eabi-gdb build/Pixhawk4/bin/arducopter

then you can use normal gdb commands. If you are not familiar with gdb then do a google search.

Note that for a source view the command "layout src" or "layout spit" is useful.

Debugging with a STLink-v2

If you have a STLink-V2 adapter (or one of the very cheap clones) then you can debug with openocd. Using openocd has the advantage that you can debug threads properly, unlike the black magic probe which can't see ChibiOS threads.

Start by installing the latest version of openocd, then copy the openocd.cfg file from this directory to the directory where you will be debugging.

You may need to edit the openocd.cfg file to set the MCU type. The one in this directory is setup for a STM32F4 board. If you have a STM32F7 or STM32H7 then edit the file in the obvious way.

Now start openocd in a terminal. You should get output like this:

Open On-Chip Debugger 0.10.0+dev-00272-gedb6796 (2018-01-19-17:26)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1800 kHz
adapter_nsrst_delay: 100
srst_only separate srst_nogate srst_open_drain connect_deassert_srst
Info : clock speed 1800 kHz
Info : STLINK v2 JTAG v29 API v2 SWIM v18 VID 0x0483 PID 0x374B
Info : using stlink api v2
Info : Target voltage: 3.253404
Info : stm32h7x.cpu: hardware has 8 breakpoints, 4 watchpoints
Info : Listening on port 3333 for gdb connections
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections

the above output is for a STM32H743 Nucleo board, but others are similar

In another terminal, copy the gdb-openocd.init file to the directory where you will be debugging, calling it .gdbinit.

Now build and load the debug enabled firmware for ArduPilot in the same manner as given above for the Black Magic probe, and start arm-none-eabi-gdb in the same manner.

To see ChibiOS threads use the "info threads" command. See the gdb documentation for more information.

Debugging Hardfaults

Getting fault dump via Serial

All one needs to do is connect the First USART(not OTG) in the SERIAL_ORDER of the board via FTDI. In the case of CubeOrange that is Telem1 and for most boards that should be the case as well. Once connected run following command:

./Tools/debug/crash_debugger.py /path/to/elf --ser-debug --ser-port /dev/ttyxxx path/to/elf/file --dump-filename logfile.txt

Additionally the logfile.txt contains a memory dump, which can be shared along with elf file. And devs can then just start up gdb using following command, and do all the analysis that needs done.

arm-none-eabi-gdb -nx path/to/elf/file -ex "set target-charset ASCII" -ex "target remote | modules/CrashDebug/bins/lin64/CrashDebug --elf path/to/elf/file --dump logfile.txt"

Getting fault dump via Flash

If a fault happens the information gets recorded in flash sector defined in hwdef define HAL_CRASH_DUMP_FLASHPAGE xx .

Only one crash will be recorded per flash cycle. At every new firmware update the flash will be ready again to record the crash log. Maybe we can erase the crash flash page via a parameter or maybe right after we fetch the crash_dump.bin. To fetch the crash dump @SYS/crash_dump.bin can be fetched via MAVFTP.

Once fetched one can either use the following command to immediately dump backtrace with locals:

./Tools/debug/crash_debugger.py /path/to/elf --dump-debug --dump-filein crash_dump.bin

or to open in gdb for further postmortem do the following:

arm-none-eabi-gdb -nx path/to/elf/file -ex "set target-charset ASCII" -ex "target remote | modules/CrashDebug/bins/lin64/CrashDebug --elf path/to/elf/file --dump crash_dump.bin"

Debugging faults using GDB:

• Connect hardware over SWD
• Place breakpoint at hardfault using b *&HardFault_Handler
• If one is lucky process stack remained untouched they can do set $sp = $psp
• Now you can simply run backtrace and potentially reach the fault
• If fault happens at startup one can run and then wait for breakpoint hit at HardFault_Handler and then set $sp = $psp and do backtrace
• One can also log the RAM, refer crash_debugger app and Tools/debug/crash_dump.scr for the same.

References:

https://interrupt.memfault.com/blog/cortex-m-fault-debug

c8e801225b

http://www.cyrilfougeray.com/2020/07/27/firmware-logs-with-stack-trace.html