mirror of https://github.com/ArduPilot/ardupilot
140 lines
6.0 KiB
Markdown
140 lines
6.0 KiB
Markdown
# Hardware Debugging with ArduPilot
|
|
|
|
This directory contains files that are useful for setting up to debug
|
|
ArduPilot using either a black magic probe or a stlink-v2 with
|
|
openocd.
|
|
|
|
This assumes you are debugging a ChibiOS based firmware on a STM32 board.
|
|
|
|
## Debugging with a Black Magic Probe
|
|
|
|
If you have a black magic probe (see
|
|
https://1bitsquared.com/products/black-magic-probe) then first make
|
|
sure it has the latest firmware. See the wiki here for details:
|
|
https://github.com/blacksphere/blackmagic/wiki
|
|
|
|
Next, copy the file gdb-black-magic.init to the ArduPilot source
|
|
directory, in the same directory where you will be starting the
|
|
debugger. Rename the file to ".gdbinit"
|
|
|
|
Now either edit the .gdbinit to give the path to the serial port for
|
|
your black magic probe, or install the provided udev rules file so
|
|
that the probe will be loaded as /dev/ttyBmpGdb
|
|
|
|
Now make sure you have the right version of arm-none-eabi-gdb
|
|
installed. We recommend version 10-2020-q4-major, which is available
|
|
here: https://firmware.ardupilot.org/Tools/STM32-tools/
|
|
|
|
Now build ArduPilot with the --debug configure option. You may also
|
|
like to include the --enable-asserts. Enabling asserts will slow down
|
|
the firmware quite a lot, but will help catch ChibiOS API usage bugs.
|
|
|
|
For example:
|
|
|
|
./waf configure --board Pixhawk1 --debug --enable-asserts
|
|
|
|
Now build and install your firmware:
|
|
|
|
./waf copter --upload
|
|
|
|
After it is loaded you can attach with gdb like this:
|
|
|
|
arm-none-eabi-gdb build/Pixhawk4/bin/arducopter
|
|
|
|
then you can use normal gdb commands. If you are not familiar with gdb
|
|
then do a google search.
|
|
|
|
Note that for a source view the command "layout src" or "layout spit"
|
|
is useful.
|
|
|
|
## Debugging with a STLink-v2
|
|
|
|
If you have a STLink-V2 adapter (or one of the very cheap clones) then
|
|
you can debug with openocd. Using openocd has the advantage that you
|
|
can debug threads properly, unlike the black magic probe which can't
|
|
see ChibiOS threads.
|
|
|
|
Start by installing the latest version of openocd, then copy the
|
|
openocd.cfg file from this directory to the directory where you will
|
|
be debugging.
|
|
|
|
You may need to edit the openocd.cfg file to set the MCU type. The one
|
|
in this directory is setup for a STM32F4 board. If you have a STM32F7
|
|
or STM32H7 then edit the file in the obvious way.
|
|
|
|
Now start openocd in a terminal. You should get output like this:
|
|
|
|
```
|
|
Open On-Chip Debugger 0.10.0+dev-00272-gedb6796 (2018-01-19-17:26)
|
|
Licensed under GNU GPL v2
|
|
For bug reports, read
|
|
http://openocd.org/doc/doxygen/bugs.html
|
|
Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.
|
|
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
|
|
adapter speed: 1800 kHz
|
|
adapter_nsrst_delay: 100
|
|
srst_only separate srst_nogate srst_open_drain connect_deassert_srst
|
|
Info : clock speed 1800 kHz
|
|
Info : STLINK v2 JTAG v29 API v2 SWIM v18 VID 0x0483 PID 0x374B
|
|
Info : using stlink api v2
|
|
Info : Target voltage: 3.253404
|
|
Info : stm32h7x.cpu: hardware has 8 breakpoints, 4 watchpoints
|
|
Info : Listening on port 3333 for gdb connections
|
|
Info : Listening on port 6666 for tcl connections
|
|
Info : Listening on port 4444 for telnet connections
|
|
```
|
|
|
|
the above output is for a STM32H743 Nucleo board, but others are
|
|
similar
|
|
|
|
In another terminal, copy the gdb-openocd.init file to the directory
|
|
where you will be debugging, calling it .gdbinit.
|
|
|
|
Now build and load the debug enabled firmware for ArduPilot in the
|
|
same manner as given above for the Black Magic probe, and start
|
|
arm-none-eabi-gdb in the same manner.
|
|
|
|
To see ChibiOS threads use the "info threads" command. See the gdb
|
|
documentation for more information.
|
|
|
|
# Debugging Hardfaults
|
|
|
|
## Getting fault dump via Serial
|
|
All one needs to do is connect the First USART(not OTG) in the SERIAL_ORDER of the board via FTDI. In the case of CubeOrange that is Telem1 and for most boards that should be the case as well. Once connected run following command:
|
|
|
|
`./Tools/debug/crash_debugger.py /path/to/elf --ser-debug --ser-port /dev/ttyxxx path/to/elf/file --dump-filename logfile.txt`
|
|
|
|
Additionally the logfile.txt contains a memory dump, which can be shared along with elf file. And devs can then just start up gdb using following command, and do all the analysis that needs done.
|
|
|
|
`arm-none-eabi-gdb -nx path/to/elf/file -ex "set target-charset ASCII" -ex "target remote | modules/CrashDebug/bins/lin64/CrashDebug --elf path/to/elf/file --dump logfile.txt"`
|
|
|
|
## Getting fault dump via Flash
|
|
If a fault happens the information gets recorded in flash sector defined in hwdef define HAL_CRASH_DUMP_FLASHPAGE xx .
|
|
|
|
Only one crash will be recorded per flash cycle. At every new firmware update the flash will be ready again to record the crash log. Maybe we can erase the crash flash page via a parameter or maybe right after we fetch the crash_dump.bin.
|
|
To fetch the crash dump @SYS/crash_dump.bin can be fetched via MAVFTP.
|
|
|
|
Once fetched one can either use the following command to immediately dump backtrace with locals:
|
|
|
|
`./Tools/debug/crash_debugger.py /path/to/elf --dump-debug --dump-filein crash_dump.bin`
|
|
|
|
or to open in gdb for further postmortem do the following:
|
|
|
|
`arm-none-eabi-gdb -nx path/to/elf/file -ex "set target-charset ASCII" -ex "target remote | modules/CrashDebug/bins/lin64/CrashDebug --elf path/to/elf/file --dump crash_dump.bin"`
|
|
|
|
## Debugging faults using GDB:
|
|
* Connect hardware over SWD
|
|
* Place breakpoint at hardfault using `b *&HardFault_Handler`
|
|
* If one is lucky process stack remained untouched they can do `set $sp = $psp`
|
|
* Now you can simply run `backtrace` and potentially reach the fault
|
|
* If fault happens at startup one can run and then wait for breakpoint hit at HardFault_Handler
|
|
and then `set $sp = $psp` and do `backtrace`
|
|
* One can also log the RAM, refer crash_debugger app and Tools/debug/crash_dump.scr for the same.
|
|
|
|
### References:
|
|
https://interrupt.memfault.com/blog/cortex-m-fault-debug
|
|
|
|
https://github.com/adamgreen/CrashCatcher/tree/c8e801225bfa12da70c01ea25b58090b2b7a2e0a
|
|
|
|
http://www.cyrilfougeray.com/2020/07/27/firmware-logs-with-stack-trace.html
|