On-Target Testing with LinkServer Runner and VS Code

Posted on by

The release 24.9.75 of LinkServer software and tools includes interesting feature: the ability to use the debug probe for automated on-target testing. It includes a ‘runner’ which can program, launch and run the application on the target through a debug probe. While the target is running, it uses semihosting or UART for communication. This makes it a perfect tool for automated testing, especially in a CI/CD environment. One such environment is running automated tests with CMake and CTest in VS Code.

Automated on-target testing with VS Code and LinkServer

Outline

The ‘runner’ feature is a new functionality in the NXP LinkServer for Microcontrollers 24.9.75 release (see NXP LinkServer 24.9.75: New GUI for Flash Programming). It offers two things in one:

  1. to program and launch an application on the target as CMSIS-DAP debug probe and
  2. to use a communication channel to set up and monitor it.

With the Runner I can program and run the application through the debug probe:

Program and Run

After that, I can configure and monitor the application using semihosting or UART:

Configure and Monitor

I can load the application with the runner, passing configuration arguments, for example which tests to run. With semihosting or UART I can exchange data, for example getting test output and test results.

The debug probe can be an external one (e.g. MCU-Link) or on-board (e.g. on NXP LPC55S16-EVK).

The combination of both makes it an excellent choice for running automated on-target tests. See for example Modern On-Target Embedded System Testing with CMake and CTest.

The benefits of this solutions are:

  1. Inexpensive hardware solution for on-target testing: on-board or NXP MCU-Link ($15)
  2. Can directly program the application on target
  3. Communicates bidirectional with the the target application using UART or semihosting.
  4. Easy integration into automated on-target testing environment with VS Code and CTest.

In the next sections I describe the infrastructure and settings to get up and running. I’m showing it with the NXP LPC55S16-EVK board, both with the on-board debug probe and an external MCU-Link debug probe. You can find an example project on GitHub (https://github.com/ErichStyger/MCUXpresso_LPC55S16_CI_CD).

LPC55S16-EVK UART Connection

One can use the on-board debug probe UART to USB-CDC connection, or the UART header (J3) on the board.

With the onboard UART to debug probe connection, the jumper JP9 needs to be open. Additionally, the jumper JP12 needs to be closed:

On-Board Debug Probe UART Jumpers

If using the J3 UART header, the jumper JP9 needs to closed, and JP12 closed:

Off-Board Debug Probe UART Jumpers and Connection

LinkServer Runner

The LinkServer includes a runner command line utility, with the ‘run’ command. Call with -h to get help:

LinkServer run -h
LinkServer Runner Commands

The --mode sets the communication channel: UART or semihosting.

With --send I can send text to the application (UART or semihosting), for example to tell which tests to run.

The different --exit-mark is used to end the application. If the application sends that text to the UART or with semihosting, the LinkServer runner terminates the program.

The --pass-mark and --fail-mark are used what error code the LinkServer runner shall use. For example CTest expects an error or return code of 0 for ‘test success’, and non-zero for ‘test failed’. That ‘pass’ or ‘fail’ text is sent by the application to the UART or with semihosting to the LinkServer runner.

Example using UART

Here is an example using the UART:

LinkServer --log-level 1 run --mode serial:COM57:115200 --exit-mark "*STOP*" --pass-mark "*** PASSED ***" --fail-mark "*** FAILED ***" --send "led1" lpc55s16 "build/debug-test/LPC55S16_Blinky.elf"
  • --log-level 1: set the amount of data shown on the console.
  • --mode serial:COM57:115200: use serial mode with COM57 and baud 115200
  • --exit-mark "*STOP*": if application writes that string, exit it.
  • –-pass-mark "*** PASSED ***“: if this string is present in the output, the runner will set exit or error code to 0 (success).
  • --fail-mark "*** FAILED ***": if this string is present, the runner exit code is -1 (non-zero, failed).
  • --send "led1": send this string to the application. The application uses it to select a test.
  • lpc55s16: use the LPC55S16 device.
  • "build/debug-test/LPC55S16_Blinky.elf": the application to load to the target.

Runner error code

If using a test tool like CTest with CMake, it expects that the runner returns 0 for success. By default, a non-zero return indicates failure. This greatly simplifies CTest usage, as otherwise I have to use regular expressions (see https://cmake.org/cmake/help/v3.0/manual/ctest.1.html). I had to use this in Modern On-Target Embedded System Testing with CMake and CTest. The LinkServer runner does not need a regular expression in the CTest.

One can easily verify the error code. Consider the the following output:

LinkServer --log-level 1 run --mode serial:COM57:115200 --exit-mark "*STOP*" --pass-mark "*** PASSED ***" --fail-mark "*** FAILED ***" --send "Led_1" lpc55s16 "build/debug-test/LPC55S16_Blinky.elf"

INFO application.c:33: Hello from the Log module
INFO application.c:47: Running Unit Tests.
INFO shell.c:67: Shell task started.
INFO tests.c:39: starting test task
INFO tests.c:46: Shell suspend
INFO McuUnity.c:35: getting UART arguments...
INFO tests.c:53: Shell to be resumed
INFO tests.c:57: uart nof: 5, buf: Led_1
INFO tests.c:100: Test arg: -1
ERROR tests.c:120: *** FAILED ***
INFO tests.c:132: *STOP*

That run has failed, and I can check it with:

echo %errorlevel%

which prints -1 on the console. It will print 0 for success.

Getting Arguments

With the --send runner option, I can send text to the running application, for example:

--send "led1"

In the application, that text can be read with the UART:

 int nofBytes = McuUnity_UART_GetArgs(buf, sizeof(buf), McuShellUart_GetStdio()->stdIn);

Another way is to use semihosting:

 int nofBytes = McuUnity_Semihost_GetArgs(buf, sizeof(buf));

nofBytes has the number of characters read, and the data is in buf.

Selecting Unit Test

The text is compared and the corresponding test number gets selected:

 if (nofBytes>0) { if (McuUtility_strcmp((char*)buf, "led1")==0) { test_arg = 1; } else if (McuUtility_strcmp((char*)buf, "led2")==0) { test_arg = 2; } } else { test_arg = -1; }

Running Unit Test

With this, the selected test gets executed:

 UNITY_BEGIN(); switch(test_arg) { case 1: RUN_TEST(TestLeds_OnOff); break; case 2: RUN_TEST(TestLeds_Toggle); break; default: RUN_TEST(TestArgFailed); break; } nofFailures = UNITY_END();

Reporting Success or Failure

The last step is to let the runner know the result of the tests, with sending the pass or fail strings:

 /* report failed or pass. LinkServer is configured with --pass-mark "*** PASSED ***" --fail-mark "*** FAILED ***" */ if (nofFailures==0) { McuLog_info("*** PASSED ***"); } else { McuLog_error("*** FAILED ***"); }

Exit the Application

The final step is to tell the runner to exit the application. Here again we send a text to the runner:

 /* send stop command. LinkServer is configured with --exit-mark "*STOP*" */ McuLog_info("*STOP*"); /* stop Linkserver test runner */

LinkServer Runner with CTest

The LinkServer test runner easily gets integrated with CMake, CTest and VS Code.. Below is an example running two tests:

#################################################################################
# NXP LinkServer Runner
#################################################################################
# set (RUNNER_CTEST_MODE --mode semihost)
set (RUNNER_CTEST_MODE --mode serial:COM51:115200)

set (RUNNER_CTEST_CONTROL --exit-mark "*STOP*" --pass-mark "*** PASSED ***" --fail-mark "*** FAILED ***")

set (RUNNER_CTEST_COMMAND $ENV{LINKSERVER_PATH}/LinkServer --log-level 1 run ${RUNNER_CTEST_MODE} ${RUNNER_CTEST_CONTROL})
set (RUNNER_CTEST_EXECUTABLE lpc55s16 ${TEST_EXECUTABLE})

add_test(
 NAME Led_1
 COMMAND ${RUNNER_CTEST_COMMAND} --send "led1" ${RUNNER_CTEST_EXECUTABLE}
)

add_test(
 NAME Led_2
 COMMAND ${RUNNER_CTEST_COMMAND} --send "led2" ${RUNNER_CTEST_EXECUTABLE}
)

set_tests_properties(
 Led_1 Led_2 
 PROPERTIES TIMEOUT 15
)

At the beginning I have settings to use either semihosting or UART for my tests.

For details, please check out my example on GitHub (see links at the end).

VS Code

This all nicely works into VS Code.

LinkServer Runner with VS Code, CMake and CTest

Summary

I also use the J-Link solution. The NXP debug probe is less expensive ($15 vs. $60 for the EDU mini). Both have semihosting, and the NXP solution offers UART/VCOM instead of RTT. The LinkServer solution supports NXP devices, while SEGGER supports many different vendors. The LinkServer runner is an excellent and cost effective choice. It suits our test lab and infrastructure needs for the NXP devices and boards.

And there are multiple use cases as a bonus:

  1. Runner as Programmer. I just use the runner to program the binary. The application sends the exit string in main and the application continues running.
  2. Runner as UART monitor: Use the runner to program the binary and use the UART communication channel for input/output. No dedicated terminal program needed.
  3. Runner as semihosting monitor: Instead using the debugger for semihosting input/output, I can use the runner instead. No debugger application or setup needed.

Happy running 🙂

Links

What do you think?

This site uses Akismet to reduce spam. Learn how your comment data is processed.