nodemcu-firmware/tests

Introduction

Welcome to the NodeMCU self-test suite. Here you will find our growing effort to ensure that our software behaves as we think it should and that we do not regress against earlier versions.

Our tests are written using NTest, a lightweight yet featureful framework for specifying unit tests.

Building and Running Test Software on NodeMCU Devices

Naturally, to test NodeMCU on its intended hardware, you will need one or more NodeMCU-capable boards. At present, the test environment is specified using two ESP8266 Devices Under Test (DUTs), but we envision expanding this to mixed ESP8266/ESP32 environments as well.

Test programs live beside this file. While many test programs run on the NodeMCU DUTs, but there is reason to want to orchestrate DUTs and the environment using the host. Files matching the glob NTest_*.lua are intended for on-DUT execution.

Modules Required

Basic test harness functionality depends on having at least the following C modules available:

• crypto
• encoder
• file

Of course, most of the test functionality itself will require additional modules.

Manual Test Invocation

At the moment, the testing regime and host-based orchestration is still in development, and so things are a little more manual than perhaps desired. The NTest-based test programs all assume that they can require "NTest", and so the easiest route to success is to

• build an LFS image containing

  • package.loader support for LFS

  • NTest itself

  • Any additional Lua support modules required (e.g., mcp23017 support )

• build a firmware with the appropriate C modules

• program the board with your firmware and LFS images

• ensure that package.loader is patched appropriately on startup

• transfer the NTest_foo program you wish to run to the device SPIFFS (or have included it in the LFS).

• at the interpreter prompt, say dofile("NTest_foo.lua") (or node.LFS.get("NTest_foo")()) to run the foo test program.

Experimental Host Orchestration

Enthusiastic testers are encouraged to try using our very new, very experimental host test runner, tap-driver.expect. To use this program, in addition to the above, the LFS environment should contain NTestTapOut, an output adapter for NTest, making it speak a slight variant of the Test Anything Protocol. This structured output is scanned for by the script on the host.

You'll need expect and TCL and some TCL libraries available; on Debian, that amounts to

apt install tcl tcllib tclx8.4 expect

This program should be invoked from beside this file with something like

TCLLIBPATH=./expectnmcu ./tap-driver.expect -serial /dev/ttyUSB3 -lfs ./lfs.img NTest_file.lua

This will...

• transfer and install the specified LFS module (and reboot the device to load LFS)

• transfer the test program

• run the test program with NTest shimmed to use the NTestTapOut output handler

• summarize the results

• return 0 if and only if all tests have passed

This tool is quite flexible and takes a number of other options and flags controlling aspects of its behavior:

• Additional files, Lua or otherwise, may be transferred by specifing them before the test to run (e.g., ./tap-driver.expect a.lua b.lua NTest_foo.lua); dually, a -noxfer flag will suppress transferring even the last file. All transferred files are moved byte-for-byte to the DUT's SPIFFS with names, but not directory components, preserved.

• The -lfs LFS.img option need not be specified and, if not given, any existing LFS image will remain on the device for use by the test.

• A -nontestshim flag will skip attempting to shim the given test program with NTestTapOut; the test program is expected to provide its own TAP output. The -tpfx argument can be used to override the leading TAP: sigil used by the NTestTapOut output handler.

• A -runfunc option indicates that the last argument is not a file to transfer but rather a function to be run. It will be invoked at the REPL with a single argument, the shimmed NTest constructor, unless -nontestshim is given, in which case the argument will be nil.

• A -notests option suppresses running tests (making the tool merely another option for loading files to the device).

Transfers will be significantly faster if pipeutils is available to require on the DUT, but a fallback strategy exists if not. We suggest either including pipeutils in LFS images, in SPIFFS, or as the first file to be transferred.

NodeMCU Testing Environment

Herein we define the environment our testing framework expects to see when it runs. It is composed of two ESP8266 devices, each capable of holding an entire NodeMCU firmware, LFS image, and SPIFFS file system, as well as additional peripheral hardware. It is designed to fit comfortably on a breadboard and so should be easily replicated and integrated into any firmware validation testing.

The test harness runs from a dedicated host computer, which is expected to have reset- and programming-capable UART links to both ESP8266 devices, as found on almost all ESP8266 boards with USB to UART adapters, but the host does not necessarily need to use USB to connect, so long as TXD, RXD, DTR, and RTS are wired across.

A particular implementation of this, with some additional goodies, can be found at Test Harness Board.

Peripherals

I2C Bus

There is an I2C bus hanging off each DUT. Attached hardware is used both as tests of modules directly and also to facilitate testing other modules (e.g., gpio).

MCP23017: I/O Expander (DUT 0, 0x20)

At address 0x20 off DUT 0. An 16-bit tristate GPIO expander, this chip is used to test I2C, GPIO, and ADC functionality. This chip's interconnections are as follows:

MPC23017	Purpose
/RESET	DUT0 reset. This resets the chip whenever the host computer resets DUT 0 over its serial link (using DTR/RTS).
B 0	4K7 resistor to DUT 0 ADC.
B 1	2K2 resistor to DUT 0 ADC.
B 2	Direct to DUT1 RST
B 3	Direct to DUT1 D3
B 4	When low, connects the alternate UART pins on DUT0 to RX,TX on DUT1 (NOTE: not yet used in the test harness)
B 5	DUT1 GPIO16/WAKE via 4K7 resitor
B 6	DUT0 GPIO13 via 4K4 resistor and DUT1 GPIO15 via 4K7 resistor (also feeds in the primary TX from DUT1 when enabled by B4)
B 7	DUT0 GPIO15 via 4K7 resistor and DUT1 GPIO13 via 4K7 resistor (also feeds the primary RX on DUT1 when enabled by B4)

Notes:

• DUT 0's ADC pin is connected via a 2K2 reistor to this chip's port B, pin 1 and via a 4K7 resistor to port B, pin 0. This gives us the ability to produce approximately 0 (both pins low), 1.1 (pin 0 high, pin 1 low), 2.2 (pin 1 high, pin 0 low), and 3.3V (both pins high) on the ADC pin.

• Port B pins 6 and 7 sit on the UART cross-wiring between DUT 0 and DUT 1. The 23017 will be tristated for inter-DUT UART tests, but these are also useful for GPIO tests on DUT1.

• All of port A remains available for expansion and are routed to the breadboard area.

• The interrupt pins are not yet routed to DUTs, but could be. We reserve DUT 0 GPIO 2 for this purpose with the understanding that the 23017's interrupt functionality will be disabled (INTA, INTB set to open-drain, GPINTEN set to 0) when not explicitly under test.

WS2812s

The test harness reserves DUT1/D4 for connection to WS2812.

DUT Connections

ESP8266 Device 0 Connections

ESP	Usage
D3/GPIO 0	Used to enter programming mode; otherwise unused in test environment.
TX/GPIO 1	Primary UART transmit; reserved for host communication
D4/GPIO 2	(+ reserved for 23017 INT[AB] connections)
RX/GPIO 3	Primary UART recieve; reserved for host communication
D2/GPIO 4	I2C SDA bus 0. Connected to MCP23017, DUT0 Oled display and the TCS34725 if present.
D1/GPIO 5	I2C SCL bus 0
GPIO 6	(Reserved for on-chip flash)
GPIO 7	(Reserved for on-chip flash)
GPIO 8	(Reserved for on-chip flash)
GPIO 9	(Reserved for on-chip flash)
GPIO 10	(Reserved for on-chip flash)
GPIO 11	(Reserved for on-chip flash)
D6/GPIO 12
D7/GPIO 13	Secondary UART RX; DUT 1 GPIO 15, I/O expander B 6
D5/GPIO 14
D8/GPIO 15	Secondary UART TX; DUT 1 GPIO 13, I/O expander B 7
D0/GPIO 16
A0/ADC 0	Resistor divider with I/O expander

ESP8266 Device 1 Connections

ESP	Usage
GPIO 0	Used to enter programming mode; otherwise unused in test environment.
GPIO 1	Primary UART transmit; reserved for host communication
GPIO 2	[Reserved for WS2812]
GPIO 3	Primary UART recieve; reserved for host communication
GPIO 4
GPIO 5
GPIO 6	[Reserved for on-chip flash]
GPIO 7	[Reserved for on-chip flash]
GPIO 8	[Reserved for on-chip flash]
GPIO 9	[Reserved for on-chip flash]
GPIO 10	[Reserved for on-chip flash]
GPIO 11	[Reserved for on-chip flash]
GPIO 12	HSPI MISO
GPIO 13	Secondary UART RX; DUT 0 GPIO 15, I/O exp B 7 via 4K7 Also used as HSPI MOSI for SPI tests
GPIO 14	HSPI CLK
GPIO 15	Secondary UART TX; DUT 0 GPIO 13, I/O exp B 6 via 4K7 Also used as HSPI /CS for SPI tests
GPIO 16	I/O expander B 5 via 4K7 resistor, for deep-sleep tests
ADC 0

Probing the Test Environment from Tests

The NTestEnv module provides convenient functions for preflight checks and limited adaptation of test programs.

Test Configuration File

Our tests expect a testenv.conf in SPIFFS to provide parameters to some tests. This file is a JSON map with the following keys:

• "DUT". Its value is either 0 or 1 indicating which DUT is running the given test.