nodemcu-firmware/tests

Introduction

Welcome to the NodeMCU self-test suite. Here you will find our growing effort to ensure that our software behaves as we think it should and that we do not regress against earlier versions.

Our tests are written using NTest, a lightweight yet featureful framework for specifying unit tests.

Building and Running Test Software on NodeMCU Devices

Naturally, to test NodeMCU on its intended hardware, you will need one or more NodeMCU-capable boards. At present, the test environment is specified using two ESP8266 Devices Under Test (DUTs), but we envision expanding this to mixed ESP8266/ESP32 environments as well.

Test programs live beside this file. While many test programs run on the NodeMCU DUTs, but there is reason to want to orchestrate DUTs and the environment using the host. Files matching the glob NTest_*.lua are intended for on-DUT execution.

Modules Required

Basic test harness functionality depends on having at least the following C modules available:

• crypto
• encoder
• file

Of course, most of the test functionality itself will require additional modules.

Manual Test Invocation

At the moment, the testing regime and host-based orchestration is still in development, and so things are a little more manual than perhaps desired. The NTest-based test programs all assume that they can require "NTest", and so the easiest route to success is to

• build an LFS image containing

  • package.loader support for LFS

  • NTest itself

  • Any additional Lua support modules required (e.g., mcp23017 support )

• build a firmware with the appropriate C modules

• program the board with your firmware and LFS images

• ensure that package.loader is patched appropriately on startup

• transfer the NTest_foo program you wish to run to the device SPIFFS (or have included it in the LFS).

• at the interpreter prompt, say dofile("NTest_foo.lua") (or node.LFS.get("NTest_foo")()) to run the foo test program.

Experimental Host Orchestration

Enthusiastic testers are encouraged to try using our very new, very experimental host test runner, tap-driver.expect. To use this program, in addition to the above, the LFS environment should contain NTestTapOut, an output adapter for NTest, making it speak a slight variant of the Test Anything Protocol. This structured output is scanned for by the script on the host.

You'll need expect and TCL and some TCL libraries available; on Debian, that amounts to

apt install tcl tcllib tclx8.4 expect

This program should be invoked from beside this file with something like

TCLLIBPATH=./expectnmcu ./tap-driver.expect -serial /dev/ttyUSB3 -lfs ./lfs.img NTest_file.lua

This will...

• transfer and install the specified LFS module (and reboot the device to load LFS)

• transfer the test program

• run the test program with NTest shimmed to use the NTestTapOut output handler

• summarize the results

• return 0 if and only if all tests have passed

This tool is quite flexible and takes a number of other options and flags controlling aspects of its behavior:

• Additional files, Lua or otherwise, may be transferred by specifing them before the test to run (e.g., ./tap-driver.expect a.lua b.lua NTest_foo.lua); dually, a -noxfer flag will suppress transferring even the last file. All transferred files are moved byte-for-byte to the DUT's SPIFFS with names, but not directory components, preserved.

• The -lfs LFS.img option need not be specified and, if not given, any existing LFS image will remain on the device for use by the test.

• A -nontestshim flag will skip attempting to shim the given test program with NTestTapOut; the test program is expected to provide its own TAP output. The -tpfx argument can be used to override the leading TAP: sigil used by the NTestTapOut output handler.

• A -runfunc option indicates that the last argument is not a file to transfer but rather a function to be run. It will be invoked at the REPL with a single argument, the shimmed NTest constructor, unless -nontestshim is given, in which case the argument will be nil.

• A -notests option suppresses running tests (making the tool merely another option for loading files to the device).

Transfers will be significantly faster if pipeutils is available to require on the DUT, but a fallback strategy exists if not. We suggest either including pipeutils in LFS images, in SPIFFS, or as the first file to be transferred.

NodeMCU Testing Environment

Herein we define the environment our testing framework expects to see when it runs. It is composed of two ESP8266 devices, each capable of holding an entire NodeMCU firmware, LFS image, and SPIFFS file system, as well as additional peripheral hardware. It is designed to fit comfortably on a breadboard and so should be easily replicated and integrated into any firmware validation testing.

The test harness runs from a dedicated host computer, which is expected to have reset- and programming-capable UART links to both ESP8266 devices, as found on almost all ESP8266 boards with USB to UART adapters, but the host does not necessarily need to use USB to connect, so long as TXD, RXD, DTR, and RTS are wired across.

A particular implementation of this can be found at Test Harness.

Peripherals

I2C Bus

There is an I2C bus hanging off DUT 0. Attached hardware is used both as tests of modules directly and also to facilitate testing other modules (e.g., gpio).

MCP23017: I/O Expander

At address 0x20. An 16-bit tristate GPIO expander, this chip is used to test I2C, GPIO, and ADC functionality. This chip's interconnections are as follows:

MPC23017	Purpose
/RESET	DUT0 reset. This resets the chip whenever the host computer resets DUT 0 over its serial link (using DTR/RTS).
B 0	4K7 resistor to DUT 0 ADC.
B 1	2K2 resistor to DUT 0 ADC.
B 5	DUT1 GPIO16/WAKE via 4K7 resitor
B 6	DUT0 GPIO13 via 4K7 resistor and DUT1 GPIO15 via 4K7 resistor
B 7	DUT0 GPIO15 via 4K7 resistor and DUT1 GPIO13 via 4K7 resistor

Notes:

• DUT 0's ADC pin is connected via a 2K2 reistor to this chip's port B, pin 1 and via a 4K7 resistor to port B, pin 0. This gives us the ability to produce approximately 0 (both pins low), 1.1 (pin 0 high, pin 1 low), 2.2 (pin 1 high, pin 0 low), and 3.3V (both pins high) on the ADC pin.
• Port B pins 6 and 7 sit on the UART cross-wiring between DUT 0 and DUT 1. The 23017 will be tristated for inter-DUT UART tests, but these
• Port B pins 2, 3, and 4, as well as all of port A, remain available for expansion.
• The interrupt pins are not yet routed, but could be. We reserve DUT 0 GPIO 2 for this purpose with the understanding that the 23017's interrupt functionality will be disabled (INTA, INTB set to open-drain, GPINTEN set to 0) when not explicitly under test.

ESP8266 Device 0 Connections

ESP	Usage
GPIO 0	Used to enter programming mode; otherwise unused in test environment.
GPIO 1	Primary UART transmit; reserved for host communication
GPIO 2	[reserved for 1-Wire] [+ reserved for 23017 INT[AB] connections]
GPIO 3	Primary UART recieve; reserved for host communication
GPIO 4	I2C SDA
GPIO 5	I2C SCL
GPIO 6	[Reserved for on-chip flash]
GPIO 7	[Reserved for on-chip flash]
GPIO 8	[Reserved for on-chip flash]
GPIO 9	[Reserved for on-chip flash]
GPIO 10	[Reserved for on-chip flash]
GPIO 11	[Reserved for on-chip flash]
GPIO 12
GPIO 13	Secondary UART RX; DUT 1 GPIO 15, I/O expander B 6
GPIO 14
GPIO 15	Secondary UART TX; DUT 1 GPIO 13, I/O expander B 7
GPIO 16
ADC 0	Resistor divider with I/O expander

ESP8266 Device 1 Connections

ESP	Usage
GPIO 0	Used to enter programming mode; otherwise unused in test environment.
GPIO 1	Primary UART transmit; reserved for host communication
GPIO 2	[Reserved for WS2812]
GPIO 3	Primary UART recieve; reserved for host communication
GPIO 4
GPIO 5
GPIO 6	[Reserved for on-chip flash]
GPIO 7	[Reserved for on-chip flash]
GPIO 8	[Reserved for on-chip flash]
GPIO 9	[Reserved for on-chip flash]
GPIO 10	[Reserved for on-chip flash]
GPIO 11	[Reserved for on-chip flash]
GPIO 12	HSPI MISO
GPIO 13	Secondary UART RX; DUT 0 GPIO 15, I/O exp B 7 via 4K7 Also used as HSPI MOSI for SPI tests
GPIO 14	HSPI CLK
GPIO 15	Secondary UART TX; DUT 0 GPIO 13, I/O exp B 6 via 4K7 Also used as HSPI /CS for SPI tests
GPIO 16	I/O expander B 5 via 4K7 resistor, for deep-sleep tests
ADC 0

Probing the Test Environment from Tests

The NTestEnv module provides convenient functions for preflight checks and limited adaptation of test programs.

Test Configuration File

Our tests expect a testenv.conf in SPIFFS to provide parameters to some tests. This file is a JSON map with the following keys:

• "DUT". Its value is either 0 or 1 indicating which DUT is running the given test.