nodemcu-firmware/tests/README.md

# Introduction

Welcome to the NodeMCU self-test suite.  Here you will find our growing effort
to ensure that our software behaves as we think it should and that we do not
regress against earlier versions.

Our tests are written using [NTest](./NTest/NTest.md), a lightweight yet
featureful framework for specifying unit tests.

# Building and Running Test Software on NodeMCU Devices

Naturally, to test NodeMCU on its intended hardware, you will need one or more
NodeMCU-capable boards.  At present, the test environment is specified using
two ESP8266 Devices Under Test (DUTs), but we envision expanding this to mixed
ESP8266/ESP32 environments as well.

Test programs live beside this file.  While many test programs run on the
NodeMCU DUTs, but there is reason to want to orchestrate DUTs and the
environment using the host.  Files matching the glob `NTest_*.lua` are intended
for on-DUT execution.

## Modules Required

Basic test harness functionality depends on having at least the following C
modules available:

* `crypto`
* `encoder`
* `file`

Of course, most of the test functionality itself will require additional
modules.

## Manual Test Invocation

At the moment, the testing regime and host-based orchestration is still in
development, and so things are a little more manual than perhaps desired.  The
`NTest`-based test programs all assume that they can `require "NTest"`, and so
the easiest route to success is to

* build an LFS image containing

  * [package.loader support for LFS](../lua_examples/lfs/_init.lua)

  * [NTest itself](./NTest/NTest.lua)

  * Any additional Lua support modules required (e.g., [mcp23017
  support](../lua_modules/mcp23017/mcp23017.lua) )

* build a firmware with the appropriate C modules

* program the board with your firmware and LFS images

* ensure that `package.loader` is patched appropriately on startup

* transfer the `NTest_foo` program you wish to run to the device SPIFFS
  (or have included it in the LFS).

* at the interpreter prompt, say `dofile("NTest_foo.lua")` (or
  `node.LFS.get("NTest_foo")()`) to run the `foo` test program.

## Experimental Host Orchestration

Enthusiastic testers are encouraged to try using our very new, very
experimental host test runner, [tap-driver.expect](./tap-driver.expect).  To
use this program, in addition to the above, the LFS environment should contain
[NTestTapOut](./tests/utils/NTestTapOut.lua), an output adapter for `NTest`,
making it speak a slight variant of the [Test Anything
Protocol](https://testanything.org/).  This structured output is scanned for
by the script on the host.

You'll need `expect` and TCL and some TCL libraries available; on Debian, that
amounts to

    apt install tcl tcllib tclx8.4 expect

This program should be invoked from beside this file with something like

    TCLLIBPATH=./expectnmcu ./tap-driver.expect -serial /dev/ttyUSB3 -lfs ./lfs.img NTest_file.lua

This will...

* transfer and install the specified LFS module (and reboot the device to load LFS)

* transfer the test program

* run the test program with `NTest` shimmed to use the `NTestTapOut` output
  handler

* summarize the results

* return 0 if and only if all tests have passed

This tool is quite flexible and takes a number of other options and flags
controlling aspects of its behavior:

* Additional files, Lua or otherwise, may be transferred by specifing them
  before the test to run (e.g., `./tap-driver.expect a.lua b.lua
  NTest_foo.lua`); dually, a `-noxfer` flag will suppress transferring even the
  last file.  All transferred files are moved byte-for-byte to the DUT's
  SPIFFS with names, but not directory components, preserved.

* The `-lfs LFS.img` option need not be specified and, if not given, any
  existing `LFS` image will remain on the device for use by the test.

* A `-nontestshim` flag will skip attempting to shim the given test program
  with `NTestTapOut`; the test program is expected to provide its own TAP
  output.  The `-tpfx` argument can be used to override the leading `TAP: `
  sigil used by the `NTestTapOut` output handler.

* A `-runfunc` option indicates that the last argument is not a file to
  transfer but rather a function to be run.  It will be invoked at the REPL
  with a single argument, the shimmed `NTest` constructor, unless `-nontestshim`
  is given, in which case the argument will be `nil`.

* A `-notests` option suppresses running tests (making the tool merely another
  option for loading files to the device).

Transfers will be significantly faster if
[pipeutils](../lua_examples/pipeutils.lua) is available to `require` on the
DUT, but a fallback strategy exists if not.  We suggest either including
`pipeutils` in LFS images, in SPIFFS, or as the first file to be transferred.

# NodeMCU Testing Environment

Herein we define the environment our testing framework expects to see
when it runs. It is composed of two ESP8266 devices, each capable of
holding an entire NodeMCU firmware, LFS image, and SPIFFS file system,
as well as additional peripheral hardware. It is designed to fit
comfortably on a breadboard and so should be easily replicated and
integrated into any firmware validation testing.

The test harness runs from a dedicated host computer, which is expected
to have reset- and programming-capable UART links to both ESP8266
devices, as found on almost all ESP8266 boards with USB to UART
adapters, but the host does not necessarily need to use USB to connect,
so long as TXD, RXD, DTR, and RTS are wired across.

A particular implementation of this, with some additional goodies, can
be found at [Test Harness Board](HardwareTestHarness.md).

## Peripherals

### I2C Bus

There is an I2C bus hanging off each DUT. Attached hardware is used both
as tests of modules directly and also to facilitate testing other
modules (e.g., gpio).

#### MCP23017: I/O Expander (DUT 0, 0x20)

At address 0x20 off DUT 0. An 16-bit tristate GPIO expander, this chip
is used to test I2C, GPIO, and ADC functionality. This chip's
interconnections are as follows:

MPC23017 | Purpose
---------|--------------------------------------------------------------
/RESET   |DUT0 reset. This resets the chip whenever the host computer resets DUT 0 over its serial link (using DTR/RTS).
B 0      |4K7 resistor to DUT 0 ADC.
B 1      |2K2 resistor to DUT 0 ADC.
B 2      |Direct to DUT1 RST
B 3      |Direct to DUT1 D3
B 4      |When low, connects the alternate UART pins on DUT0 to RX,TX on DUT1 (NOTE: not yet used in the test harness)
B 5      |DUT1 GPIO16/WAKE via 4K7 resitor
B 6      |DUT0 GPIO13 via 4K4 resistor and DUT1 GPIO15 via 4K7 resistor (also feeds in the primary TX from DUT1 when enabled by B4)
B 7      |DUT0 GPIO15 via 4K7 resistor and DUT1 GPIO13 via 4K7 resistor (also feeds the primary RX on DUT1 when enabled by B4)

Notes:

-   DUT 0's ADC pin is connected via a 2K2 reistor to this chip's port
    B, pin 1 and via a 4K7 resistor to port B, pin 0. This gives us the
    ability to produce approximately 0 (both pins low), 1.1 (pin 0 high,
    pin 1 low), 2.2 (pin 1 high, pin 0 low), and 3.3V (both pins high)
    on the ADC pin.

-   Port B pins 6 and 7 sit on the UART cross-wiring between DUT 0 and
    DUT 1. The 23017 will be tristated for inter-DUT UART tests, but
    these are also useful for GPIO tests on DUT1.

-   All of port A remains available for expansion and are routed to the
    breadboard area.

-   The interrupt pins are not yet routed to DUTs, but could be.  We
    reserve DUT 0 GPIO 2 for this purpose with the understanding that
    the 23017's interrupt functionality will be disabled (INTA, INTB
    set to open-drain, GPINTEN set to 0) when not explicitly under test.

### WS2812s

The test harness reserves DUT1/D4 for connection to WS2812.

# DUT Connections

ESP8266 Device 0 Connections
----------------------------

ESP        | Usage
-----------|----------------------------------------------------------
D3/GPIO 0  |Used to enter programming mode; otherwise unused in test environment.
TX/GPIO 1  |Primary UART transmit; reserved for host communication
D4/GPIO 2  |(+ reserved for 23017 INT[AB] connections)
RX/GPIO 3  |Primary UART recieve; reserved for host communication
D2/GPIO 4  |I2C SDA bus 0. Connected to MCP23017, DUT0 Oled display and the TCS34725 if present.
D1/GPIO 5  |I2C SCL bus 0
GPIO 6     |(Reserved for on-chip flash)
GPIO 7     |(Reserved for on-chip flash)
GPIO 8     |(Reserved for on-chip flash)
GPIO 9     |(Reserved for on-chip flash)
GPIO 10    |(Reserved for on-chip flash)
GPIO 11    |(Reserved for on-chip flash)
D6/GPIO 12 |
D7/GPIO 13 |Secondary UART RX; DUT 1 GPIO 15, I/O expander B 6
D5/GPIO 14 |
D8/GPIO 15 |Secondary UART TX; DUT 1 GPIO 13, I/O expander B 7
D0/GPIO 16 |
A0/ADC 0   |Resistor divider with I/O expander

ESP8266 Device 1 Connections
----------------------------

ESP       | Usage
----------|----------------------------------------------------------
GPIO 0    |Used to enter programming mode; otherwise unused in test environment.
GPIO 1    |Primary UART transmit; reserved for host communication
GPIO 2    |[Reserved for WS2812]
GPIO 3    |Primary UART recieve; reserved for host communication
GPIO 4    |
GPIO 5    |
GPIO 6    |[Reserved for on-chip flash]
GPIO 7    |[Reserved for on-chip flash]
GPIO 8    |[Reserved for on-chip flash]
GPIO 9    |[Reserved for on-chip flash]
GPIO 10   |[Reserved for on-chip flash]
GPIO 11   |[Reserved for on-chip flash]
GPIO 12   |HSPI MISO
GPIO 13   |Secondary UART RX; DUT 0 GPIO 15, I/O exp B 7 via 4K7 Also used as HSPI MOSI for SPI tests
GPIO 14   |HSPI CLK
GPIO 15   |Secondary UART TX; DUT 0 GPIO 13, I/O exp B 6 via 4K7 Also used as HSPI /CS for SPI tests
GPIO 16   |I/O expander B 5 via 4K7 resistor, for deep-sleep tests
ADC 0     |

# Probing the Test Environment from Tests

The `NTestEnv` module provides convenient functions for preflight checks
and limited adaptation of test programs.

## Test Configuration File

Our tests expect a `testenv.conf` in SPIFFS to provide parameters to
some tests.  This file is a JSON map with the following keys:

- "DUT".  Its value is either 0 or 1 indicating which DUT is running the
  given test.