2017-10-28 23:22:34 +02:00
|
|
|
## ESP8266 Lua OTA
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
Espressif use an optional update approach for their firmware know as OTA (over the air).
|
|
|
|
This module offers an equivalent facility for Lua applications developers, and enables
|
|
|
|
module development and production updates by carrying out automatic synchronisation
|
2017-10-28 23:22:34 +02:00
|
|
|
with a named provisioning service at reboot.
|
|
|
|
|
|
|
|
### Overview
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
This `luaOTA` provisioning service uses a different approach to
|
2017-10-28 23:22:34 +02:00
|
|
|
[enduser setup](https://nodemcu.readthedocs.io/en/dev/en/modules/enduser-setup/).
|
2019-02-17 19:26:29 +01:00
|
|
|
The basic concept here is that the ESP modules are configured with a pre-imaged file
|
|
|
|
system that includes a number of files in the luaOTA namespace. (SPIFFS doesn't
|
2017-10-28 23:22:34 +02:00
|
|
|
implement a directory hierarchy as such, but instead simply treats the conventional
|
|
|
|
directory separator as a character in the filename. Nonetheless, the "luaOTA/"
|
2019-02-17 19:26:29 +01:00
|
|
|
prefix serves to separate the lc files in the luaOTA namespace.)
|
2017-10-28 23:22:34 +02:00
|
|
|
|
|
|
|
- `luaOTA/check.lc` This module should always be first executed at startup.
|
2019-02-17 19:26:29 +01:00
|
|
|
- `luaOTA/_init.lc`
|
2017-10-28 23:22:34 +02:00
|
|
|
- `luaOTA/_doTick.lc`
|
|
|
|
- `luaOTA/_provision.lc`
|
|
|
|
|
|
|
|
A fifth file `luaOTA/config.json` contains a JSON parameterised local configuration that
|
|
|
|
can be initially create by and subsequently updated by the provisioning process. Most
|
|
|
|
importantly this configuration contains the TCP address of the provisioning service, and
|
|
|
|
a shared secret that is used to sign any records exchanged between the ESP client and
|
2019-02-17 19:26:29 +01:00
|
|
|
the provisioning service.
|
2017-10-28 23:22:34 +02:00
|
|
|
|
|
|
|
Under this approach, `init.lua` is still required but it is reduced to a one-line lua
|
|
|
|
call which invokes the `luaOTA` module by a `require "luaOTA.check"` statement.
|
|
|
|
|
|
|
|
The `config.json` file which provides the minimum configuration parameters to connect to
|
|
|
|
the WiFi and provisioning server, however these can by overridden through the UART by
|
|
|
|
first doing a `tmr.stop(0)` and then a manual initialisation as described in the
|
|
|
|
[init.lua](#initlua) section below.
|
|
|
|
|
|
|
|
`luaOTA` configures the wifi and connects to the required sid in STA mode using the
|
|
|
|
local configuration. The ESP's IP address is allocated using DHCP unless the optional
|
|
|
|
three static IP parameters have been configured. It then attempts to establish a
|
|
|
|
connection to the named provisioning service. If this is absent, a timeout occurs or the
|
|
|
|
service returns a "no update" status, then module does a full clean up of all the
|
|
|
|
`luaOTA` resources (if the `leave` parameter is false, then the wifi stack is then also
|
|
|
|
shutdown.), and it then transfers control by a `node.task.post()` to the configured
|
|
|
|
application module and function.
|
|
|
|
|
|
|
|
If `luaOTA` does establish a connection to IP address:port of the provisioning service,
|
|
|
|
it then issues a "getupdate" request using its CPU ID and a configuration parameter
|
|
|
|
block as context. This update dialogue uses a simple JSON protocol(described below) that
|
|
|
|
enables the provision server either to respond with a "no update", or to start a
|
|
|
|
dialogue to reprovision the ESP8266's SPIFFS.
|
|
|
|
|
|
|
|
In the case of "no update", `luaOTA` is by design ephemeral, that is it shuts down the
|
|
|
|
net services and does a full resource clean up. Hence the presence of the provisioning
|
|
|
|
service is entirely optional and it doesn't needed to be online during normal operation,
|
|
|
|
as `luaOTA` will fall back to transferring control to the main Lua application.
|
|
|
|
|
|
|
|
In the case of an active update, **the ESP is restarted** so resource cleanup on
|
|
|
|
completion is not an issue. The provisioning dialogue is signed, so the host
|
|
|
|
provisioning service and the protocol are trusted, with the provisioning service driving
|
2019-02-17 19:26:29 +01:00
|
|
|
the process. This greatly simplifies the `luaOTA` client coding as this is a simple
|
|
|
|
responder, which actions simple commands such as:
|
|
|
|
- download a file,
|
2017-10-28 23:22:34 +02:00
|
|
|
- download and compile file,
|
|
|
|
- upload a file
|
|
|
|
- rename (or delete) a file
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
with the ESP being rebooted on completion of the updates to the SPIFFS. Hence in
|
|
|
|
practice the ESP boots into one one two modes:
|
|
|
|
- _normal execution_ or
|
2017-10-28 23:22:34 +02:00
|
|
|
- _OTA update_ followed by reboot and normal execution.
|
|
|
|
|
|
|
|
Note that even though NodeMCU follows the Lua convention of using the `lua` and `lc`
|
|
|
|
extensions respectively for source files that need to be compiled, and for pre-compiled
|
|
|
|
files, the Lua loader itself only uses the presence of a binary header to determine the
|
|
|
|
file mode. Hence if the `init.lua` file contains pre-compiled content, and similarly all
|
|
|
|
loaded modules use pre-compiled lc files, then the ESP can run in production mode
|
|
|
|
_without needing to invoke the compiler at all_.
|
|
|
|
|
|
|
|
The simplest strategy for the host provisioning service is to maintain a reference
|
|
|
|
source directory on the host (per ESP module). The Lua developer can maintain this under
|
|
|
|
**git** or equivalent and make any changes there, so that synchronisation of the ESP
|
|
|
|
will be done automatically on reboot.
|
|
|
|
|
|
|
|
### init.lua
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
This is typically includes a single line:
|
2017-10-28 23:22:34 +02:00
|
|
|
```Lua
|
|
|
|
require "LuaOTA.check"
|
|
|
|
```
|
2019-02-17 19:26:29 +01:00
|
|
|
however if the configuration is incomplete then this can be aborted as manual process
|
2017-10-28 23:22:34 +02:00
|
|
|
by entering the manual command through the UART
|
|
|
|
```Lua
|
|
|
|
tmr.stop(0); require "luaOTA.check":_init {ssid ="SOMESID" --[[etc. ]]}
|
|
|
|
```
|
2019-02-17 19:26:29 +01:00
|
|
|
where the parameters to the `_init` method are:
|
2017-10-28 23:22:34 +02:00
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- `ssid` and `spwd`. The SSID of the Wifi service to connect to, together with its
|
2017-10-28 23:22:34 +02:00
|
|
|
password.
|
|
|
|
- `server` and `port`. The name or IP address and port of the provisioning server.
|
2019-02-17 19:26:29 +01:00
|
|
|
- `secret`. A site-specific secret shared with the provisioning server for MD5-based
|
2017-10-28 23:22:34 +02:00
|
|
|
signing of the protocol messages.
|
|
|
|
- `leave`. If true the STA service is left connected otherwise the wifi is shutdown
|
2019-02-17 19:26:29 +01:00
|
|
|
- `espip`,`gw`,`nm`,`ns`. These parameters are omitted if the ESP is using a DHCP
|
|
|
|
service for IP configuration, otherwise you need to specify the ESP's IP, gateway,
|
2017-10-28 23:22:34 +02:00
|
|
|
netmask and default nameserver.
|
|
|
|
|
|
|
|
If the global `DEBUG` is set, then LuaOTA will also dump out some diagnostic debug.
|
|
|
|
|
|
|
|
### luaOTA.check
|
|
|
|
|
|
|
|
This only has one public method: `_init` which can be called with the above parameters.
|
|
|
|
However the require wrapper in the check module also posts a call to `self:_init()` as a
|
|
|
|
new task. This new task function includes a guard to prevent a double call in the case
|
2019-02-17 19:26:29 +01:00
|
|
|
where the binding require includes an explicit call to `_init()`
|
2017-10-28 23:22:34 +02:00
|
|
|
|
|
|
|
Any provisioning changes results in a reboot, so the only normal "callback" is to invoke
|
|
|
|
the application entry method as defined in `config.json` using a `node.task.post()`
|
|
|
|
|
|
|
|
|
|
|
|
### luaOTAserver.lua
|
|
|
|
|
|
|
|
This is often tailored to specific project requirements, but a simple example of a
|
|
|
|
provisioning server is included which provides the corresponding server-side
|
|
|
|
functionality. This example is coded in Lua and can run on any development PC or server
|
|
|
|
that supports Lua 5.1 - 5.3 and the common modules `socket`, `lfs`, `md5` and `cjson`.
|
|
|
|
It can be easily be used as the basis of one for your specific project needs.
|
|
|
|
|
|
|
|
Note that even though this file is included in the `luaOTA` subdirectory within Lua
|
|
|
|
examples, this is designed to run on the host and should not be included in the
|
|
|
|
ESP SPIFFS.
|
|
|
|
|
|
|
|
## Implementation Notes
|
|
|
|
|
|
|
|
- The NodeMCu build must include the following modules: `wifi`, `net`, `file`, `tmr`,
|
2019-02-17 19:26:29 +01:00
|
|
|
`crypto` and`sjason`.
|
2017-10-28 23:22:34 +02:00
|
|
|
|
|
|
|
- This implementation follow ephemeral practices, that it is coded to ensure that all
|
2019-02-17 19:26:29 +01:00
|
|
|
resources used are collected by the Lua GC, and hence the available heap on
|
2017-10-28 23:22:34 +02:00
|
|
|
application start is the same as if luaOTA had not been called.
|
|
|
|
|
|
|
|
- Methods in the `check` file are static and inherit self as an upvalue.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- In order to run comfortably within ESP resources, luaOTA executes its main
|
2017-10-28 23:22:34 +02:00
|
|
|
functionality as a number of overlay methods. These are loaded dynamically (and largely
|
|
|
|
transparently) by an `__index` metamethod.
|
|
|
|
|
|
|
|
- Methods starting with a "_" are call-once and return the function reference
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- All others are also entered in the self table so that successive calls will use
|
2017-10-28 23:22:34 +02:00
|
|
|
the preloaded function. The convention is that any dynamic function is called in object
|
|
|
|
form so they are loaded and executed with self as the first parameter, and hence are
|
|
|
|
called using the object form self:someFunc() to get the context as a parameter.
|
|
|
|
|
|
|
|
- Some common routines are also defined as closures within the dynamic methods
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- This coding also makes a lot of use of tailcalls (See PiL 6.3) to keep the stack size
|
|
|
|
to a minimum.
|
2017-10-28 23:22:34 +02:00
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The update process uses a master timer in `tmr` slot 0. The index form is used here
|
2017-10-28 23:22:34 +02:00
|
|
|
in preference to the object form because of the reduced memory footprint. This also
|
|
|
|
allows the developer to abort the process early in the boot sequence by issuing a
|
|
|
|
`tmr.stop(0)` through UART0.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The command protocol is unencrypted and uses JSON encoding, but all exchanges are
|
2017-10-28 23:22:34 +02:00
|
|
|
signed by a 6 char signature taken extracted from a MD5 based digest across the JSON
|
|
|
|
string. Any command which fails the signature causes the update to be aborted. Commands
|
|
|
|
are therefore regarded as trusted, and this simplifies the client module on the ESP.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The process can support both source and compiled code provisioning, but the latter
|
|
|
|
is recommended as this permits a compile-free runtime for production use, and hence
|
2017-10-28 23:22:34 +02:00
|
|
|
minimises the memory use and fragmentation that occurs as a consequence of compilation.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- In earlier versions of the provisioning service example, I included `luaSrcDiet` but
|
2017-10-28 23:22:34 +02:00
|
|
|
this changes the line numbering which I found real pain for debugging, so I now just
|
|
|
|
include a simple filter to remove "--" comments and leading and trailing whitespace if
|
|
|
|
the source includes a `--SAFETRIM` flag. This typically reduced the size of lua files
|
|
|
|
transferred by ~30% and this also means that developers have no excuse for not properly
|
|
|
|
commenting their code!
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The chip ID is included in the configuration identification response to permit the
|
2017-10-28 23:22:34 +02:00
|
|
|
provisioning service to support different variants for different ESP8266 chips.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The (optional update & reboot) operate model also has the side effect that the
|
2017-10-28 23:22:34 +02:00
|
|
|
`LuaOTA` client can reprovision itself.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- Though the simplest approach is always to do a `luaOTA.check` immediately on reboot,
|
|
|
|
there are other strategies that could be applied, for example to test a gpio pin or a
|
|
|
|
flag in RTC memory or even have the application call the require directly (assuming that
|
|
|
|
there's enough free RAM for it to run and this way avoid the connection delay to the
|
2017-10-28 23:22:34 +02:00
|
|
|
WiFi.
|
|
|
|
|
|
|
|
## Discussion on RAM usage
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
`luaOTA` also itself serves as a worked example of how to write ESP-friendly
|
2017-10-28 23:22:34 +02:00
|
|
|
applications.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The functionality is divided into autoloaded processing chunks using a self
|
2017-10-28 23:22:34 +02:00
|
|
|
autoloader, so that `self:somefunction()` calls can load new code from flash in
|
2019-02-17 19:26:29 +01:00
|
|
|
a way that is simple and largely transparent to the application. The autoloader
|
2017-10-28 23:22:34 +02:00
|
|
|
preferentially loads the `lc` compiled code variant if available.
|
|
|
|
|
|
|
|
- The local environment is maintained in a self array, to keep scoping explicit. Note
|
2019-02-17 19:26:29 +01:00
|
|
|
that since loaded code cannot inherit upvalues, then `self` must be passed to the
|
2017-10-28 23:22:34 +02:00
|
|
|
function using an object constructor `self:self:somefunction()`, but where the function
|
2019-02-17 19:26:29 +01:00
|
|
|
can have a self argument then the alternative is to use an upvalue binding. See the
|
2017-10-28 23:22:34 +02:00
|
|
|
`tmr` alarm call at the end of `_init.lua` as an example:
|
|
|
|
```Lua
|
|
|
|
tmr.alarm(0, 500, tmr.ALARM_AUTO, self:_doTick())
|
|
|
|
```
|
2019-02-17 19:26:29 +01:00
|
|
|
- The `self:_doTick()` is evaluated before the alarm API call. This autoloads
|
|
|
|
`luaOTA/_doTick.lc` which stores `self` as a local and returns a function which takes
|
2017-10-28 23:22:34 +02:00
|
|
|
no arguments; it is this last returned function that is used as the timer callback,
|
|
|
|
and when this is called it can still access self as an upvalue.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- This code makes a lot of use of locals and upvalues as these are both fast and use
|
2017-10-28 23:22:34 +02:00
|
|
|
less memory footprint than globals or table entries.
|
|
|
|
|
2019-02-17 19:26:29 +01:00
|
|
|
- The lua GC will mark and sweep to reclaim any unreferenced resources: tables,
|
2017-10-28 23:22:34 +02:00
|
|
|
strings, functions, userdata. So if your code at the end of a processing phase leaves
|
2019-02-17 19:26:29 +01:00
|
|
|
no variables (directly or indirectly) in _G or the Lua registry, then all of the
|
|
|
|
resources that were loaded to carry out your application will be recovered by the GC.
|
2017-10-28 23:22:34 +02:00
|
|
|
In this case heap at the end of a "no provisioning" path is less than 1Kb smaller than
|
2019-02-17 19:26:29 +01:00
|
|
|
if luaOTA had not been called and this is an artifact of how the lua_registry system
|
2017-10-28 23:22:34 +02:00
|
|
|
adopts a lazy reuse of registry entries.
|
|
|
|
|
|
|
|
- If you find that an enumeration of `debug.getregistry()` includes function references
|
2019-02-17 19:26:29 +01:00
|
|
|
or tables other than ROMtables, then you have not been tidying up by doing the
|
|
|
|
appropriate closes or unregister calls. Any such stuck resources can result in a
|
|
|
|
stuck cascade due to upvalues being preserved in the function closure or entries in a
|
2017-10-28 23:22:34 +02:00
|
|
|
table.
|