nodemcu-firmware/lua_examples/luaOTA/README.md

231 lines
12 KiB
Markdown
Raw Normal View History

2017-10-28 23:22:34 +02:00
## ESP8266 Lua OTA
2019-02-17 19:26:29 +01:00
Espressif use an optional update approach for their firmware know as OTA (over the air).
This module offers an equivalent facility for Lua applications developers, and enables
module development and production updates by carrying out automatic synchronisation
2017-10-28 23:22:34 +02:00
with a named provisioning service at reboot.
### Overview
2019-02-17 19:26:29 +01:00
This `luaOTA` provisioning service uses a different approach to
2017-10-28 23:22:34 +02:00
[enduser setup](https://nodemcu.readthedocs.io/en/dev/en/modules/enduser-setup/).
2019-02-17 19:26:29 +01:00
The basic concept here is that the ESP modules are configured with a pre-imaged file
system that includes a number of files in the luaOTA namespace. (SPIFFS doesn't
2017-10-28 23:22:34 +02:00
implement a directory hierarchy as such, but instead simply treats the conventional
directory separator as a character in the filename. Nonetheless, the "luaOTA/"
2019-02-17 19:26:29 +01:00
prefix serves to separate the lc files in the luaOTA namespace.)
2017-10-28 23:22:34 +02:00
- `luaOTA/check.lc` This module should always be first executed at startup.
2019-02-17 19:26:29 +01:00
- `luaOTA/_init.lc`
2017-10-28 23:22:34 +02:00
- `luaOTA/_doTick.lc`
- `luaOTA/_provision.lc`
A fifth file `luaOTA/config.json` contains a JSON parameterised local configuration that
can be initially create by and subsequently updated by the provisioning process. Most
importantly this configuration contains the TCP address of the provisioning service, and
a shared secret that is used to sign any records exchanged between the ESP client and
2019-02-17 19:26:29 +01:00
the provisioning service.
2017-10-28 23:22:34 +02:00
Under this approach, `init.lua` is still required but it is reduced to a one-line lua
call which invokes the `luaOTA` module by a `require "luaOTA.check"` statement.
The `config.json` file which provides the minimum configuration parameters to connect to
the WiFi and provisioning server, however these can by overridden through the UART by
first doing a `tmr.stop(0)` and then a manual initialisation as described in the
[init.lua](#initlua) section below.
`luaOTA` configures the wifi and connects to the required sid in STA mode using the
local configuration. The ESP's IP address is allocated using DHCP unless the optional
three static IP parameters have been configured. It then attempts to establish a
connection to the named provisioning service. If this is absent, a timeout occurs or the
service returns a "no update" status, then module does a full clean up of all the
`luaOTA` resources (if the `leave` parameter is false, then the wifi stack is then also
shutdown.), and it then transfers control by a `node.task.post()` to the configured
application module and function.
If `luaOTA` does establish a connection to IP address:port of the provisioning service,
it then issues a "getupdate" request using its CPU ID and a configuration parameter
block as context. This update dialogue uses a simple JSON protocol(described below) that
enables the provision server either to respond with a "no update", or to start a
dialogue to reprovision the ESP8266's SPIFFS.
In the case of "no update", `luaOTA` is by design ephemeral, that is it shuts down the
net services and does a full resource clean up. Hence the presence of the provisioning
service is entirely optional and it doesn't needed to be online during normal operation,
as `luaOTA` will fall back to transferring control to the main Lua application.
In the case of an active update, **the ESP is restarted** so resource cleanup on
completion is not an issue. The provisioning dialogue is signed, so the host
provisioning service and the protocol are trusted, with the provisioning service driving
2019-02-17 19:26:29 +01:00
the process. This greatly simplifies the `luaOTA` client coding as this is a simple
responder, which actions simple commands such as:
- download a file,
2017-10-28 23:22:34 +02:00
- download and compile file,
- upload a file
- rename (or delete) a file
2019-02-17 19:26:29 +01:00
with the ESP being rebooted on completion of the updates to the SPIFFS. Hence in
practice the ESP boots into one one two modes:
- _normal execution_ or
2017-10-28 23:22:34 +02:00
- _OTA update_ followed by reboot and normal execution.
Note that even though NodeMCU follows the Lua convention of using the `lua` and `lc`
extensions respectively for source files that need to be compiled, and for pre-compiled
files, the Lua loader itself only uses the presence of a binary header to determine the
file mode. Hence if the `init.lua` file contains pre-compiled content, and similarly all
loaded modules use pre-compiled lc files, then the ESP can run in production mode
_without needing to invoke the compiler at all_.
The simplest strategy for the host provisioning service is to maintain a reference
source directory on the host (per ESP module). The Lua developer can maintain this under
**git** or equivalent and make any changes there, so that synchronisation of the ESP
will be done automatically on reboot.
### init.lua
2019-02-17 19:26:29 +01:00
This is typically includes a single line:
2017-10-28 23:22:34 +02:00
```Lua
require "LuaOTA.check"
```
2019-02-17 19:26:29 +01:00
however if the configuration is incomplete then this can be aborted as manual process
2017-10-28 23:22:34 +02:00
by entering the manual command through the UART
```Lua
tmr.stop(0); require "luaOTA.check":_init {ssid ="SOMESID" --[[etc. ]]}
```
2019-02-17 19:26:29 +01:00
where the parameters to the `_init` method are:
2017-10-28 23:22:34 +02:00
2019-02-17 19:26:29 +01:00
- `ssid` and `spwd`. The SSID of the Wifi service to connect to, together with its
2017-10-28 23:22:34 +02:00
password.
- `server` and `port`. The name or IP address and port of the provisioning server.
2019-02-17 19:26:29 +01:00
- `secret`. A site-specific secret shared with the provisioning server for MD5-based
2017-10-28 23:22:34 +02:00
signing of the protocol messages.
- `leave`. If true the STA service is left connected otherwise the wifi is shutdown
2019-02-17 19:26:29 +01:00
- `espip`,`gw`,`nm`,`ns`. These parameters are omitted if the ESP is using a DHCP
service for IP configuration, otherwise you need to specify the ESP's IP, gateway,
2017-10-28 23:22:34 +02:00
netmask and default nameserver.
If the global `DEBUG` is set, then LuaOTA will also dump out some diagnostic debug.
### luaOTA.check
This only has one public method: `_init` which can be called with the above parameters.
However the require wrapper in the check module also posts a call to `self:_init()` as a
new task. This new task function includes a guard to prevent a double call in the case
2019-02-17 19:26:29 +01:00
where the binding require includes an explicit call to `_init()`
2017-10-28 23:22:34 +02:00
Any provisioning changes results in a reboot, so the only normal "callback" is to invoke
the application entry method as defined in `config.json` using a `node.task.post()`
### luaOTAserver.lua
This is often tailored to specific project requirements, but a simple example of a
provisioning server is included which provides the corresponding server-side
functionality. This example is coded in Lua and can run on any development PC or server
that supports Lua 5.1 - 5.3 and the common modules `socket`, `lfs`, `md5` and `cjson`.
It can be easily be used as the basis of one for your specific project needs.
Note that even though this file is included in the `luaOTA` subdirectory within Lua
examples, this is designed to run on the host and should not be included in the
ESP SPIFFS.
## Implementation Notes
- The NodeMCu build must include the following modules: `wifi`, `net`, `file`, `tmr`,
2019-02-17 19:26:29 +01:00
`crypto` and`sjason`.
2017-10-28 23:22:34 +02:00
- This implementation follow ephemeral practices, that it is coded to ensure that all
2019-02-17 19:26:29 +01:00
resources used are collected by the Lua GC, and hence the available heap on
2017-10-28 23:22:34 +02:00
application start is the same as if luaOTA had not been called.
- Methods in the `check` file are static and inherit self as an upvalue.
2019-02-17 19:26:29 +01:00
- In order to run comfortably within ESP resources, luaOTA executes its main
2017-10-28 23:22:34 +02:00
functionality as a number of overlay methods. These are loaded dynamically (and largely
transparently) by an `__index` metamethod.
- Methods starting with a "_" are call-once and return the function reference
2019-02-17 19:26:29 +01:00
- All others are also entered in the self table so that successive calls will use
2017-10-28 23:22:34 +02:00
the preloaded function. The convention is that any dynamic function is called in object
form so they are loaded and executed with self as the first parameter, and hence are
called using the object form self:someFunc() to get the context as a parameter.
- Some common routines are also defined as closures within the dynamic methods
2019-02-17 19:26:29 +01:00
- This coding also makes a lot of use of tailcalls (See PiL 6.3) to keep the stack size
to a minimum.
2017-10-28 23:22:34 +02:00
2019-02-17 19:26:29 +01:00
- The update process uses a master timer in `tmr` slot 0. The index form is used here
2017-10-28 23:22:34 +02:00
in preference to the object form because of the reduced memory footprint. This also
allows the developer to abort the process early in the boot sequence by issuing a
`tmr.stop(0)` through UART0.
2019-02-17 19:26:29 +01:00
- The command protocol is unencrypted and uses JSON encoding, but all exchanges are
2017-10-28 23:22:34 +02:00
signed by a 6 char signature taken extracted from a MD5 based digest across the JSON
string. Any command which fails the signature causes the update to be aborted. Commands
are therefore regarded as trusted, and this simplifies the client module on the ESP.
2019-02-17 19:26:29 +01:00
- The process can support both source and compiled code provisioning, but the latter
is recommended as this permits a compile-free runtime for production use, and hence
2017-10-28 23:22:34 +02:00
minimises the memory use and fragmentation that occurs as a consequence of compilation.
2019-02-17 19:26:29 +01:00
- In earlier versions of the provisioning service example, I included `luaSrcDiet` but
2017-10-28 23:22:34 +02:00
this changes the line numbering which I found real pain for debugging, so I now just
include a simple filter to remove "--" comments and leading and trailing whitespace if
the source includes a `--SAFETRIM` flag. This typically reduced the size of lua files
transferred by ~30% and this also means that developers have no excuse for not properly
commenting their code!
2019-02-17 19:26:29 +01:00
- The chip ID is included in the configuration identification response to permit the
2017-10-28 23:22:34 +02:00
provisioning service to support different variants for different ESP8266 chips.
2019-02-17 19:26:29 +01:00
- The (optional update & reboot) operate model also has the side effect that the
2017-10-28 23:22:34 +02:00
`LuaOTA` client can reprovision itself.
2019-02-17 19:26:29 +01:00
- Though the simplest approach is always to do a `luaOTA.check` immediately on reboot,
there are other strategies that could be applied, for example to test a gpio pin or a
flag in RTC memory or even have the application call the require directly (assuming that
there's enough free RAM for it to run and this way avoid the connection delay to the
2017-10-28 23:22:34 +02:00
WiFi.
## Discussion on RAM usage
2019-02-17 19:26:29 +01:00
`luaOTA` also itself serves as a worked example of how to write ESP-friendly
2017-10-28 23:22:34 +02:00
applications.
2019-02-17 19:26:29 +01:00
- The functionality is divided into autoloaded processing chunks using a self
2017-10-28 23:22:34 +02:00
autoloader, so that `self:somefunction()` calls can load new code from flash in
2019-02-17 19:26:29 +01:00
a way that is simple and largely transparent to the application. The autoloader
2017-10-28 23:22:34 +02:00
preferentially loads the `lc` compiled code variant if available.
- The local environment is maintained in a self array, to keep scoping explicit. Note
2019-02-17 19:26:29 +01:00
that since loaded code cannot inherit upvalues, then `self` must be passed to the
2017-10-28 23:22:34 +02:00
function using an object constructor `self:self:somefunction()`, but where the function
2019-02-17 19:26:29 +01:00
can have a self argument then the alternative is to use an upvalue binding. See the
2017-10-28 23:22:34 +02:00
`tmr` alarm call at the end of `_init.lua` as an example:
```Lua
tmr.alarm(0, 500, tmr.ALARM_AUTO, self:_doTick())
```
2019-02-17 19:26:29 +01:00
- The `self:_doTick()` is evaluated before the alarm API call. This autoloads
`luaOTA/_doTick.lc` which stores `self` as a local and returns a function which takes
2017-10-28 23:22:34 +02:00
no arguments; it is this last returned function that is used as the timer callback,
and when this is called it can still access self as an upvalue.
2019-02-17 19:26:29 +01:00
- This code makes a lot of use of locals and upvalues as these are both fast and use
2017-10-28 23:22:34 +02:00
less memory footprint than globals or table entries.
2019-02-17 19:26:29 +01:00
- The lua GC will mark and sweep to reclaim any unreferenced resources: tables,
2017-10-28 23:22:34 +02:00
strings, functions, userdata. So if your code at the end of a processing phase leaves
2019-02-17 19:26:29 +01:00
no variables (directly or indirectly) in _G or the Lua registry, then all of the
resources that were loaded to carry out your application will be recovered by the GC.
2017-10-28 23:22:34 +02:00
In this case heap at the end of a "no provisioning" path is less than 1Kb smaller than
2019-02-17 19:26:29 +01:00
if luaOTA had not been called and this is an artifact of how the lua_registry system
2017-10-28 23:22:34 +02:00
adopts a lazy reuse of registry entries.
- If you find that an enumeration of `debug.getregistry()` includes function references
2019-02-17 19:26:29 +01:00
or tables other than ROMtables, then you have not been tidying up by doing the
appropriate closes or unregister calls. Any such stuck resources can result in a
stuck cascade due to upvalues being preserved in the function closure or entries in a
2017-10-28 23:22:34 +02:00
table.