Initial support for using external components/modules.

CMakeLists.txt

@@ -1,4 +1,11 @@
 cmake_minimum_required(VERSION 3.5)
+
+if(DEFINED ENV{EXTRA_COMPONENT_DIRS})
+  # Turn space-separated, quote-aware environment var into CMake list
+  separate_arguments(
+    EXTRA_COMPONENT_DIRS UNIX_COMMAND "$ENV{EXTRA_COMPONENT_DIRS}")
+endif()
+
 include($ENV{IDF_PATH}/tools/cmake/project.cmake)
 project(nodemcu)
 

docs/build.md

@@ -121,3 +121,28 @@ Partition Table --->
   (components/platform/partitions-2MB.csv) Custom partition CSV file
   (0x10000) Factory app partition offset
 ```
+
+### Using external components
+
+It is possible, and relatively easy, to include external components and modules in NodeMCU. It is not uncommon to have one or more custom modules one wishes to include in the firmware. To enable this NodeMCU leverages the standard IDF `EXTRA_COMPONENT_DIRS` functionality. As such, it is possible to not only add extra Lua C modules, but also other components such as libraries.
+
+To include one (or more) additional IDF components, simply set the `EXTRA_COMPONENT_DIRS` environment variable to the space-separated list of directories of said components. E.g.
+
+```
+export EXTRA_COMPONENT_DIRS="/path/to/mymod /path/to/mylib"
+make menuconfig
+make
+```
+
+To get started, a template directory structure is provided in [extcomp-template/](../extcomp-template) which provides a skeleton for a simple Lua C module, including the build logic in `CMakeLists.txt`, the configuration option in `Kconfig` and the Lua C module code in `mymod.c`. A detailed discussion on the specifics is beyond this document, but the first two are described comprehensively in the [official IDF documentation](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html), and module development is covered in [Programming in NodeMCU](nodemcu-pil.md).
+
+In fact, to quickly try it out it's even possible to include the template itself, as-is:
+
+
+```
+export EXTRA_COMPONENT_DIRS="$PWD/extcomp-template"
+make menuconfig
+make
+```
+
+after which the command `mymod.hello()` is available in the Lua environment.

docs/nodemcu-pil.md

@@ -6,7 +6,7 @@ NodeMCU developers are also able to develop and incorporate their own C modules
 
 Those developers who wish to develop or to modify existing C modules should have access to the LRM, PiL and NRM and familiarise themselves with these references.  These are the primary references; and this document does not repeat this content, but rather provide some NodeMCU-specific information to supplement it.
 
-From a perspective of developing C modules, there is very little difference from that of developing modules in standard Lua.  All of the standard Lua library modules (`bit`, `coroutine`, `debug`, `math`, `string`, `table`, `utf8`) use the C API for Lua and the NodeMCU versions have been updated to use NRM extensions. so their source code is available for browsing and using as style template (see the corresponding `lXXXlib.c` file in GitHub [NodeMCU lua53](../app/lua53) folder).
+From a perspective of developing C modules, there is very little difference from that of developing modules in standard Lua.  All of the standard Lua library modules (`bit`, `coroutine`, `debug`, `math`, `string`, `table`, `utf8`) use the C API for Lua and the NodeMCU versions have been updated to use NRM extensions. so their source code is available for browsing and using as style template (see the corresponding `lXXXlib.c` file in GitHub [NodeMCU lua53](../components/lua/lua-5.3) folder).
 
 The main functional change is that NodeMCU supports a read-only subclass of the `Table` type, known as a **`ROTable`**, which can be statically declared within the module source using static `const` declarations.  There are also limitations on the valid types for ROTable keys and value in order to ensure that these are consistent with static declaration; and hence ROTables are stored in code space (and therefore in flash memory on the IoT device). Hence unlike standard Lua tables, ROTables do not take up RAM resources. 
 
@@ -18,9 +18,9 @@ The `NODEMCU_MODULE` macro is used in each module to register it in an entry in
 -  All `ROM` entries will resolve globally
 -  The Lua runtime scans the `ROMentry` ROTable during its start up, and it will execute any non-NULL `CFunction` values in this table.  This enables C modules to hook in any one-time start-up functions if they are needed.
 
-Note that the standard `make` will include any modules found in the `app/modules` folder within a firmware build _if_ the corresponding `LUA_USE_MODULES_modname` macro has been defined. These defines are conventionally set in a common include file `user_modules.h`, and this practice is mandated for any user-submitted modules that are added to to the NodeMCU distribution.  However, this does not prevent developers adding their own local modules to the `app/modules` folder and simply defining the corresponding  `LUA_USE_MODULES_modname` inline.
+For a module to be included in the build, it has to be enabled in the sdkconfig file (e.g. via running `make menuconfig`). Some modules are enabled by default. Between compile time macros based on the sdkconfig and linker processing only the enabled modules are actually included into the firmware.
 
-This macro + linker approach renders the need for `luaL_reg` declarations and use of `luaL_openlib()` unnecessary, and these are not permitted in project-adopted `app/modules` files.
+This macro + linker approach renders the need for `luaL_reg` declarations and use of `luaL_openlib()` unnecessary, and these are not permitted in project-adopted `components/modules` files.
 
 Hence a NodeMCU C library module typically has a standard layout that parallels that of the standard Lua library modules and uses the same C API to access the Lua runtime:
 
@@ -33,10 +33,10 @@ Hence a NodeMCU C library module typically has a standard layout that parallels
    -  Whilst the ROTable search algorithm is a simply linear scan of the ROTable entries, the runtime also maintains a LRU cache of ROTable accesses, so typically over 95% of ROTable accesses bypass the linear scan and do a direct access to the appropriate entry.
    -  ROTables are also reasonable lightweight and well integrated into the Lua runtime, so the normal metamethod processing works well.  This means that developers can use the `__index` method to implement other key and value typed entries through an index function.
 -  NodeMCU modules are intended to be compilable against both our Lua 5.1 and Lua 5.3 runtimes.  The NRM discusses the implications and constraints here.  However note that:
-   -  We have back-ported many new Lua 5.3 features into the NodeMCU Lua 5.1 API, so in general you can use the 5.3 API to code your modules.  Again the NRM notes the exceptions where you will either need variant code or to decide to limit yourself to the the 5.3 runtime. In this last case the simplest approach is to `#if LUA_VERSION_NUM != 503` to disable the 5.3 content so that 5.1 build can compile and link.   Note that all modules currently in the `app/modules` folder will compile against and execute within both the Lua 5.1 and the 5.3 environments.
+   -  We have back-ported many new Lua 5.3 features into the NodeMCU Lua 5.1 API, so in general you can use the 5.3 API to code your modules.  Again the NRM notes the exceptions where you will either need variant code or to decide to limit yourself to the the 5.3 runtime. In this last case the simplest approach is to `#if LUA_VERSION_NUM != 503` to disable the 5.3 content so that 5.1 build can compile and link.   Note that all modules currently in the `components/modules` folder will compile against and execute within both the Lua 5.1 and the 5.3 environments.
    -  Lua 5.3 uses a 32-bit representation for all numerics with separate subtypes for integer (stored as a 32 bit signed integer) and float (stored as 32bit single precision float).  This achieves the same RAM storage density as Lua 5.1 integer builds without the loss of use of floating point when convenient.  We have therefore decided that there is no benefit in having a separate Integer 5.3 build variant.
  -  We recommend that developers make use of the full set of `luaL_` API calls to minimise code verbosity.  We have also added a couple of registry access optimisations that both simply and improve runtime performance when using the Lua registry for callback support.
     - `luaL_reref()` replaces an existing registry reference in place (or creates a new one if needed).  Less code and faster execution than a `luaL_unref()` plus `luaL_ref()` construct.
     - `luaL_unref2()` does the unref and set the static int hook to `LUA_NOREF`.
 
-Rather than include simple examples of module templates, we suggest that you review the modules in our GitHub repository, such as the [`utf8`](../app/lua53/lutf8lib.c) library.  Note that whilst all of the existing modules in `app/modules` folder compile and work, we plan to do a clean up of the core modules to ensure that they conform to best practice.
+Rather than include simple examples of module templates, we suggest that you review the modules in our GitHub repository, such as the [`utf8`](../components/lua/lua-5.3/lutf8lib.c) library.  Note that whilst all of the existing modules in `components/modules` folder compile and work, we plan to do a clean up of the core modules to ensure that they conform to best practice.

extcomp-template/CMakeLists.txt

@@ -0,0 +1,37 @@
+# Modify this list as necessary to include all your source files.
+set(extmod_srcs
+  "mymod.c"
+)
+
+# If necessary, add items to the PRIV_REQUIRES list below
+idf_component_register(
+  SRCS ${extmod_srcs}
+  INCLUDE_DIRS "." "${CMAKE_CURRENT_BINARY_DIR}"
+  PRIV_REQUIRES
+    "base_nodemcu"
+    "lua"
+    "platform"
+)
+
+# The remainder is boiler-plate glue to get the linker to actually include
+# the modules enabled in Kconfig. No user-serviceable parts inside.
+
+# Match up all the extmod source files with their corresponding Kconfig
+# option in the form NODEMCU_CMODULE_<modname> and if enabled, add a
+# "-u <modname>_module_selected1" option to force the linker to include
+# the module. See components/core/include/module.h for further details on
+# how this works.
+set(extmods_enabled)
+foreach(extmod_src ${extmod_srcs})
+  string(REPLACE ".c" "" module_name ${extmod_src})
+  string(TOUPPER ${module_name} module_ucase)
+  set(mod_opt "CONFIG_NODEMCU_CMODULE_${module_ucase}")
+  if (${${mod_opt}})
+    list(APPEND extmods_enabled ${module_ucase})
+  endif()
+endforeach()
+message("Including the following modules: ${extmods_enabled}")
+
+foreach(mod ${extmods_enabled})
+  target_link_libraries(${COMPONENT_LIB} "-u ${mod}_module_selected1")
+endforeach()

extcomp-template/Kconfig

@@ -0,0 +1,13 @@
+menu "External modules"
+
+    config NODEMCU_CMODULE_MYMOD
+        bool "Mymod module"
+        default "y"
+        help
+            Includes the mymod module. This module is only an example for
+            showing how to use external modules. Note that the config option
+            name has to be prefixed with NODEMCU_CMODULE_ and the suffix
+            has to match the first argument in the NODEMCU_MODULE() macro
+            in the .c file.
+
+endmenu

extcomp-template/mymod.c

@@ -0,0 +1,21 @@
+#include "module.h"
+
+static int lmymod_hello(lua_State *L)
+{
+  if (lua_isnoneornil(L, 1))
+    lua_pushliteral(L, "world");
+
+  lua_getglobal(L, "print");
+  lua_pushliteral(L, "Hello,");
+  lua_pushvalue(L, 1);
+  lua_call(L, 2, 0);
+
+  return 0;
+}
+
+
+LROT_BEGIN(mymod, NULL, 0)
+  LROT_FUNCENTRY(hello,   lmymod_hello)
+LROT_END(mymod, NULL, 0)
+
+NODEMCU_MODULE(MYMOD, "mymod", mymod, NULL);