nodemcu-firmware/docs/extn-developer-faq.md

# Extension Developer FAQ

**# # # Work in Progress # # #**

### Changes in IDF compared to non-OS/RTOS SDK
This is a non-exhaustive list, obviously, but some key points are:

  - **No more `c_types.h`**. Standard C types are finally the norm.
    - `stdint.h` for all your `[u]intX_t` needs
    - `stdbool.h` for bool (note `true`/`false` vs old `TRUE`/`FALSE`)
    - `stddef.h` for `size_t`

  - **A real C library**. All the `os_`, `ets_` and `c_` prefixes for
    standard library functions are gone (except for the special case
    c_getenv, for now). stdout/stdin are wired up to the UART0 console.
    The NodeMCU vfs layer is not currently hooked up to the C library,
    but that would be a nice thing to do.

  - **Everything builds on at least C99 level**, with plenty of warnings
    enabled. Fix the code so it doesn't produce warnings - don't turn
    off the warnings! Yes, there *may* be exceptions, but they're rare.

  - **`user_config.h` is no more**. All configuration is now handled
    via Kconfig (`make menuconfig` to configure). From the developer
    perspective, simply include `sdkconfig.h` and test the
    corresponding CONFIG_YOUR_FEATURE macro. The `platform.h` header
    is guaranteed to include `sdkconfig.h`, btw.

  - **`user_modules.h` is also gone**. Module selection is now done
    via Kconfig. Rather than adding a #define to `user_modules.h`,
    add an option in `components/modules/Kconfig` of the form
    `LUA_MODULE_XYZ`, and the existing `NODEMCU_MODULE()` macros
    will take care of the rest. Example Kconfig entry:
    ```
    config LUA_MODULE_XYZ
      bool "Xyz module"
      default "y"
      help
          Includes the XYZ module. Provides features X, Y and Z.
    ```

  - **Preemptive multithreading**. The network stack and other drivers
    now run in their own threads with private stacks. This makes for
    a more robust architecture, but does mean proper synchronization
    *MUST* be employed between the threads. This includes between
    API callbacks and main Lua thread.
    Note that many API callbacks have turned into events in the IDF,
    and NodeMCU handle those in the context of the main Lua thread.

  - **Logical flash partitions**. Rather than hardcoding assumptions of
    flash area usage, there is now an actual logical partition table
    kept on the flash.

### NodeMCU task paradigm

NodeMCU uses a task based approach for scheduling work to run within
the Lua VM. This interface is defined in `task/task.h`, and comprises
three aspects:
  - Task registration (via `task_getid()`)
  - Task posting (via `task_post()` and associated macros)
  - Task processing (via `task_pump_messages()`)

This NodeMCU task API is designed to complement the Lua library model, so
that a library can declare one or more task handlers and that both ISRs
and library functions can then post a message for delivery to a task handler.

Note that NodeMCU tasks must not be confused with the FreeRTOS tasks.
A FreeRTOS task is fully preemptible thread of execution managed by the
OS, while a NodeMCU task is a non-preemptive\* a callback invoked by the
Lua FreeRTOS task. It effectively implements cooperative multitasking.
To reduce confusion, from here on FreeRTOS tasks will be referred to as
threads, and NodeMCU tasks simply as tasks.  Most NodeMCU developers
will not need to concern themselves with threads unless they're doing
low-level driver development.

The Lua runtime is NOT reentrant, and hence any code which calls any
Lua API must run within the Lua task context. _ISRs, other threads and API
callbacks must not access the Lua API or Lua resources._ This typically
means that messages need to be posted using the NodeMCU task API for
handling within the correct thread context. Depending on the scenario,
data may need to be put on a FreeRTOS queue to transfer ownership safely
between the threads, or posted directly across with the NodeMCU task API.

The application has no control over the relative time ordering of
tasks and API callbacks, and no assumptions can be made about whether a
task and any posted successors will run consecutively.

\*) Non-preemptive in the sense that other NodeMCU tasks will not preempt it.
The RTOS may still preempt the task to run ISRs or other threads.

#### Task registration
Each module which wishes to receive events and process those within the
context of the Lua thread need to register their callback. This is done
by calling `task_get_id(module_callback_fn)`. A non-zero return value
indicates successful registration, and the returned handle can be used
to post events to this task. The task registration is typically done in
the module\_init function.

#### Task posting
To signal a task, a message is posted to it using
`task_post(prio, handle, param)` or the helper macros `task_post_low()/task_post_medium()/task_post_high()`. Each message carries a single parameter value
of type `intptr_t` and may be used to carry either pointers or raw values.
Each task defines its own schema depending on its needs.

Note that the message queues can theoretically fill up, and this is reported
by a `false` return value from the `task_post()` call. In this case no
message was posted. In most cases nothing much can be done, and the best
approach may be to simply ignore the failure and drop the data. Be careful
not to accidentally leak memory in this circumstance however! A good
habit would be to do something like this:
```
  char *buf = malloc (len);
  // do something with buf...
  if (!task_post_medium(handle, buf))
    free (buf);
```

The `task_post*()` function and macros can be safely called from any ISR or
thread.

#### Task processing and priorities
The Lua runtime executes in a single thread, and at its root level runs
the task message processing loop. Task messages arrive on three queues,
one for each priority level. The queues are services in order of their
priority, so while a higher-priority queue contains messages, no
lower-priority messages will be delivered. It is thus quite possible to
jam up the Lua runtime by continually posting high-priority messages.
Don't do that.

Unless there are particular reasons not to, messages should be posted
with medium priority. This is the friendly, cooperative level. If something
is time sensitive (e.g. audio buffer running low), high priority may be
warranted. The low priority level is intended for background processing
during otherwise idle time.


### Processing system events
The IDF is quite flexible in how system events may be handled, and NodeMCU
takes advantage of this to get all system events handled by the main Lua
thread. Event listening registration is very similar to module registration,
and happens at link-time. The following snippet shows how to use this:
```
#include "nodemcu_esp_event.h"

static void on_got_ip (const system_event_t *evt)
{
  // Do stuff, maybe invoke a Lua callback
}

// Register for the event SYSTEM_EVENT_STA_GOT_IP (see esp_event.h for list).
NODEMCU_ESP_EVENT(SYSTEM_EVENT_STA_GOT_IP, on_got_ip);
```


### Memory allocation in modules
There are three main ways of allocating memory when writing modules for
NodeMCU - stack (for temporaries), heap, and Lua heap. Using the stack
is the same as for other embedded development, i.e. feel free to put
small(ish) temporary objects there, but don't expect to get away with
hundreds of bytes on the stack. Under good circumstances RTOS will detect
a stack overflow and throw an error, under less good circumstances anything
can happen.

Heap allocation is through the standard C malloc and friends. Their use
is best limited to third-party libraries which do not allow for custom
allocators. Also, if dynamic memory allocation is done in a thread other
than the main Lua thread, this is the only available option.

The best way to allocate memory in a module however, is to use the luaM
memory allocation routines. What makes this the best option is that an
allocation made this way may trigger a garbage collection. Where a regular
C malloc might have failed due to out of memory, the Lua allocation can
succeed by virtue of freeing up memory first. The downside of course is
that this is only possible to do while running in the main Lua thread.
Note that memory allocated via e.g. `luaM_malloc()` is not subject to
garbage collection, it behaves just like memory from the C `malloc()`,
and must be explicitly free'd via `luaM_free()`.

A quick example:
```
static int some_func (lua_State *L)
{
  char *buf = luaM_malloc (L, 512);
  int result = calc_using_large_buf (buf, 1, 2, 3);
  lua_pushinteger (L, result);
  luaM_free (buf);
  return 1;
}
```

#### Caution
Do note that `luaM_malloc()` raises a Lua error on allocation failure, and
will exit the calling function right then and there. This can lead to
resource leaks if care is not taken, though in most cases a failure
will result in a Lua panic and require a reboot.

On the upside, there is never a need to test the return value from
`luaM_malloc()` for NULL, as on failure the function does not return.