# Extension Developer FAQ **# # # Work in Progress # # #** ### Changes in IDF compared to non-OS/RTOS SDK This is a non-exhaustive list, obviously, but some key points are: - **No more `c_types.h`**. Standard C types are finally the norm. - `stdint.h` for all your `[u]intX_t` needs - `stdbool.h` for bool (note `true`/`false` vs old `TRUE`/`FALSE`) - `stddef.h` for `size_t` - **A real C library**. All the `os_`, `ets_` and `c_` prefixes for standard library functions are gone (except for the special case c_getenv, for now). stdout/stdin are wired up to the UART0 console. The NodeMCU vfs layer is not currently hooked up to the C library, but that would be a nice thing to do. - **Everything builds on at least C99 level**, with plenty of warnings enabled. Fix the code so it doesn't produce warnings - don't turn off the warnings! Yes, there *may* be exceptions, but they're rare. - **`user_config.h` is no more**. All configuration is now handled via Kconfig (`make menuconfig` to configure). From the developer perspective, simply include `sdkconfig.h` and test the corresponding CONFIG_YOUR_FEATURE macro. The `platform.h` header is guaranteed to include `sdkconfig.h`, btw. - **`user_modules.h` is also gone**. Module selection is now done via Kconfig. Rather than adding a #define to `user_modules.h`, add an option in `components/modules/Kconfig` of the form `LUA_MODULE_XYZ`, and the existing `NODEMCU_MODULE()` macros will take care of the rest. Example Kconfig entry: ``` config LUA_MODULE_XYZ bool "Xyz module" default "y" help Includes the XYZ module. Provides features X, Y and Z. ``` - **Preemptive multithreading**. The network stack and other drivers now run in their own threads with private stacks. This makes for a more robust architecture, but does mean proper synchronization *MUST* be employed between the threads. This includes between API callbacks and main Lua thread. Note that many API callbacks have turned into events in the IDF, and NodeMCU handle those in the context of the main Lua thread. - **Logical flash partitions**. Rather than hardcoding assumptions of flash area usage, there is now an actual logical partition table kept on the flash. ### NodeMCU task paradigm NodeMCU uses a task based approach for scheduling work to run within the Lua VM. This interface is defined in `task/task.h`, and comprises three aspects: - Task registration (via `task_getid()`) - Task posting (via `task_post()` and associated macros) - Task processing (via `task_pump_messages()`) This NodeMCU task API is designed to complement the Lua library model, so that a library can declare one or more task handlers and that both ISRs and library functions can then post a message for delivery to a task handler. Note that NodeMCU tasks must not be confused with the FreeRTOS tasks. A FreeRTOS task is fully preemptible thread of execution managed by the OS, while a NodeMCU task is a non-preemptive\* a callback invoked by the Lua FreeRTOS task. It effectively implements cooperative multitasking. To reduce confusion, from here on FreeRTOS tasks will be referred to as threads, and NodeMCU tasks simply as tasks. Most NodeMCU developers will not need to concern themselves with threads unless they're doing low-level driver development. The Lua runtime is NOT reentrant, and hence any code which calls any Lua API must run within the Lua task context. _ISRs, other threads and API callbacks must not access the Lua API or Lua resources._ This typically means that messages need to be posted using the NodeMCU task API for handling within the correct thread context. Depending on the scenario, data may need to be put on a FreeRTOS queue to transfer ownership safely between the threads, or posted directly across with the NodeMCU task API. The application has no control over the relative time ordering of tasks and API callbacks, and no assumptions can be made about whether a task and any posted successors will run consecutively. \*) Non-preemptive in the sense that other NodeMCU tasks will not preempt it. The RTOS may still preempt the task to run ISRs or other threads. #### Task registration Each module which wishes to receive events and process those within the context of the Lua thread need to register their callback. This is done by calling `task_get_id(module_callback_fn)`. A non-zero return value indicates successful registration, and the returned handle can be used to post events to this task. The task registration is typically done in the module\_init function. #### Task posting To signal a task, a message is posted to it using `task_post(prio, handle, param)` or the helper macros `task_post_low()/task_post_medium()/task_post_high()`. Each message carries a single parameter value of type `intptr_t` and may be used to carry either pointers or raw values. Each task defines its own schema depending on its needs. Note that the message queues can theoretically fill up, and this is reported by a `false` return value from the `task_post()` call. In this case no message was posted. In most cases nothing much can be done, and the best approach may be to simply ignore the failure and drop the data. Be careful not to accidentally leak memory in this circumstance however! A good habit would be to do something like this: ``` char *buf = malloc (len); // do something with buf... if (!task_post_medium(handle, buf)) free (buf); ``` The `task_post*()` function and macros can be safely called from any ISR or thread. #### Task processing and priorities The Lua runtime executes in a single thread, and at its root level runs the task message processing loop. Task messages arrive on three queues, one for each priority level. The queues are services in order of their priority, so while a higher-priority queue contains messages, no lower-priority messages will be delivered. It is thus quite possible to jam up the Lua runtime by continually posting high-priority messages. Don't do that. Unless there are particular reasons not to, messages should be posted with medium priority. This is the friendly, cooperative level. If something is time sensitive (e.g. audio buffer running low), high priority may be warranted. The low priority level is intended for background processing during otherwise idle time. ### Processing system events The IDF is quite flexible in how system events may be handled, and NodeMCU takes advantage of this to get all system events handled by the main Lua thread. Event listening registration is very similar to module registration, and happens at link-time. The following snippet shows how to use this: ``` #include "nodemcu_esp_event.h" static void on_got_ip (const system_event_t *evt) { // Do stuff, maybe invoke a Lua callback } // Register for the event SYSTEM_EVENT_STA_GOT_IP (see esp_event.h for list). NODEMCU_ESP_EVENT(SYSTEM_EVENT_STA_GOT_IP, on_got_ip); ``` ### Memory allocation in modules There are three main ways of allocating memory when writing modules for NodeMCU - stack (for temporaries), heap, and Lua heap. Using the stack is the same as for other embedded development, i.e. feel free to put small(ish) temporary objects there, but don't expect to get away with hundreds of bytes on the stack. Under good circumstances RTOS will detect a stack overflow and throw an error, under less good circumstances anything can happen. Heap allocation is through the standard C malloc and friends. Their use is best limited to third-party libraries which do not allow for custom allocators. Also, if dynamic memory allocation is done in a thread other than the main Lua thread, this is the only available option. The best way to allocate memory in a module however, is to use the luaM memory allocation routines. What makes this the best option is that an allocation made this way may trigger a garbage collection. Where a regular C malloc might have failed due to out of memory, the Lua allocation can succeed by virtue of freeing up memory first. The downside of course is that this is only possible to do while running in the main Lua thread. Note that memory allocated via e.g. `luaM_malloc()` is not subject to garbage collection, it behaves just like memory from the C `malloc()`, and must be explicitly free'd via `luaM_free()`. A quick example: ``` static int some_func (lua_State *L) { char *buf = luaM_malloc (L, 512); int result = calc_using_large_buf (buf, 1, 2, 3); lua_pushinteger (L, result); luaM_free (buf); return 1; } ``` #### Caution Do note that `luaM_malloc()` raises a Lua error on allocation failure, and will exit the calling function right then and there. This can lead to resource leaks if care is not taken, though in most cases a failure will result in a Lua panic and require a reboot. On the upside, there is never a need to test the return value from `luaM_malloc()` for NULL, as on failure the function does not return.