The IDF-provided VFS resolves several issues:
- The IDF components having a different view of the (virtual) file system
compared to the Lua environment.
- RTOS task/thread safety. Our legacy VFS was only ever safe to use
from the LVM thread, which limited its usability. Upgrading it
would have effectively required a reimplementation of the IDF VFS,
which would have been a bigger task with larger on-going maintenance
issues.
- We're no longer needing to maintain our own SPIFFS component.
- We're no longer needing to maintain our own FATFS component.
- The legacy of the 8266's lack of standard C interface to the file system
is no longer holding us back, meaning that we can use the standard
Lua `io` module rather than the cobbled-together swiss army knife
also known as the file module.
Of course, the downside is that we'll either have to declare a backwards
breakage in regard to the file module, or provide a Lua shim for the old
functions, where applicable.
Also included is some necessary integer type fixups in unrelated code,
which apparently had depended on some non-standard types in either the
SPIFFS or FATFS headers.
A memory leak issue in the sdmmc module was also found and fixed while
said module got switched over to the Espressif VFS.
Module documentation has been updated to match the new reality (and I
discovered in some places it wasn't even matching the old reality).
Search-and-replace considered harmful. I completely missed the need to
explicitly declare "fast" tag functions (__xyz) in the mask field to
LROT_BEGIN()/LROT_END() when I brought over the 5.1+5.3 support.
Without those flags set properly, the LVM doesn't even bother going
looking for those methods, which in this case led to garbage collection
not calling the __gc functions, among other horrible things.
Mea culpa.
The IDF provides all we need these days, and the old driver was just
needlessly conflicting with the IDF settings and setup.
This also simplifies our uart input path as we no longer need to
duplicate the raw byte handling for when "run_input" is false.
Changes have been kept to a minimum, but a serious chunk of work was
needed to move from 8266isms to IDFisms.
Some things got refactored into components/lua/common, in particular
the LFS location awareness.
As part of this work I also evicted our partition table manipulation
code, as with the current IDF it kept breaking checksums and rendering
things unbootable, which is the opposite of helpful (which was the
original intent behind it).
The uart module got relocated from base_nodemcu to the modules component
properly, after I worked out how to force its inclusion using Kconfig alone.
The uzlib and parts of Lua had to be switched over to use the
C standard int types, as their custom typedefs conflicted with
RISC-V toolchain provided typedefs.
UART console driver updated to do less direct register meddling
and use the IDF uart driver interface for setup. Still using our
own ISR rather than the default driver ISR. Down the line we
might want to investigate whether the IDF ISR would be a better
fit.
Lua C modules have been split into common and ESP32/ESP32-S
specific ones. In the future there might also be ESP32-C3
specific modules, which would go into components/modules-esp32c3
at that point.
Our old automatic fixup of flash size has been discarded as it
interferes with the checksumming done by the ROM loader and
results in unbootable systems. The IDF has already taken on
this work via the ESPTOOL_FLASHSIZE_DETECT option, which handles
this situation properly.
Only compile-tested so far.
Of note is that the WiFi auto-connect (flag) functionality has been removed
from the IDF, and as a follow-on so has the "auto" field in the wifi config.
On the Ethernet side, support for the TLK110 PHY seems to have been removed,
but on the other hand there is now new support for several others.
Some parts dry-coded in the disabled modules; to be fixed when sorting out
the deprecated/removed APIs used in said modules.
Still untested beyond compile/linking.
Using the NODEMCU_ namespace prefix makes it obvious that these are not
part of Lua proper (contrast, e.g., LUA_BUILTIN_STRING). Using
"CMODULE" gives us room to differentiate between modules whose
implementation is in C and whose implemenation is in Lua ("LMODULE").
The ESP8266 branch can adopt the same convention when it moves to
Kconfig; see https://github.com/nodemcu/nodemcu-firmware/issues/3130
Manifests as multiple definition of esp_event_send during compilation, as bthci triggers inclusion of `event_loop.c`. Also improved lbth_init() to support BTDM or BLE_ONLY controller modes.
* Fix uart regressions & bugs.
Using `uart.on()` with a search character was broken in that it did
not invoke the callback on a full UART buffer as documented. Logic reworked
to match docs again.
Fixed memory leak on `task_post()` failure (eep!).
Improved logic to attempt to coalesce input bytes to reduce the number of
`task_post()` slots used up by the platform uart.
Finally, added a semaphore to prevent the platform uart from overrunning
the `task_post()` slots all the time on high baud rates (e.g. 1mbit).
With the semaphore in there, the LVM RTOS task gets a chance to actually
process the received data and free up a `task_post()` slot or two.
The above mentioned read coalescing then allows the platform uart to
immediately catch up.
Also added an error log message if the `task_post()` actually does fail.
* Don't cache the uart delims.
Doing so makes reconfiguring those settings from within the callback not
take effect until the currently buffered bytes have been processed.
To avoid races between the lwIP callbacks (lwIP RTOS task) and the Lua
handlers (LVM RTOS task), the data flow and ownership has been simplified
and cleaned up.
lwIP callbacks now have no visibility of the userdata struct. They are
limited to creating small event objects and task_post()ing them over
to the LVM "thread", passing ownership in doing so. The shared identifier
then becomes the struct netconn*.
On the LVM side, we keep a linked list of active userdata objects. This
allows us to retrieve the correct userdata when we get an event with
a netconn pointer. Because this list is only ever used within the LVM
task, no locking is necessary.
The old approach of stashing a userdata pointer into the 'socket' field
on the netconn has been removed entirely, as this was both not
thread/RTOS-task safe, and also interfered with the IDFs internal use
of the socket field (even when using only the netconn layer). As an
added benefit, this removed the need for all the SYS_ARCH_PROTECT()
locking stuff.
The need to track receive events before the corresponding userdata object
has been established has been removed by virtue of not reordering the
"accept" and the "recv" events any more (previously accepts were posted
with medium priority, while the receives where high priority, leading
to the observed reordering and associated headaches).
The workaround for IDF issue 784 has been removed as it is now not needed
and is in fact directly harmful as it results in a double-free. Yay for
getting rid of old workarounds!
DNS resolution code paths were merged for the two instances of "socket"
initiated resolves (connect/dns functions).
Also fixed an instance of using a stack variable for receiving the resolved
IP address, with said variable going out of scope before the DNS resolution
necessarily completed (hello, memory corruption!).
Where possible, moved to use the Lua allocator rather than plain malloc.
Finally, the NodeMCU task posting mechanism got a polish and an adjustment.
Given all the Bad(tm) that tends to happen if something fails task posting,
I went through a couple of iterations on how to avoid that. Alas, the
preferred solution of blocking non-LVM RTOS tasks until a slot is free
turned out to not be viable, as this easily resulted in deadlocks with the
lwIP stack. After much deliberation I settled on increasing the number of
available queue slots for the task_post() mechanism, but in the interest
of user control also now made it user configurable via Kconfig.
The netbuf_data() function only returns data from the first pbuf in the
chain. We need to use netbuf_first() and netbuf_next() to walk the
pbuf chain and act on each in turn.
* Leaner, meaner crypto module; now with HMAC
Based on my testing, mbedtls pulls in all its algorithm regardless of
whether the NodeMCU crypto module was using them or not. As such, the
space savings from omitting algorithms were only in the tens of bytes.
By switching to using the mbedtls generic message digest interface, the
crypto module itself could be shrunk in size and complexity. Despite
adding support for HMAC on all algorithms (plus including RIPEMD160),
this version is 330 bytes smaller.
* Updated crypto module docs.
* Removed superfluous brackets in crypto docs.
Copy-paste considered harmful... >.>
With the IDF asserting full control over the linker scripts and insisting on
the application description being the first entry in the .flash.rodata
section, or previous method of doing link-time arrays stopped working.
Why? Because the build patched in a SHA256 digest straight into our arrays.
With the limited language of the gcc linker scripts I could find no other
way of getting it in cleanly.
The IDF "linker fragments" support can not be made to work for our needs:
- no support for setting alignment before including objects
- no support for declaring symbols
- no support for adding our terminating zeros
- insists on grouping objects by lib rather than by declared grouping,
which means we could at most have a single link-time-array using
the IDF mechanism
- also does not like underscores in section names, but that's just an
annoyance
So, the least bad option that I could come up with was to use a project-wide
makefile snippet to add a target in-between the IDF's generation of the
esp32.project.ld file, and the linking of our NodeMCU.elf. In this target
we read in the esp32.project.ld linker script, check whether we have our
arrays in there, and if not rewrites the linker script.
Oh, and the esp32.project.ld file only came into existence on the IDF 3.3
branch, so I had to change up the IDF to the latest release/3.3 as well.
I would've preferred a stable tag, but the v3.3-beta3 had a really nasty
regression for us (can't add partition entry), so that was a no-go.
* ESP32: Added pulsecnt module
The pulsecnt module let's you use the ESP32's pulse counter capabilities from Lua.
* ESP32: Pulsecnt module. Better/faster callback.
Reduced the amount of callback variables to speed things up and shift more logic to Lua than in the C code.
* ESP32: Completed docs for pulsecnt
* ESP32: Final release of pulsecnt
* ESP32: Production release of pulsecnt
* ESP32: Release (tweaked docs)
* ESP32: Pulse Counter Release. Cleaned up .gitignore
* ESP32: Pulse counter release (changed ch1 gpio to int to match ch0)
* ESP32: Add option to set IP/dns config
This commit adds support for setting:
* Hostname
* Static IP / Dns server (Sta mode)
* Changing AP network ip config
* Setting DNS server IP for DHCP
* ESP32: Documentation for setting IP/dns/hostname config
* Documented new functions
* sethostname() now returns true if success
* ESP32: add support for RS485
This commit adds support for switching UART mode to RS485/IRDA.
Also included are patches for memory leaks then handling UART events other than data.
* ESP32: Documentation for uart.setmode()
* ESP32: Add time modules
New time module for manipulating system time/ calendar and controlling SNTP server
* ESP32: Time module documentation & style fixes
* added documentation for time modules
* style fixes as pointed out by @devsaurus
* ESP32: Time module small fixes
* Couple small fixes
* Esp32: Add SJSON module
This adds SJSON module taken directly from master
* ESP32: Fixes for sjson lib
Fixed compilation not including config header, thus braking some of libs functionality
* ESP32: Upgraded SJSON to master
* Adding qrcodegen module for generating QR Codes
* Added LUA_MODULE_QRCODEGEN KConfig
* Changed qrcodegen.encodeText() to use an options table
Created common.h with new option table helper fns.
* Reworked http.c to use new common.h options table APIs
* esp32: Reinstate tmr.now() function using esp_timer_get_time()
Also added larger limit before wrapping if Lua's number type supports
it.
* Renamed tmr.now() to node.uptime()
Added second return value to indicate timer wrap
Bernd Meyer
Johny Mattsson
Marcel Stör
Johny Mattsson
tomsci
jmdasnoy
Kit Klein
Kit Klein
Nathaniel Wesley Filardo
David Thornley
Javier Peletier
f4grx
John Lauer
Arnim Läuger
Skirmantas Lauzikas
Kevin Leung