A generic fifo and fifosock wrapper, under telnet and http server (#2650)

* lua_modules/fifo: a generic queue & socket wrapper

One occasionally wants a generic fifo, so here's a plausible
implementation that's reasonably flexible in its usage.

One possible consumer of this is a variant of TerryE's two-level fifo
trick currently in the telnetd example.  Factor that out to fifosock for
more general use.

* lua_examples/telnet: use factored out fifosock

* lua_modules/http: improve implementation

Switch to fifosock for in-order sending and waiting for everything to be
sent before closing.

Fix header callback by moving the invocation of the handler higher

* fifosock: optimistically cork and delay tx

If we just pushed a little bit of data into a fifosock that had idled,
wait a tick (1 ms) before transmitting.  Hopefully, this means that
we let the rest of the system push more data in before we send the first
packet.  But in a high-throughput situation, where we are streaming data
without idling the fifo, there won't be any additional delay and we'll
coalesce during operation as usual.

The fifosocktest mocks up enough of tmr for this to run, but assumes
an arbitrarily slow processor. ;)

docs/lua-modules/fifo.md

@@ -0,0 +1,108 @@
+# FIFO Module
+| Since  | Origin / Contributor  | Maintainer  | Source  |
+| :----- | :-------------------- | :---------- | :------ |
+| 2019-02-10 | [nwf](https://github.com/nwf) | [nwf](https://github.com/nwf) | [fifo.lua](../../lua_modules/fifo/fifo.lua) |
+
+This module provides flexible, generic FIFOs built around Lua tables and
+callback functions.  It is specifically engineered to work well with the
+NodeMCU event-based and memory-constrained environment.
+
+## Constructor
+```lua
+fifo = (require "fifo").new()
+```
+
+## fifo.dequeue()
+#### Syntax
+`fifo:dequeue(k)`
+
+Fetch an element from the fifo and pass it to the function `k`, together with a
+boolean indicating whether this is the last element in the fifo.  If the fifo
+is empty, `k` will not be called and the fifo will enter "immediate dequeue"
+mode (see below).
+
+Assuming `k` is called, ordinarily, `k` will return `nil`, which will cause the
+element given to `k` to be removed from the fifo and the queue to advance.  If,
+however, `k` returns a non-`nil` value, that value will replace the element at
+the head of the fifo.  This may be useful for generators, for example, which
+stand in for several elements.
+
+When `k` returns `nil`, it may also return a boolean as its second result.  If
+that is `false`, processing ends and `fifo:dequeue` returns.  If that is
+`true`, the fifo will be advanced again (i.e. `fifo:dequeue(k)` will be *tail
+called*).  Elements for which `k` returns `nil, true` are called "phantom", as
+they cause the fifo to act as though they were not there.  Phantom elements are
+useful for callback-like behavior as the fifo advances: when `k` sees a phantom
+element, it knows that all prior entries in the fifo have been seen, but the
+phantom does not necessarily know how to generate the next element of the fifo.
+
+#### Returns
+`true` if the queue contained at least one non-phantom entry, `false` otherwise.
+
+## fifo.queue()
+#### Syntax
+`fifo:queue(a,k)`
+
+Enqueue the element `a` onto the fifo.  If `k` is not `nil` and the fifo is in
+"immediate dequeue" mode (whence it starts), immediately pass the first element
+of the fifo (usually, but not necessarily, `a`) to `k`, as if `fifo:dequeue(k)`
+had been called, and exit "immediate dequeue" mode.
+
+## FIFO Elements
+
+The elements stored in the FIFO are simply the integer indices of the fifo
+table itself, with `1` being the head of the fifo.  The depth of the queue for
+a given `fifo` is just its table size, i.e. `#fifo`.  Direct access to the
+elements is strongly discouraged.  The number of elements in the fifo is also
+unlikely to be of interest; especially, decisions about the fifo's emptiness
+should instead be rewritten to use the existing interface, if possible, or may
+peek a bit at the immediate dequeueing state (see below).  See the discussion
+of corking, below, too.
+
+## Immediate Dequeueing
+
+The "immediate dequeue" behavior may seem counterintuitive, but it is very
+useful for the case that `fifo:dequeue`'s `k` arranges for subsequent
+invocations of `fifo:dequeue`, say by scheduling the next invocation of a timer
+or by sending on a socket with an `on("sent")` callback wired to
+`fifo:dequeue`.
+
+Because the fifo enters "immediate dequeue" mode only when `dequeue` has been
+called and the fifo was empty at the time of the call, rather than when the
+fifo *becomes* empty, `fifo:queue` will sometimes not invoke its `k` even if
+the queued element `a` ends up at the front of the fifo.  This, too, is quite
+useful: it ensures that `k` will not be called in contexts where it would
+overlap any ongoing processing of the most-recently dequeued, fifo-emptying
+element.
+
+The immediate deququeing status of the fifo is visible as the `_go` member,
+which may be read (even if said reads are politely discouraged, but on occasion
+it is handy to know) but should never be written.
+
+## Corking
+
+The fifo has no special support for corking (that is, queueing several elements
+which are guaranteed to not be dequeued until some later point, called
+"uncorking").  As one often wants to cork only when the fifo is transitioning
+out of immediate deququeing mode, the existing machinery is generally good
+enough to provide an easy emulation thereof.  While it is typical to pass the
+same `k` to both `:queue` and `:dequeue`, there is nothing necessitating this
+convention.  And so one may, as in the `fifosock` module, use the `:queue`
+`k` to record the transition out of immediate dequeueing mode for later,
+when one wishes to uncork:
+
+```lua
+local corked = false
+fifo:queue(e1, function(e) corked = true ; return e end)
+  -- e1 is now in the fifo, and corked is true if the fifo has exited
+  -- immediate dequeue mode.  e1 will be returned back to the fifo and
+  -- so will not be deququed by the function argument.
+
+  -- We can now queue more elements to the fifo.  These will certainly
+  -- queue behind e1.
+fifo:queue(e2)
+
+  -- If we should have initiated draining the fifo above, we can do so now,
+  -- instead, having built up a backlog as desired.
+if corked then fifo:dequeue(k) end
+```

docs/lua-modules/fifosock.md

@@ -0,0 +1,70 @@
+# fifosock Module
+| Since  | Origin / Contributor  | Maintainer  | Source  |
+| :----- | :-------------------- | :---------- | :------ |
+| 2019-02-10 | [TerryE](https://github.com/TerryE) | [nwf](https://github.com/nwf) | [fifosock.lua](../../lua_modules/fifo/fifosock.lua) |
+
+This module provides a convenient, efficient wrapper around the `net.socket`
+`send` method.  It ensures in-order transmission while striving to minimize
+memory footprint and packet count by coalescing queued strings.  It also serves
+as a detailed, worked example of the `fifo` module.
+
+## Use
+```lua
+ssend = (require "fifosock")(sock)
+
+ssend("hello, ") ssend("world\n")
+```
+
+Once the `sock`et has been wrapped, one should use only the resulting `ssend`
+function in lieu of `sock:send`, and one should not change the
+`sock:on("sent")` callback.
+
+### Advanced Use
+
+In addition to passing strings representing part of the stream to be sent, it
+is possible to pass the resulting `ssend` function *functions*.  These
+functions will be given no parameters, but should return two values:
+
+- A string to be sent on the socket, or `nil` if no output is desired
+- A replacement function, or `nil` if the function is to be dequeued.  Functions
+  may, of course, offer themselves as their own replacement to stay at the front
+  of the queue.
+
+This facility is useful for providing a replacement for the `sock:on("sent")`
+callback channel.  In the fragment below, "All sent" will be `print`ed only
+when the entirety of "hello, world\n" has been successfully sent on the
+`sock`et.
+
+```lua
+ssend("hello, ") ssend("world\n")
+ssend(function() print("All sent") end) -- implicitly returns nil, nil
+```
+
+This facility is also useful for *generators* of the stream, roughly akin to
+`sendfile`-like primitives in larger systems.  Here, for example, we can stream
+SPIFFS data across the network without ever holding a large amount in RAM.
+
+```lua
+local function sendfile(fn)
+ local offset = 0
+ local function send()
+  local f = file.open(fn, "r")
+  if f and f:seek("set", offset) then
+    r = f:read()
+    f:close()
+    if r then
+      offset = offset + #r
+      return r, send
+    end
+  end
+  -- implicitly returns nil, nil and falls out of the stream
+ end
+ return send, function() return offset end
+end
+
+local fn = "test"
+ssend(("Sending file '%s'...\n"):format(fn))
+dosf, getsent = sendfile(fn)
+ssend(dosf)
+ssend(("Sent %d bytes from '%s'\n"):format(getsent(), fn))
+```

docs/lua-modules/httpserver.md

@@ -32,7 +32,8 @@ Function to start HTTP server.
 #### Notes
 Callback function has 2 arguments: `req` (request) and `res` (response). The first object holds values:
 
-- `conn`: `net.socket` sub module
+- `conn`: `net.socket` sub module.  **DO NOT** call `:on` or `:send` on this
+  object.
 - `method`: Request method that was used (e.g.`POST` or `GET`)
 - `url`: Requested URL
 - `onheader`: value to setup handler function for HTTP headers like `content-type`. Handler function has 3 parameters:

lua_examples/telnet/telnet.lua

@@ -26,113 +26,28 @@ would exceed 256 bytes. Once over this threashold, the contents of the FIFO are
 concatenated into a 2nd level FIFO entry of upto 256 bytes, and the 1st level FIFO
 cleared down to any residue.
 
-The sender dumps the 2nd level FIFO aggregating records up to 1024 bytes and once this
-is empty dumps an aggrate of the 1st level.
-
 ]]
 local node, table, tmr, wifi, uwrite,     tostring = 
       node, table, tmr, wifi, uart.write, tostring
 
 local function telnet_listener(socket) 
-  local insert,       remove,       concat,       heap,      gc   = 
+  local queueLine = (require "fifosock")(socket)
-        table.insert, table.remove, table.concat, node.heap, collectgarbage
-  local fifo1, fifo1l, fifo2, fifo2l = {}, 0, {}, 0                  
-  local s -- s is a copy of the TCP socket if and only if sending is in progress
-
-  local wdclr, cnt = tmr.wdclr, 0
-  local function debug(fmt, ...)
-    if (...) then fmt = fmt:format(...) end
-    uwrite(0, "\r\nDBG: ",fmt,"\r\n" )
-    cnt = cnt + 1
-    if cnt % 10 then wdclr() end
-  end
-
-  local function flushGarbage()
-    if heap() < 13440 then gc() end
-  end
-
-  local function sendLine()
- -- debug("entering sendLine")  
-    if not s then return end
-
-    if fifo2l + fifo1l == 0 then -- both FIFOs empty, so clear down s
-        s = nil     
-     -- debug("Q cleared")        
-        return
-    end
-
-    flushGarbage()
-  
-    if #fifo2 < 4 then -- Flush FIFO1 into FIFO2
-      insert(fifo2,concat(fifo1))
-   -- debug("flushing %u bytes / %u recs of FIFO1 into FIFO2[%u]", fifo1l, #fifo1, #fifo2) 
-      fifo2l, fifo1, fifo1l = fifo2l + fifo1l, {}, 0
-    end
-
-    -- send out first 4 FIFO2 recs (or all if #fifo2<5)  
-    local rec =  remove(fifo2,1)        .. (remove(fifo2,1) or '') ..
-                (remove(fifo2,1) or '') .. (remove(fifo2,1) or '')
-    fifo2l = fifo2l - #rec
-
-    flushGarbage()
-    s:send(rec)
- -- debug( "sending %u bytes (%u buffers remain)\r\n%s ", #rec, #fifo2, rec)
-  end
-  local F1_SIZE = 256
-  local function queueLine(str)
-    -- Note that this algo does work for strings longer than 256 but it is sub-optimal
-    -- as it does string splitting, but this isn't really an issue IMO, as in practice 
-    -- fields of this size are very infrequent.
- 
- -- debug("entering queueLine(l=%u)", #str)  
-
-    while #str > 0 do  -- this is because str might be longer than the packet size!
-      local k, l = F1_SIZE - fifo1l, #str
-      local chunk
-
-      -- Is it time to batch up and flush FIFO1 into a new FIFO2 entry? Note that it's
-      -- not worth splitting a string to squeeze the last ounce out of a buffer size.
-
-   -- debug("#fifo1 = %u, k = %u, l = %u", #fifo1, k, l) 
-      if #fifo1 >= 32 or (k < l and k < 16) then 
-        insert(fifo2, concat(fifo1))
-     -- debug("flushing %u bytes / %u recs of FIFO1 into FIFO2[%u]", fifo1l, #fifo1, #fifo2) 
-        fifo2l, fifo1, fifo1l, k = fifo2l + fifo1l, {}, 0, F1_SIZE
-      end
-
-      if l > k+16 then -- also tolerate a size overrun of 16 bytes to avoid a split 
-        chunk, str = str:sub(1,k), str:sub(k+1)
-      else
-        chunk, str = str, ''
-      end
-  
-   -- debug("pushing %u bytes into FIFO1[l=%u], %u bytes remaining", #chunk, fifo1l, #str) 
-      insert(fifo1, chunk)
-      fifo1l = fifo1l + #chunk
-    end
-
-    if not s and socket then
-      s = socket
-      sendLine()      
-    else
-      flushGarbage()
-    end
-
-  end
 
   local function receiveLine(s, line)
- -- debug( "received: %s", line) 
     node.input(line)
   end
 
   local function disconnect(s)
-    fifo1, fifo1l, fifo2, fifo2l, s = {}, 0, {}, 0, nil
+    socket:on("disconnection", nil)
+    socket:on("reconnection", nil)
+    socket:on("connection", nil)
+    socket:on("receive", nil)
+    socket:on("sent", nil)
     node.output(nil)
   end
 
   socket:on("receive",       receiveLine)
   socket:on("disconnection", disconnect)
-  socket:on("sent",          sendLine)
   node.output(queueLine, 0)
   print(("Welcome to NodeMCU world (%d mem free, %s)"):format(node.heap(), wifi.sta.getip()))
 end

lua_modules/fifo/fifo.lua

@@ -0,0 +1,45 @@
+-- A generic fifo module.  See docs/lua-modules/fifo.md for use examples.
+
+local tr, ti = table.remove, table.insert
+
+-- Remove an element and pass it to k, together with a boolean indicating that
+-- this is the last element in the queue; if that returns a value, leave that
+-- pending at the top of the fifo.
+--
+-- If k returns nil, the fifo will be advanced.  Moreover, k may return a
+-- second result, a boolean, indicating "phantasmic" nature of this element.
+-- If this boolean is true, then the fifo will advance again, passing the next
+-- value, if there is one, to k, or priming itself for immediate execution at
+-- the next call to queue.
+--
+-- If the queue is empty, do not invoke k but flag it to enable immediate
+-- execution at the next call to queue.
+--
+-- Returns 'true' if the queue contained at least one non-phantom entry,
+-- 'false' otherwise.
+local function dequeue(q,k)
+  if #q > 0
+   then
+     local new, again = k(q[1], #q == 1)
+     if new == nil
+       then tr(q,1)
+            if again then return dequeue(q, k) end -- note tail call
+       else q[1] = new
+     end
+     return true
+   else q._go = true ; return false
+  end
+end
+
+-- Queue a on queue q and dequeue with `k` if the fifo had previously emptied.
+local function queue(q,a,k)
+  ti(q,a)
+  if k ~= nil and q._go then q._go = false; dequeue(q, k) end
+end
+
+-- return a table containing just the FIFO constructor
+return {
+  ['new'] = function()
+    return { ['_go'] = true ; ['queue'] = queue ; ['dequeue'] = dequeue }
+   end
+}

lua_modules/fifo/fifosock.lua

@@ -0,0 +1,134 @@
+-- Wrap a two-staged fifo around a socket's send; see
+-- docs/lua-modules/fifosock.lua for more documentation.
+--
+-- See fifosocktest.lua for some examples of use or tricky cases.
+--
+-- Our fifos can take functions; these can be useful for either lazy
+-- generators or callbacks for parts of the stream having been sent.
+
+local BIGTHRESH = 256   -- how big is a "big" string?
+local SPLITSLOP = 16    -- any slop in the big question?
+local FSMALLLIM = 32    -- maximum number of small strings held
+local COALIMIT  = 3
+
+local concat = table.concat
+local insert = table.insert
+local gc     = collectgarbage
+
+local fifo = require "fifo"
+
+return function(sock)
+  -- the two fifos
+  local fsmall, lsmall, fbig = {}, 0, fifo.new()
+
+  -- ssend last aggregation string and aggregate count
+  local ssla, sslan = nil, 0
+  local ssend  = function(s,islast)
+    local ns = nil
+
+    -- Optimistically, try coalescing FIFO dequeues.  But, don't try to
+    -- coalesce function outputs, since functions might be staging their
+    -- execution on the send event implied by being called.
+
+    if type(s) == "function" then
+      if sslan ~= 0 then
+        sock:send(ssla)
+        ssla, sslan = nil, 0; gc()
+        return s, false -- stay as is and wait for :on("sent")
+      end
+      s, ns = s()
+    elseif type(s) == "string" and sslan < COALIMIT then
+      if sslan == 0
+       then ssla, sslan = s, 1
+       else ssla, sslan = ssla .. s, sslan + 1
+      end
+      if islast then
+        -- this is shipping; if there's room, steal the small fifo, too
+        if sslan < COALIMIT then
+          sock:send(ssla .. concat(fsmall))
+          fsmall, lsmall = {}, 0
+        else
+          sock:send(ssla)
+        end
+        ssla, sslan = "", 0; gc()
+        return nil, false
+      else
+        return nil, true
+      end
+    end
+
+    -- Either that was a function or we've hit our coalescing limit or
+    -- we didn't ship above.  Ship now, if there's something to ship.
+    if s ~= nil then
+      if sslan == 0 then sock:send(s) else sock:send(ssla .. s) end
+      ssla, sslan = nil, 0; gc()
+      return ns or nil, false
+    elseif sslan ~= 0 then
+      assert (ns == nil)
+      sock:send(ssla)
+      ssla, sslan = nil, 0; gc()
+      return nil, false
+    else
+      assert (ns == nil)
+      return nil, true
+    end
+  end
+
+  -- Move fsmall to fbig; might send if fbig empty
+  local function promote(f)
+    if #fsmall == 0 then return end
+    local str = concat(fsmall)
+    fsmall, lsmall = {}, 0
+    fbig:queue(str, f or ssend)
+  end
+
+  local function sendnext()
+    if not fbig:dequeue(ssend) then promote() end
+  end
+
+  sock:on("sent", sendnext)
+
+  return function(s)
+    -- don't sweat the petty things
+    if s == nil or s == "" then return end
+
+    -- Function?  Go ahead and queue this thing in the right place.
+    if type(s) == "function" then promote(); fbig:queue(s, ssend); return; end
+
+    s = tostring(s)
+
+    -- cork sending until the end in case we're the head of line
+    local corked = false
+    local function corker(t) corked = true; return t end
+
+    -- small fifo would overfill?  promote it
+    if lsmall + #s > BIGTHRESH or #fsmall >= FSMALLLIM then promote(corker) end
+
+    -- big string?  chunk and queue big components immediately
+    -- behind any promotion that just took place
+    while #s > BIGTHRESH + SPLITSLOP do
+     local pfx
+     pfx, s = s:sub(1,BIGTHRESH), s:sub(BIGTHRESH+1)
+     fbig:queue(pfx, corker)
+    end
+
+    -- Big string?  queue and maybe tx now
+    if #s > BIGTHRESH then fbig:queue(s, corker)
+    -- small and fifo in immediate dequeue mode
+    elseif fbig._go and lsmall == 0 then fbig:queue(s, corker)
+    -- small and queue already moving; let it linger in the small fifo
+    else insert(fsmall, s) ; lsmall = lsmall + #s
+    end
+
+    -- if it happened that we corked the transmission above...
+    --   if we queued a good amount of data, go ahead and start transmitting;
+    --   otherwise, wait a tick and hopefully we will queue more in the interim
+    --   before transmitting.
+    if corked then
+      if #fbig <= COALIMIT
+       then tmr.create():alarm(1, tmr.ALARM_SINGLE, sendnext)
+       else sendnext()
+      end
+    end
+  end
+end

lua_modules/fifo/fifosocktest.lua

@@ -0,0 +1,141 @@
+--
+-- Set verbose to 0 for quiet output (either the first assertion failure or
+-- "All tests OK"), to 1 to see the events ("SEND", "SENT", "CHECK") without
+-- the actual bytes, or to 2 to see the events with the bytes.
+--
+local verbose = 0
+
+local vprint = (verbose > 0) and print or function() end
+
+--
+-- Mock up enough of the nodemcu tmr structure, but pretend that nothing
+-- happens between ticks.  This won't exercise the optimistic corking logic,
+-- but that's probably fine.
+--
+tmr = {}
+tmr.ALARM_SINGLE = 0
+function tmr.create()
+  local r = {}
+  function r:alarm(_i, _t, cb) vprint("TMR") cb() end
+  return r
+end
+
+--
+-- Mock up enough of the nodemcu net.socket type; have it log all the sends
+-- into this "outs" array so that we can later check against it.
+--
+local outs = {}
+local fakesock = {
+  cb = nil,
+  on = function(this, _, cb) this.cb = cb end,
+  send = function(this, s) vprint("SEND", (verbose > 1) and s) table.insert(outs, s) end,
+}
+local function sent() vprint("SENT") fakesock.cb() end
+
+-- And wrap a fifosock around this fake socket
+local fsend = require "fifosock" (fakesock)
+
+-- Verify that the next unconsumed output is as indicated
+local function fcheck(x)
+  vprint ("CHECK", (verbose > 1) and x)
+  assert (#outs > 0)
+  assert (x == outs[1])
+  table.remove(outs, 1)
+end
+
+-- Enqueue an empty function to prevent coalescing within the fifosock
+local function nocoal() fsend(function() return nil end) end
+
+-- Send and check, for when the string should be sent exactly as is
+local function fsendc(x) fsend(x) fcheck(x) end
+
+-- Check that there are no more outputs
+local function fchecke() vprint("CHECKE") assert (#outs == 0) end
+
+--
+-- And now for the tests, which start easy and grow in complexity
+--
+
+fsendc("abracadabra none")
+sent() ; fchecke()
+
+fsendc("abracadabra three")
+fsend("short")
+fsend("string")
+fsend("build")
+sent() ; fcheck("shortstringbuild")
+sent() ; fchecke()
+
+-- Hit default FSMALLLIM while building up
+fsendc("abracadabra lots small")
+for i = 1, 32 do fsend("a") end
+nocoal()
+for i = 1, 4 do fsend("a") end
+sent() ; fcheck(string.rep("a", 32))
+sent() ; fcheck(string.rep("a", 4))
+sent() ; fchecke()
+
+-- Hit string length while building up
+fsendc("abracadabra overlong")
+for i = 1, 10 do fsend(string.rep("a",32)) end
+sent() ; fcheck(string.rep("a", 320))
+sent() ; fchecke()
+
+-- Hit neither before sending a big string
+fsendc("abracadabra mid long")
+for i = 1, 6 do fsend(string.rep("a",32)) end
+fsend(string.rep("b", 256))
+nocoal()
+for i = 1, 6 do fsend(string.rep("c",32)) end
+sent() ; fcheck(string.rep("a", 192) .. string.rep("b", 256))
+sent() ; fcheck(string.rep("c", 192))
+sent() ; fchecke()
+
+-- send a huge string, verify that it coalesces
+fsendc(string.rep("a",256) .. string.rep("b", 256) .. string.rep("c", 260))
+sent() ; fchecke()
+
+-- send a huge string, verify that it coalesces save for the short bit at the end
+fsend(string.rep("a",256) .. string.rep("b", 256) .. string.rep("c", 256) .. string.rep("d",256))
+fsend("e")
+fcheck(string.rep("a",256) .. string.rep("b", 256) .. string.rep("c", 256))
+sent() ; fcheck(string.rep("d",256) .. "e")
+sent() ; fchecke()
+
+-- send enough that our 4x lookahead still leaves something in the queue
+fsend(string.rep("a",512) .. string.rep("b", 512) .. string.rep("c", 512))
+fcheck(string.rep("a",512) .. string.rep("b", 512))
+sent() ; fcheck(string.rep("c",512))
+sent() ; fchecke()
+
+-- test a lazy generator
+local ix = 0
+local function gen() vprint("GEN", ix); ix = ix + 1; return ("a" .. ix), ix < 3 and gen end
+fsend(gen)
+fsend("b")
+fcheck("a1")
+sent() ; fcheck("a2")
+sent() ; fcheck("a3")
+sent() ; fcheck("b")
+sent() ; fchecke()
+
+-- test a completion-like callback that does send text
+local ix = 0
+local function gen() vprint("GEN"); ix = 1; return "efgh", nil end
+fsend("abcd"); fsend(gen); fsend("ijkl")
+assert (ix == 0)
+         fcheck("abcd"); assert (ix == 0)
+sent() ; fcheck("efgh"); assert (ix == 1); ix = 0
+sent() ; fcheck("ijkl"); assert (ix == 0)
+sent() ; fchecke()
+
+-- and one that doesn't
+local ix = 0
+local function gen() vprint("GEN"); ix = 1; return nil, nil end
+fsend("abcd"); fsend(gen); fsend("ijkl")
+assert (ix == 0)
+         fcheck("abcd"); assert (ix == 0)
+sent() ; fcheck("ijkl"); assert (ix == 1); ix = 0
+sent() ; fchecke()     ; assert (ix == 0)
+
+print("All tests OK")

lua_modules/http/http-example.lua

@@ -9,7 +9,7 @@ require("httpserver").createServer(80, function(req, res)
   print("+R", req.method, req.url, node.heap())
   -- setup handler of headers, if any
   req.onheader = function(self, name, value)
-    -- print("+H", name, value)
+    print("+H", name, value)
     -- E.g. look for "content-type" header,
     --   setup body parser to particular format
     -- if name == "content-type" then
@@ -23,13 +23,11 @@ require("httpserver").createServer(80, function(req, res)
   -- setup handler of body, if any
   req.ondata = function(self, chunk)
     print("+B", chunk and #chunk, node.heap())
-    -- request ended?
     if not chunk then
       -- reply
-      --res:finish("")
       res:send(nil, 200)
       res:send_header("Connection", "close")
-      res:send("Hello, world!")
+      res:send("Hello, world!\n")
       res:finish()
     end
   end

lua_modules/http/httpserver.lua

@@ -12,27 +12,24 @@ do
   -- request methods
   ------------------------------------------------------------------------------
   local make_req = function(conn, method, url)
-    local req = {
+    return {
       conn = conn,
       method = method,
       url = url,
     }
-    -- return setmetatable(req, {
-    -- })
-    return req
   end
 
   ------------------------------------------------------------------------------
   -- response methods
   ------------------------------------------------------------------------------
-  local send = function(self, data, status)
+  local make_res = function(csend, cfini)
-    local c = self.conn
+   local send = function(self, data, status)
     -- TODO: req.send should take care of response headers!
     if self.send_header then
-      c:send("HTTP/1.1 ")
+      csend("HTTP/1.1 ")
-      c:send(tostring(status or 200))
+      csend(tostring(status or 200))
       -- TODO: real HTTP status code/name table
-      c:send(" OK\r\n")
+      csend(" OK\r\n")
       -- we use chunked transfer encoding, to not deal with Content-Length:
       --   response header
       self:send_header("Transfer-Encoding", "chunked")
@@ -43,55 +40,51 @@ do
       if self.send_header then
         self.send_header = nil
         -- end response headers
-        c:send("\r\n")
+        csend("\r\n")
       end
       -- chunked transfer encoding
-      c:send(("%X\r\n"):format(#data))
+      csend(("%X\r\n"):format(#data))
-      c:send(data)
+      csend(data)
-      c:send("\r\n")
+      csend("\r\n")
     end
-  end
+   end
-  local send_header = function(self, name, value)
+   local send_header = function(self, name, value)
-    local c = self.conn
     -- NB: quite a naive implementation
-    c:send(name)
+    csend(name)
-    c:send(": ")
+    csend(": ")
-    c:send(value)
+    csend(value)
-    c:send("\r\n")
+    csend("\r\n")
-  end
+   end
-  -- finalize request, optionally sending data
+   -- finalize request, optionally sending data
-  local finish = function(self, data, status)
+   local finish = function(self, data, status)
-    local c = self.conn
+    -- NB: res.send takes care of response headers
-    -- NB: req.send takes care of response headers
     if data then
       self:send(data, status)
     end
     -- finalize chunked transfer encoding
-    c:send("0\r\n\r\n")
+    csend("0\r\n\r\n")
     -- close connection
-    c:close()
+    cfini()
-  end
+   end
-  --
+   --
-  local make_res = function(conn)
+    local res = { }
-    local res = {
-      conn = conn,
-    }
-    -- return setmetatable(res, {
-    --  send_header = send_header,
-    --  send = send,
-    --  finish = finish,
-    -- })
     res.send_header = send_header
     res.send = send
     res.finish = finish
     return res
   end
 
-    ------------------------------------------------------------------------------
+  ------------------------------------------------------------------------------
   -- HTTP parser
   ------------------------------------------------------------------------------
   local http_handler = function(handler)
     return function(conn)
+      local csend = (require "fifosock")(conn)
+      local cfini = function()
+        conn:on("receive", nil)
+        conn:on("disconnection", nil)
+        csend(function() conn:on("sent", nil) conn:close() end)
+      end
       local req, res
       local buf = ""
       local method, url
@@ -108,7 +101,7 @@ do
           cnt_len = tonumber(v)
         end
         if k == "expect" and v == "100-continue" then
-          conn:send("HTTP/1.1 100 Continue\r\n")
+          csend("HTTP/1.1 100 Continue\r\n")
         end
         -- delegate to request object
         if req and req.onheader then
@@ -118,8 +111,6 @@ do
       -- body data handler
       local body_len = 0
       local ondata = function(conn, chunk)
-        -- NB: do not reset node in case of lengthy requests
-        tmr.wdclr()
         -- feed request data to request handler
         if not req or not req.ondata then return end
         req:ondata(chunk)
@@ -153,8 +144,10 @@ do
             if method then
               -- make request and response objects
               req = make_req(conn, method, url)
-              res = make_res(conn)
+              res = make_res(csend, cfini)
             end
+            -- spawn request handler
+            handler(req, res)
           -- header line?
           elseif #line > 0 then
             -- parse header
@@ -166,11 +159,6 @@ do
             end
           -- headers end
           else
-            -- spawn request handler
-            -- NB: do not reset in case of lengthy requests
-            tmr.wdclr()
-            handler(req, res)
-            tmr.wdclr()
             -- NB: we feed the rest of the buffer as starting chunk of body
             ondata(conn, buf)
             -- buffer no longer needed

mkdocs.yml

@@ -41,6 +41,8 @@ pages:
     - 'bh1750': 'lua-modules/bh1750.md'
     - 'ds18b20': 'lua-modules/ds18b20.md'
     - 'ds3231': 'lua-modules/ds3231.md'
+    - 'fifo' : 'lua-modules/fifo.md'
+    - 'fifosock' : 'lua-modules/fifosock.md'
     - 'ftpserver': 'lua-modules/ftpserver.md'
     - 'hdc1000': 'lua-modules/hdc1000.md'
     - 'httpserver': 'lua-modules/httpserver.md'