File LFS Lua module initial commit (#3332)
* File LFS module initial commit * LFS file module update #1 * LFS file module update #2 - doc update and file.stat() returning read only attribute * Implementing file.list() * Fine-tuning `file_lfs` module * Adding `file_lfs` to mkdocs.yml * Implementing file.list() update #1 * Fine-tuning * Fine-tuning #2
This commit is contained in:
Lukáš Voborský
GitHub
parent
af689a606a
commit
949875d590
|
|
@ -0,0 +1,197 @@
|
|||
# File LFS module
|
||||
| Since | Origin / Contributor | Maintainer | Source |
|
||||
| :----- | :-------------------- | :---------- | :------ |
|
||||
| 2020-11-30 | [vsky279](https://github.com/vsky279) | [vsky279](https://github.com/vsky279) | [file_lfs.lua](../../lua_modules/file_lfs/file_lfs.lua)|
|
||||
|
||||
Provides access to arbitrary files stored in LFS.
|
||||
|
||||
An arbitrary file can be stored in LFS and still can be accessed using `file` functions. This module is an overlay over `file` base functions providing access to such LFS files.
|
||||
|
||||
The module tries to be transparent as much as possible so it makes no difference where LFS file or standard SPIFFS file is accessed. LFS file is read only. If the file is open for writing a standard SPIFFS file is opened instead.
|
||||
Both basic and object model can be used to access LFS file (see [file](../../docs/modules/file.md) module documentation).
|
||||
|
||||
## `resource.lua` file
|
||||
|
||||
Files to be stored in LFS needs to be preprocessed, i.e. a Lua file with its contents needs to be generated. This file called `resource.lua` is then included in the LFS image. A Lua script [`make_resource.lua`](../../lua_modules/file_lfs/make_resource.lua) can be used to generate `resource.lua` script.
|
||||
|
||||
A structure of the `resource.lua` file is simple. It returns a string, i.e. file content, depending on filename parameter passed to it. It returns table with list of files stored when called without any parameter.
|
||||
```Lua
|
||||
local arg = ...
|
||||
if arg == "index.html" then return "<!DOCTYPE html><html><body>Hi, there!</body></html>" end
|
||||
if arg == "favicon.ico" then return ""\000\000\000\000\000\000..." end
|
||||
if arg == nil then return {"index.html", "favicon.ico"} end
|
||||
```
|
||||
|
||||
## `make_resource.lua` script
|
||||
|
||||
Lua script to be run on PC to generate `resource.lua` file.
|
||||
|
||||
### Syntax
|
||||
```bash
|
||||
./make_resource.lua [-o outputfile] file1 [file2]
|
||||
```
|
||||
|
||||
### Example
|
||||
Create `resource.lua` file with all files in the `resource` directory
|
||||
```bash
|
||||
./make_resource resource/*
|
||||
```
|
||||
|
||||
## Basic usage
|
||||
```Lua
|
||||
file = require("file_lfs")
|
||||
|
||||
f = file.open("index.html") -- let's assume the above resource.lua file is embedded in LFS
|
||||
print(f:readline())
|
||||
-- prints: <!DOCTYPE html><html><body>Hi, there!</body></html>
|
||||
f:close()
|
||||
|
||||
f = file.open("init.lua")
|
||||
-- init.lua file is not stored in LFS (does not have entry in resource.lua stored in LFS) -> SPIFFS files is opened instead
|
||||
print(f:readline())
|
||||
f:close()
|
||||
```
|
||||
|
||||
Methods implemented - basically all `file` module functions are available though only some of them work with LFS files. The other functions are just passed through to the base `file` functions.
|
||||
|
||||
## file_lfs.list()
|
||||
|
||||
Lists all files in the file system. It works almost in the same way as [`file.list()`](../../docs/modules/file.md#filelist)
|
||||
|
||||
#### Syntax
|
||||
`file.list([pattern], [SPIFFs_only])`
|
||||
|
||||
#### Parameters
|
||||
- `pattern` only files matching the Lua pattern will be returned
|
||||
- `SPIFFs_only` if not `nil` LFS files won't be included in the result (LFS files are returned only if the parameter is `nil`)
|
||||
|
||||
#### Returns
|
||||
a Lua table which contains all {file name: file size} pairs, if no pattern
|
||||
given. If a pattern is given, only those file names matching the pattern
|
||||
(interpreted as a traditional [Lua pattern](https://www.lua.org/pil/20.2.html),
|
||||
not, say, a UNIX shell glob) will be included in the resulting table.
|
||||
`file.list` will throw any errors encountered during pattern matching.
|
||||
|
||||
|
||||
## file.rename()
|
||||
|
||||
Renames a file. If a file is currently open, it will be closed first. It works almost in the same way as [`file.rename()`](../../docs/modules/file.md#filerename)
|
||||
|
||||
#### Syntax
|
||||
`file.rename(oldname, newname)`
|
||||
|
||||
#### Parameters
|
||||
- `oldname` old file name
|
||||
- `newname` new file name
|
||||
|
||||
#### Returns
|
||||
`true` on success, `false` when the file is stored in LFS (so read-only) or on error
|
||||
|
||||
## file_lfs.open()
|
||||
|
||||
Opens a LFS file included in LFS in the `resource.lua` file. If it cannot be found in LFS not standard [`file.open()`](../../docs/modules/file.md#fileopen) function is called.
|
||||
LFS file is opened only when "r" access is requested.
|
||||
|
||||
#### Syntax
|
||||
`file.open(filename, mode)`
|
||||
|
||||
#### Parameters
|
||||
- `filename` file to be opened
|
||||
- `mode`:
|
||||
- "r": read mode (the default). If file of the same name is present in SPIFFS then SPIFFS file is opened instead of LFS file.
|
||||
- "w": write mode - as LFS file is read-only a SPIFFS file of the same name is created and opened for writing.
|
||||
- "r+", "w+", "a", "a+": as LFS file is read-only and all these modes allow file updates the LFS file is copied to SPIFFS and then it is opened with correspondig open mode.
|
||||
|
||||
#### Returns
|
||||
LFS file object (Lua table) or SPIFFS file object if file opened ok. `nil` if file not opened, or not exists (read modes).
|
||||
|
||||
## file.read(), file.obj:read()
|
||||
|
||||
Read content from the open file. It has the same parameters and returns values as [`file.read()` / `file.obj:read()`](../../docs/modules/file.md#fileread-fileobjread)
|
||||
|
||||
#### Syntax
|
||||
`file.read([n_or_char])`
|
||||
|
||||
`fd:read([n_or_char])`
|
||||
|
||||
#### Parameters
|
||||
- `n_or_char`:
|
||||
- if nothing passed in, then read up to `FILE_READ_CHUNK` bytes or the entire file (whichever is smaller).
|
||||
- if passed a number `n`, then read up to `n` bytes or the entire file (whichever is smaller).
|
||||
- if passed a string containing the single character `char`, then read until `char` appears next in the file, `FILE_READ_CHUNK` bytes have been read, or EOF is reached.
|
||||
|
||||
#### Returns
|
||||
File content as a string, or `nil` when EOF
|
||||
|
||||
|
||||
## file.readline(), file.obj:readline()
|
||||
|
||||
Read the next line from the open file. Lines are defined as zero or more bytes ending with a EOL ('\n') byte. If the next line is longer than 1024, this function only returns the first 1024 bytes.
|
||||
It has the same parameters and return values as [`file.readline()` / `file.obj:readline()`](../../docs/modules/file.md#filereadline-fileobjreadline)
|
||||
|
||||
|
||||
#### Syntax
|
||||
`file.readline()`
|
||||
|
||||
`fd:readline()`
|
||||
|
||||
#### Parameters
|
||||
none
|
||||
|
||||
#### Returns
|
||||
File content in string, line by line, including EOL('\n'). Return `nil` when EOF.
|
||||
|
||||
|
||||
## file.seek(), file.obj:seek()
|
||||
|
||||
Sets and gets the file position, measured from the beginning of the file, to the position given by offset plus a base specified by the string whence.
|
||||
It has the same parameters and return values as [`file.seek()` / `file.obj:seek()`](../../docs/modules/file.md#fileseek-fileobjseek)
|
||||
|
||||
#### Syntax
|
||||
`file.seek([whence [, offset]])`
|
||||
|
||||
`fd:seek([whence [, offset]])`
|
||||
|
||||
#### Parameters
|
||||
- `whence`
|
||||
- "set": base is position 0 (beginning of the file)
|
||||
- "cur": base is current position (default value)
|
||||
- "end": base is end of file
|
||||
- `offset` default 0
|
||||
|
||||
If no parameters are given, the function simply returns the current file offset.
|
||||
|
||||
#### Returns
|
||||
the resulting file position, or `nil` on error
|
||||
|
||||
## file.stat()
|
||||
|
||||
Get attribtues of a file or directory in a table. Elements of the table are:
|
||||
|
||||
- `size` file size in bytes
|
||||
- `name` file name
|
||||
- `time` table with time stamp information. Default is 1970-01-01 00:00:00 in case time stamps are not supported (on SPIFFS).
|
||||
|
||||
- `year`
|
||||
- `mon`
|
||||
- `day`
|
||||
- `hour`
|
||||
- `min`
|
||||
- `sec`
|
||||
|
||||
- `is_dir` flag `true` if item is a directory, otherwise `false`
|
||||
- `is_rdonly` flag `true` if item is read-only, otherwise `false`
|
||||
- `is_hidden` flag `true` if item is hidden, otherwise `false`
|
||||
- `is_sys` flag `true` if item is system, otherwise `false`
|
||||
- `is_arch` flag `true` if item is archive, otherwise `false`
|
||||
- `is_LFS` flag `true` if item is stored in LFS, otherwise it is not present in the `file.stat()` result table - **the only difference to `file.stat()`**
|
||||
|
||||
#### Syntax
|
||||
`file.stat(filename)`
|
||||
|
||||
#### Parameters
|
||||
`filename` file name
|
||||
|
||||
#### Returns
|
||||
table containing file attributes
|
||||
|
||||
|
|
@ -168,7 +168,7 @@ Lists all files in the file system.
|
|||
`file.list([pattern])`
|
||||
|
||||
#### Parameters
|
||||
none
|
||||
- `pattern` only files matching the Lua pattern will be returned
|
||||
|
||||
#### Returns
|
||||
a Lua table which contains all {file name: file size} pairs, if no pattern
|
||||
|
|
|
|||
|
|
@ -0,0 +1,158 @@
|
|||
local FILE_READ_CHUNK = 1024
|
||||
|
||||
local _file = file
|
||||
local file_exists, file_open, file_getcontents, file_rename, file_stat, file_putcontents, file_close, file_list =
|
||||
_file.exists, _file.open, _file.getcontents, _file.rename, _file.stat, _file.putcontents, _file.close, _file.list
|
||||
local node_LFS_resource = node.LFS.resource or function(filename) if filename then return else return {} end end -- luacheck: ignore
|
||||
|
||||
local file_lfs = {}
|
||||
local current_file_lfs
|
||||
|
||||
local function file_lfs_create (filename)
|
||||
local content = node_LFS_resource(filename)
|
||||
local pos = 1
|
||||
|
||||
local read = function (_, n_or_char)
|
||||
local p1, p2
|
||||
n_or_char = n_or_char or FILE_READ_CHUNK
|
||||
if type(n_or_char) == "number" then
|
||||
p1, p2 = pos, pos + n_or_char - 1
|
||||
elseif type(n_or_char) == "string" and #n_or_char == 1 then
|
||||
p1 = pos
|
||||
local _
|
||||
_, p2 = content:find(n_or_char, p1, true)
|
||||
if not p2 then p2 = p1 + FILE_READ_CHUNK - 1 end
|
||||
else
|
||||
error("invalid parameter")
|
||||
end
|
||||
if p2 - p1 > FILE_READ_CHUNK then p2 = pos + FILE_READ_CHUNK - 1 end
|
||||
if p1>#content then return end
|
||||
pos = p2+1
|
||||
return content:sub(p1, p2)
|
||||
end
|
||||
|
||||
local seek = function (_, whence, offset)
|
||||
offset = offset or 0
|
||||
local len = #content + 1 -- position starts at 1
|
||||
if whence == "set" then
|
||||
pos = offset + 1 -- 0 offset means position 1
|
||||
elseif whence == "cur" then
|
||||
pos = pos + offset
|
||||
elseif whence == "end" then
|
||||
pos = len + offset
|
||||
elseif whence then -- not nil not one of above
|
||||
return -- on error return nil
|
||||
end
|
||||
local pos_ok = true
|
||||
if pos < 1 then pos = 1; pos_ok = false end
|
||||
if pos > len then pos = len; pos_ok = false end
|
||||
return pos_ok and pos - 1 or nil
|
||||
end
|
||||
|
||||
local obj = {}
|
||||
obj.read = read
|
||||
obj.readline = function(self) return read(self, "\n") end
|
||||
obj.seek = seek
|
||||
obj.close = function() current_file_lfs = nil end
|
||||
|
||||
setmetatable(obj, {
|
||||
__index = function (_, k)
|
||||
return function () --...)
|
||||
error (("LFS file unsupported function '%s'") % tostring(k))
|
||||
--return _file[k](...)
|
||||
end
|
||||
end
|
||||
})
|
||||
|
||||
return obj
|
||||
end
|
||||
|
||||
file_lfs.exists = function (filename)
|
||||
return (node_LFS_resource(filename) ~= nil) or file_exists(filename)
|
||||
end
|
||||
|
||||
file_lfs.open = function (filename, mode)
|
||||
mode = mode or "r"
|
||||
if file_exists(filename) then
|
||||
return file_open(filename, mode)
|
||||
elseif node_LFS_resource(filename) then
|
||||
if mode ~= "r" and mode:find("^[rwa]%+?$") then
|
||||
-- file does not exist in SPIFFS but exists in LFS -> copy to SPIFFS
|
||||
file_putcontents(filename, node_LFS_resource(filename))
|
||||
return file_open(filename, mode)
|
||||
else -- "r" or anything else
|
||||
current_file_lfs = file_lfs_create (filename)
|
||||
return current_file_lfs
|
||||
end
|
||||
else
|
||||
return file_open(filename, mode)
|
||||
end
|
||||
end
|
||||
|
||||
file_lfs.close = function (...)
|
||||
current_file_lfs = nil
|
||||
return file_close(...)
|
||||
end
|
||||
|
||||
file_lfs.getcontents = function(filename)
|
||||
if file_exists(filename) then
|
||||
return file_getcontents(filename)
|
||||
else
|
||||
return node_LFS_resource(filename) or file_getcontents(filename)
|
||||
end
|
||||
end
|
||||
|
||||
file_lfs.rename = function(oldname, newname)
|
||||
if node_LFS_resource(oldname) ~= nil and not file_exists(oldname) then
|
||||
-- error "LFS file cannot be renamed"
|
||||
return false
|
||||
else
|
||||
return file_rename(oldname, newname)
|
||||
end
|
||||
end
|
||||
|
||||
file_lfs.stat = function(filename)
|
||||
if node_LFS_resource(filename) ~= nil and not file_exists(filename) then
|
||||
return {
|
||||
name = filename,
|
||||
size = #node_LFS_resource(filename),
|
||||
time = {day = 1, hour = 0, min = 0, year = 1970, sec = 0, mon = 1},
|
||||
is_hidden = false, is_rdonly = true, is_dir = false, is_arch = false, is_sys = false, is_LFS = true
|
||||
}
|
||||
else
|
||||
return file_stat(filename)
|
||||
end
|
||||
end
|
||||
|
||||
file_lfs.list = function (pattern, SPIFFs_only)
|
||||
local filelist = file_list(pattern)
|
||||
if not SPIFFs_only then
|
||||
local fl = node_LFS_resource()
|
||||
if fl then
|
||||
for _, f in ipairs(fl) do
|
||||
if not(filelist[f]) and (not pattern or f:match(pattern)) then
|
||||
filelist[f] = #node_LFS_resource(f)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
return filelist
|
||||
end
|
||||
|
||||
setmetatable(file_lfs, {
|
||||
__index = function (_, k)
|
||||
return function (...)
|
||||
local t = ...
|
||||
if type(t) == "table" then
|
||||
return t[k](...)
|
||||
elseif not t and current_file_lfs then
|
||||
return current_file_lfs[k](...)
|
||||
else
|
||||
return _file[k](...)
|
||||
end
|
||||
end
|
||||
end
|
||||
})
|
||||
|
||||
print ("[file_lfs] LFS file routines loaded")
|
||||
return file_lfs
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
#!/usr/bin/lua
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
-- Script to be used on PC to build resource.lua file
|
||||
-- Parameters: [-o outputfile] file1 [file2] ...
|
||||
-- Example: ./make_resource resource/*
|
||||
-- creates resource.lua file with all files in resource directory
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
local OUT = "resource.lua"
|
||||
|
||||
local function readfile(file)
|
||||
local f = io.open(file, "rb")
|
||||
if f then
|
||||
local lines = f:read("*all")
|
||||
f:close()
|
||||
return lines
|
||||
end
|
||||
end
|
||||
|
||||
-- tests the functions above
|
||||
print(string.format("make_resource script - %d parameter(s)", #arg))
|
||||
if #arg==0 or arg[1]=="--help" then
|
||||
print("parameters: [-o outputfile] file1 [file2] ...")
|
||||
return
|
||||
end
|
||||
|
||||
local larg = {}
|
||||
local outpar = false
|
||||
for i, a in pairs(arg) do
|
||||
if i>0 then
|
||||
if outpar then
|
||||
OUT = a
|
||||
outpar = false
|
||||
else
|
||||
if a == "-o" then
|
||||
outpar = true
|
||||
else
|
||||
table.insert(larg, a)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
print(string.format("output set to: %s", OUT))
|
||||
|
||||
local res = io.open(OUT, "w")
|
||||
res:write("-- luacheck: max line length 10000\nlocal arg = ...\n")
|
||||
res:close(file)
|
||||
|
||||
local filelist = ""
|
||||
for _, a in pairs(larg) do
|
||||
local inp = string.match(a, ".*[/](.*)")
|
||||
if not inp then inp = a end
|
||||
local content = readfile(a)
|
||||
print(string.format("# processing %s", inp))
|
||||
|
||||
if content then
|
||||
res = io.open(OUT, "a")
|
||||
filelist = filelist .. ('"%s",'):format(inp)
|
||||
res:write(('if arg == "%s" then return %q end\n\n'):format(inp, content))
|
||||
res:close(file)
|
||||
end
|
||||
end
|
||||
|
||||
res = io.open(OUT, "a")
|
||||
res:write(('if arg == nil then return {%s} end\n'):format(filelist))
|
||||
res:close(file)
|
||||
|
|
@ -54,6 +54,7 @@ pages:
|
|||
- 'ds3231': 'lua-modules/ds3231.md'
|
||||
- 'fifo' : 'lua-modules/fifo.md'
|
||||
- 'fifosock' : 'lua-modules/fifosock.md'
|
||||
- 'file_lfs': 'lua-modules/file_lfs.md'
|
||||
- 'ftpserver': 'lua-modules/ftpserver.md'
|
||||
- 'gossip': 'lua-modules/gossip.md'
|
||||
- 'hdc1000': 'lua-modules/hdc1000.md'
|
||||
|
|
|
|||
|
|
@ -0,0 +1,102 @@
|
|||
-- luacheck: globals file
|
||||
-- luacheck: new read globals node.LFS.resource
|
||||
file = require("file_lfs")
|
||||
|
||||
local Nt = ...
|
||||
Nt = (Nt or require "NTest")
|
||||
|
||||
-- check standard SPIFFS file functions
|
||||
loadfile("NTest_file.lua")(Nt)
|
||||
|
||||
local N = Nt("file_lfs")
|
||||
|
||||
N.test('resource.lua in LFS', function()
|
||||
ok(node.LFS.resource~=nil, "resource.lua embedded in LFS")
|
||||
end)
|
||||
|
||||
local testfile = "index.html"
|
||||
if node.LFS.resource("index.html") then
|
||||
testfile = "index.html"
|
||||
elseif node.LFS.resource("favicon.ico") then
|
||||
testfile = "favicon.ico"
|
||||
elseif node.LFS.resource("test.txt") then
|
||||
testfile = "test.txt"
|
||||
else
|
||||
error "No 'index.html' nor 'favicon.ico' nor 'text.txt' file stored in LFS resource module. Can't run LFS file tests."
|
||||
end
|
||||
|
||||
N.test('exist file LFS', function()
|
||||
ok(file.exists(testfile), "existing file")
|
||||
end)
|
||||
|
||||
N.test('getcontents file LFS', function()
|
||||
local testcontent = node.LFS.resource(testfile)
|
||||
local content = file.getcontents(testfile)
|
||||
ok(eq(testcontent, content),"contents")
|
||||
end)
|
||||
|
||||
N.test('read more than 1K file LFS', function()
|
||||
local f = file.open(testfile,"r")
|
||||
local size = #node.LFS.resource(testfile)
|
||||
local buffer = f:read()
|
||||
print(#buffer)
|
||||
ok(eq(#buffer < 1024 and size or 1024, 1024), "first block")
|
||||
buffer = f:read()
|
||||
f:close()
|
||||
ok(eq(#buffer, size-1024 > 1024 and 1024 or size-1024), "second block")
|
||||
end)
|
||||
|
||||
N.test('open existing file LFS', function()
|
||||
file.remove(testfile)
|
||||
|
||||
local function testopen(mode, position)
|
||||
file.putcontents(testfile, "testcontent")
|
||||
ok(file.open(testfile, mode), mode)
|
||||
file.write("")
|
||||
ok(eq(file.seek(), position), "seek check")
|
||||
file.close()
|
||||
end
|
||||
|
||||
testopen("r", 0)
|
||||
testopen("w", 0)
|
||||
file.remove(testfile)
|
||||
testopen("a", 11)
|
||||
file.remove(testfile)
|
||||
testopen("r+", 0)
|
||||
file.remove(testfile)
|
||||
testopen("w+", 0)
|
||||
file.remove(testfile)
|
||||
testopen("a+", 11)
|
||||
file.remove(testfile)
|
||||
end)
|
||||
|
||||
N.test('seek file LFS', function()
|
||||
|
||||
local testcontent = node.LFS.resource(testfile)
|
||||
local content
|
||||
|
||||
local f = file.open(testfile)
|
||||
f:seek("set", 0)
|
||||
content = f:read(10)
|
||||
ok(eq(testcontent:sub(1,10), content),"set 0")
|
||||
|
||||
f:seek("set", 99)
|
||||
content = f:read(10)
|
||||
ok(eq(testcontent:sub(100,109), content),"set 100")
|
||||
|
||||
f:seek("cur", 10)
|
||||
content = f:read(10)
|
||||
ok(eq(testcontent:sub(120,129), content),"cur 10")
|
||||
|
||||
f:seek("cur", -50)
|
||||
content = f:read(10)
|
||||
ok(eq(testcontent:sub(80,89), content),"cur -50")
|
||||
|
||||
f:seek("end", -50)
|
||||
content = f:read(10)
|
||||
ok(eq(testcontent:sub(#testcontent+1-50,#testcontent+1-41), content),"end -50")
|
||||
end)
|
||||
|
||||
N.test('rename file LFS', function()
|
||||
nok(file.rename(testfile, "testfile"), "cannot rename LFS file")
|
||||
end)
|
||||
Loading…
Reference in New Issue