Commit Graph

198 Commits

Author SHA1 Message Date
Rory Powell 207512396a Merge pull request #10738 from Budibase/automation-investigation
Automation investigation logging + max doc size fix for automation log
2023-05-30 16:45:27 +01:00
Rory Powell 6230e62b9b Automation investigation logging + max doc size fix for automation log 2023-05-30 16:37:39 +01:00
Peter Clement 3119ba5a8d remove log 2023-05-30 14:41:04 +01:00
Peter Clement 0a91e5bed1 update let to const 2023-05-30 14:26:49 +01:00
Peter Clement 5a7f0ba586 Fix performance issue with looping and context 2023-05-30 14:25:28 +01:00
Peter Clement b0a32f03b4 remove log 2023-05-25 09:14:09 +01:00
Peter Clement a9bb534cdf set flag in automation orchestrator 2023-05-24 14:41:58 +01:00
Peter Clement d271bbdadb rename syncWebHooks to syncAutomations 2023-05-19 10:07:02 +01:00
Peter Clement e1eda14204 move sync automations away from bull and call directly 2023-05-18 17:01:53 +01:00
Rory Powell 578b599428 Lint 2023-05-17 14:10:22 +01:00
Rory Powell 42f472b038 Type updates for automation jobs 2023-05-17 12:26:07 +01:00
mike12345567 0d7959f720 Fixing some un-identified listeners, queue messages not being correctly printed, also improving the mechanism for detecting if in the main thread or not. 2023-05-04 17:20:21 +01:00
Michael Drury 1c7c06e983 Bulk of the automation plugin system, supporting custom plugins. 2023-04-10 23:48:54 +01:00
Peter Clement 0a567a3302 add new status to handle no iterations during loop 2023-03-31 12:23:10 +01:00
mike12345567 3859a4159f Minor fix for automations, the wrong function had the environment variables being added to environment, this is very minor, it simply makes sure that environment variables are in context for the whole of the automation runner - e.g. if utilising many datasources each of them doesn't need to re-retrieve the environment variables, instead they will be available based on the env vars retrieved at the start of the execution. 2023-02-10 14:14:43 +00:00
Martin McKeaveney d4a0e44b31 merge 2023-02-05 22:11:37 +00:00
Rory Powell da2c809a09 Merge branch 'develop' into feature/environment-variables 2023-01-27 11:27:11 +00:00
mike12345567 fe7b632727 Quick updates for #8989 - this issue appeared to be fixed already but just adding a few extra bits of security to make sure that looping only occurs when valid data is found in the binding. 2023-01-24 19:09:36 +00:00
mike12345567 5e68a4d814 Minor updates to the context system and also adding env vars to automation context. 2023-01-16 16:47:59 +00:00
Martin McKeaveney fbf8230a3d allow all user bindings in SQL queries (#9354) 2023-01-16 12:24:32 +00:00
mike12345567 fc5f6ca530 Adding the ability to store environment variables to context, to make them more easily available/re-usable, as well as exposing them to queries. 2023-01-13 19:53:46 +00:00
mike12345567 87cf52192d Small update to properly support environments with environment variables, making sure that datasources correctly select based on app ID. 2023-01-12 14:38:12 +00:00
mike12345567 d27674de5c Replacing some require statements with imports. 2023-01-11 19:07:01 +00:00
mike12345567 69b9af51ff Merge branch 'develop' of github.com:Budibase/budibase into feature/environment-variables 2023-01-11 18:08:49 +00:00
mike12345567 03df57d077 Adding in basic implementation of variable usage, getting from pro and enriching through new datasource SDK. 2023-01-11 17:57:51 +00:00
Rory Powell fba7ef26ad Replace ts-jest with swc/jest (#9289)
- Add swc dependencies
- Add transform change to jest.config.ts
- Replace `export =` with `export default` in src code to enable code coverage to work with swc transformer
- Restructure backend-core errors package to allow for exporting error classes with ESM syntax
- Update backend-core to no longer use `export =`, export individual packages instead of replacing with `export default` for backwards compatibility
- Update event publishers to use `export default` - this was required for the `jest.spyOn` usage inside backend-core common mocks
- Restructure some jest.mock usages where declaring the jest.fn as a variable outside of the package mock threw an error
2023-01-11 09:37:37 +00:00
Peter Clement 57a5d8505d assign wasLoopStep correctly 2023-01-05 12:16:45 +00:00
Peter Clement 90f3d28bb5 fix for looping issue wher output was incorrect 2023-01-05 10:52:50 +00:00
Peter Clement 9d841bc947 Server and Worker tests (#8928)
* automation unit tests

* row processor tests

* update executeQuery test

* update groups assertion

* some more worker tests

* plugin tests and tidying up

* linting

* temporarily disable group tests

* more tests

* fix import

* fix backup tests

* add pro mocks to worker

* check for app existence in import test

* test new tsconfig

* testing changes

* Pro test utils (#9020)

* Pro test utils changes

* Add test fixes

* Remove ts-ignore

Co-authored-by: mike12345567 <me@michaeldrury.co.uk>
Co-authored-by: Rory Powell <rory.codes@gmail.com>
2022-12-12 22:02:32 +00:00
Michael Drury 4f4eb4da26 Merge branch 'master' of github.com:Budibase/budibase into develop 2022-11-29 11:13:54 +00:00
mike12345567 658f12281d Fixes based on server build so far. 2022-11-26 15:10:41 +00:00
Dean f4c027c25b User/Auth context now being passed to dynamic variable request updates 2022-11-24 10:16:05 +00:00
mike12345567 02ede5d067 Finally removing all usages of backend-core/ type imports from server, including some further typescript conversions. 2022-11-22 19:49:59 +00:00
mike12345567 5f658a0398 Converting further DB layers to TS - fixing mocks. 2022-11-15 18:33:24 +00:00
mike12345567 b702c7482a Fixes for cronjob stop - correctly handle this without stalled job handle. 2022-10-14 13:26:42 +01:00
Peter Clement bd47c948c5 one more log 2022-10-05 10:01:11 +01:00
Peter Clement f4c9fc6897 remove logs 2022-10-05 09:53:47 +01:00
Peter Clement 2f220d668b fix looping issues in automations 2022-10-05 09:20:59 +01:00
Rory Powell 45a43baee2 Merge branch 'master' into develop 2022-09-12 14:05:36 +01:00
mike12345567 86c8618e8f Fix for #7431 - reboot didn't work at all previously which is why apps couldn't be published with it enabled, this is now a self host only feature, I've removed the ability to enable a reboot cron in the Cloud and it will not run the lookup/execution. 2022-09-07 17:05:17 +01:00
mike12345567 320cbfada3 Merge branch 'develop' of github.com:Budibase/budibase into plugins-dev-experience 2022-08-31 15:53:32 +01:00
Rory Powell dd86875ea2 Update shutdown logging 2022-08-31 10:47:41 +01:00
Rory Powell 28cdb16e08 enable stalled job disabling 2022-08-20 11:02:22 +01:00
Rory Powell f273a6f779 Add logging to all event listeners for automations + stalled handling stub 2022-08-19 23:14:19 +01:00
mike12345567 51e06a9c7a Adding custom datasources to UI. 2022-08-15 18:38:09 +01:00
mike12345567 b803a3fd93 Replacing all plural TS enumerations with singular - this involves a major rename of the DocumentTypes. 2022-08-11 13:50:05 +01:00
mike12345567 f4db17502a Merge branch 'develop' of github.com:Budibase/budibase into feature/automation-error-stop 2022-07-26 17:28:57 +01:00
mike12345567 61f24fe03f Adding UI for displaying when the CRON job has been stopped and alert the user to next steps. 2022-07-26 13:59:22 +01:00
mike12345567 3fad32899f Main body of updates to support disabling the cron, some cleanup of metadata at deployment to make sure they can be re-ran. 2022-07-25 20:45:29 +01:00
Michael Drury 07de5faa2d Main bulk of logic around detecting cron recurring failure, still need to disable. 2022-07-20 22:38:06 +01:00
mike12345567 ff0c3e501f Converting main automation thread to typescript. 2022-07-20 19:05:54 +01:00
mike12345567 078cf9e95f Fixing an issue with automations being unable to access app databases due to new context stack up. 2022-07-18 17:38:58 +01:00
Peter Clement 688c69fce2 Ensure a binding exists before continuing with loop 2022-07-18 13:10:18 +01:00
Dean d34739f459 Updated query tests to remove altered datasource. Updated messaging for refresh failure messaging 2022-07-08 11:19:41 +01:00
Dean 02c0b433bb Fixed issue where mutation was causing expression bindings to be lost. Request retries will now have them intact for re-enrichment 2022-07-08 09:29:16 +01:00
Dean a59d0c7f31 Linting commit 2022-07-06 13:09:13 +01:00
Dean 084a469442 Surface refresh process error messaging 2022-07-06 13:03:25 +01:00
Dean 9fcedfa979 Properly invalidate the cached user ensuring up-to-date credentials are always used 2022-07-06 11:51:48 +01:00
Dean 4805f73cb5 Merge remote-tracking branch 'origin/develop' into sso-rest-requests 2022-07-05 09:28:56 +01:00
deanhannigan 37ce29fa2d Merge pull request #6315 from Budibase/feature/current-user-rest-bindings
REST query bindings for Current User
2022-07-05 09:22:03 +01:00
Dean b0daf30b4a Merge remote-tracking branch 'origin/develop' into sso-rest-requests 2022-07-05 09:07:33 +01:00
Dean 1e6845d5cb Refresh the OAuth tokens automatically when making rest calls. Fix to remove the password from the api token authentication. 2022-07-03 21:13:15 +01:00
Dean 85e755fc7f Ignore authconfigs for datasources if they are empty 2022-07-03 18:55:44 +01:00
Dean 31b51e1ecf Fixes for datasource authentication parsing. Mapping UX updates 2022-07-01 17:27:24 +01:00
mike12345567 21a78d6fc8 Merge branch 'develop' of github.com:Budibase/budibase into feature/automation-logs 2022-07-01 15:14:12 +01:00
Dean ac1df878d4 Merge remote-tracking branch 'origin/develop' into feature/current-user-rest-bindings 2022-06-29 18:50:14 +01:00
Dean 705af49e5a Merge remote-tracking branch 'origin/develop' into sso-rest-requests 2022-06-29 09:17:18 +01:00
mike12345567 74261661f1 Updating filter/condition action to show a stopped status rather than error, updating everything else to be aware of this. 2022-06-28 17:02:24 +01:00
Peter Clement fadd4292ad update handling of max iterations 2022-06-23 16:08:25 +01:00
Dean 5fd385ab30 Merge commit 2022-06-23 14:29:19 +01:00
Dean 42cd5e2c63 Fix to ignore global rest query headers when they are not configured. 2022-06-15 10:09:47 +01:00
Dean d50a62fd37 Context binding for authenticated user in REST API querys. Includes fix for REST datasource UI 2022-06-14 10:14:05 +01:00
mike12345567 fe84c0f21c Adding pagination control to the API and to the frontend, as well as getting view working as expected, emitting different key combinations to be able to search by any pattern. 2022-06-01 22:39:51 +01:00
mike12345567 34759c7916 Adding real history to the history tab, getting storage to couch and retrieval + api working. 2022-06-01 16:01:06 +01:00
mike12345567 702f4057bd Progress towards using couchdb for automation logs, after testing redis and realising it won't work. 2022-05-31 19:52:35 +01:00
mike12345567 fa30457f2b Fixing issue with server not shutting down correctly when an error occurs, making sure that everything clears up gracefully. 2022-05-30 21:22:06 +01:00
Peter Clement 4c77cf2516 parse max iterations value correctly 2022-05-19 16:38:16 +01:00
mike12345567 637ce41b3f Fixing an issue where HBS statements couldn't be used for passing arrays through to a looping action in automations. 2022-05-11 11:29:33 +01:00
Peter Clement a2b38adac3 remove log 2022-05-04 13:30:08 +01:00
Peter Clement dea5e73317 account for objects in failure condition 2022-05-04 13:25:46 +01:00
Peter Clement 45fd19583c linting 2022-05-04 11:55:26 +01:00
Peter Clement af9b9b6921 Some foreach block fixes 2022-05-04 10:24:28 +01:00
mike12345567 9386a82e6f Fixing similar issue in queries for context handling and fixing some issues raised by IDE. 2022-05-03 16:04:59 +01:00
mike12345567 c66cbcd522 Fix for automation issue, doInAppContext for automation thread wasn't async, was immediately closing the database, rather than waiting for the operations to complete. 2022-05-03 15:54:03 +01:00
Martin McKeaveney 9ca4dacbaa merge with master 2022-04-26 10:21:45 +01:00
Martin McKeaveney fa66c07578 PR review 2022-04-25 11:45:15 +01:00
Michael Drury 94419024a4 Merge pull request #5510 from Budibase/fix/pouch-leak
PouchDB memory leak protection
2022-04-22 15:32:51 +01:00
Rory Powell d2a4e0a934 Merge branch 'develop' into feature/licensing 2022-04-20 14:59:36 +01:00
Rory Powell 41144db055 Remove remaining direct usage of CouchDB. Add common pouch config to backend-core 2022-04-19 16:45:27 +01:00
Peter Clement 1000ef35e1 Improve handling of loop handlebars string replacement 2022-04-18 09:22:23 +01:00
mike12345567 80a41235dd Fixing REST UI query error notification and adding better error for query timeout. 2022-04-14 17:28:14 +01:00
mike12345567 5180603374 Fix for #5411 - trimming the various binding components for SQL in queries. Also moving SQL interpolation out into a Typescript - prep for further unit testing/improvements. 2022-04-14 13:06:22 +01:00
Peter Clement 43b4dd8737 add environment variable 2022-04-13 10:23:40 +01:00
Rory Powell 03ba6d914f Merge branch 'develop' into feature/licensing 2022-04-12 16:18:41 +01:00
Peter Clement 2d75e68717 add max iterations env variable 2022-04-12 09:13:01 +01:00
Peter Clement 47480450d8 fix failure condition 2022-04-11 23:10:29 +01:00
Peter Clement 999199dcf6 Add handling for failure conditions 2022-04-11 10:26:59 +01:00
Peter Clement b439e59137 Merge remote-tracking branch 'origin/develop' into feature/foreach-block 2022-04-07 22:32:44 +01:00
mike12345567 4d084383f3 Fix for #5242 - string interpolation in SQL queries was too wide, it was combining parts of multiple strings, rather than searching for bindings within distinct strings. 2022-04-04 19:08:56 +01:00
mike12345567 035cda1177 Extension of fix for 4978 - fixing an issue where parentheses are added to the IN query, causing the query system to not recognise the need to switch out the binding. 2022-04-04 15:39:38 +01:00