Merge branch 'develop' into cypress-testing

.github/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 From opening a bug report to creating a pull request: every contribution is appreciated and welcome. If you're planning to implement a new feature or change the api please create an issue first. This way we can ensure that your precious work is not in vain.
 
-### Not Sure Where to Start?
+## Not Sure Where to Start?
 
 Budibase is a low-code web application builder that creates svelte based web applications.
 
@@ -14,6 +14,8 @@ Budibase is a monorepo managed by [lerna](https://github.com/lerna/lerna). Lerna
 
 - **packages/server** - The budibase server. This [Koa](https://koajs.com/) app is responsible for serving the JS for the builder and budibase apps, as well as providing the API for interaction with the database and file system.
 
+- **packages/worker** - This [Koa](https://koajs.com/) app is responsible for providing global apis for managing your budibase installation. Authentication, Users, Email, Org and Auth configs are all provided by the worker. 
+
 ## Contributor License Agreement (CLA)
 
 In order to accept your pull request, we need you to submit a CLA. You only need to do this once. If you are submitting a pull request for the first time, just submit a Pull Request and our CLA Bot will give you instructions on how to sign the CLA before merging your Pull Request.
@@ -62,8 +64,6 @@ A component is the basic frontend building block of a budibase app.
 
 Component libraries are collections of components as well as the definition of their props contained in a file called `components.json`. 
 
-
-
 ## Contributing to Budibase
 
 * Please maintain the existing code style. 
@@ -74,31 +74,30 @@ Component libraries are collections of components as well as the definition of t
 
 * If the project diverges from your branch, please rebase instead of merging. This makes the commit graph easier to read.
 
-* Once your work is completed, please raise a PR against the main branch with some information about what has changed and why.
+* Once your work is completed, please raise a PR against the `develop` branch with some information about what has changed and why.
 
 ### Getting Started For Contributors
-
+#### 1.  Prerequisites
-### 1.  Prerequisites
 
 *yarn -* `npm install -g yarn`
 
 *jest* - `npm install -g jest`
 
-### 2. Clone this repository
+#### 2. Clone this repository
 
 `git clone https://github.com/Budibase/budibase.git`
 
 then `cd ` into your local copy.
 
-### 3.  Install and Build
+#### 3.  Install and Build
 
 To develop the Budibase platform you'll need [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) installed.
 
-#### Quick method
+##### Quick method
 
 `yarn setup` will check that all necessary components are installed and setup the repo for usage.
 
-#### Manual method
+##### Manual method
 
 The following commands can be executed to manually get Budibase up and running (assuming Docker/Docker Compose has been installed).
 
@@ -108,15 +107,7 @@ The following commands can be executed to manually get Budibase up and running (
 
 `yarn build` will build all budibase packages.
 
-### 4. Initialising Budibase and Creating a Budibase App
+#### 4. Running
-
-Starting up the budibase electron app should initialise budibase for you. A Budibase apps folder will have been created in `~/.budibase`.
-
-This is a blank apps folder, so you will need to create yourself an app, you can do this by clicking "Create New App" from the budibase builder.
-
-This will create a new budibase application in the `~/.budibase/<your-app-uuid>` directory, and NPM install the component libraries for that application. Let's start building your app with the budibase builder!
-
-### 4. Running
 
 To run the budibase server and builder in dev mode (i.e. with live reloading):
 
@@ -126,7 +117,7 @@ To run the budibase server and builder in dev mode (i.e. with live reloading):
 
 This will enable watch mode for both the builder app, server, client library and any component libraries.
 
-### 5. Debugging using VS Code
+#### 5. Debugging using VS Code
 
 To debug the budibase server and worker a VS Code launch configuration has been provided. 
 
@@ -135,75 +126,90 @@ Alternatively to start both components simultaneously select `Start Budibase`.
 
 In addition to the above, the remaining budibase components may be ran in dev mode using: `yarn dev:noserver`.
 
-### 6. Cleanup
+#### 6. Cleanup
 
 If you wish to delete all the apps created in development and reset the environment then run the following:
 
 1. `yarn nuke:docker` will wipe all the Budibase services
 2. `yarn dev` will restart all the services
 
-## Data Storage
+### Backend
-
-When you are running locally, budibase stores data on disk using [PouchDB](https://pouchdb.com/), as well as some JSON on local files. After setting up budibase, you can find all of this data in the `~/.budibase` directory.
-
-A client can have one or more budibase applications. Budibase applications are stored in `~/.budibase/<app-uuid>`. Files used by your budibase application when running are stored in the `public` directory. Everything else is dev files used for the development of your apps in the builder.
-
-#### Frontend
-
-To see the current individual JSON definitions for your pages and screens used by the builder, have a look at `~/.budibase/<app-uuid>/pages`.
-
-For your actual running application (not in dev), the frontend tree structure of the application (known as `clientFrontendDefinition`) is stored as JSON on disk. This is what the budibase client library reads to create your app at runtime. This can be found at `~/.budibase/<app-uuid>/public/clientFrontendDefinition.js`
-
-The HTML and CSS for your apps runtime pages, as well as the budibase client library JS is stored at:
-
-- `~/.budibase/<app-uuid>/public/main`
-- `~/.budibase/<app-uuid>/public/unauthenticated`
-
-#### Backend
 
 For the backend we run [Redis](https://redis.io/), [CouchDB](https://couchdb.apache.org/), [MinIO](https://min.io/) and [Envoy](https://www.envoyproxy.io/) in Docker compose. This means that to develop Budibase you will need Docker and Docker compose installed. The backend services are then ran separately as Node services with nodemon so that they can be debugged outside of Docker.
 
-### Publishing Budibase to NPM
+### Data Storage
 
-#### Testing In Electron
+When you are running locally, budibase stores data on disk using docker volumes. The volumes and the types of data associated with each are:
 
-At budibase, we pride ourselves on giving our users a fast, native and slick local development experience. As a result, we use the electron to provide a native GUI for the budibase builder. In order to release budibase out into the wild, you should test your changes in a packaged electron application. To do this, first build budibase from the root directory. 
+- `redis_data` 
+  - Sessions, email tokens
+- `couchdb3_data` 
+  - Global and app databases
+- `minio_data` 
+  - App manifest, budibase client, static assets
+
+### Devlopment Modes
+
+A combination of environment variables controls the mode that budibase runs in. 
+Yarn commands can be used to mimic the different modes that budibase can be ran in
+
+#### Self Hosted
+The default mode. A single tenant installation with no usage restrictions. 
+
+To enable this mode, use:
 ```
-yarn build
+yarn mode:self
 ```
 
-Now everything is built, you can package up your electron application.
+#### Cloud
+The cloud mode, with account portal turned off. 
+
+To enable this mode, use:
 ```
-cd packages/server
+yarn mode:cloud
-yarn build:electron
 ```
-
+#### Cloud & Account
-Your new electron application will be stored in `packages/server/dist/<operating-system>`. Open up the executable and make sure everything is working smoothly.
+The cloud mode, with account portal turned on. This is a replica of the mode that runs at https://budibase.app 
 
 
-#### Publishing to NPM
+To enable this mode, use:
-
-Once you are happy that your changes work in electron, you can publish all the latest versions of the monorepo packages by running:
-
 ```
-yarn publishnpm
+yarn mode:account
 ```
+### CI
 
-from your root directory.
+#### PR Job
 
-#### CI Release
+After your pr is submitted a github action (can be found at `.github/workflows/budibase_ci.yml`) will run to perform some checks against the changes such as linting, build and test. 
 
-After NPM has successfully published the budibase packages, a new tag will be pushed to master. This will kick off a github action (can be found at `.github/workflows/release.yml`) this will build and package the electron application for every OS (Windows, Mac, Linux). The binaries will be stored under the new tag on the [budibase releases page](https://github.com/Budibase/budibase/releases).
+The job will run when changes are pushed to or targetted at `master` and `develop`
+#### Release Develop
+
+To test changes before a release, a prerelease action (can be found at `.github/workflows/release-develop.yml`) will run to build and release develop versions of npm packages and docker images. On each subsequent commit to develop a new alpha version of npm packages will be created and released. 
+
+For example:
+
+- `feature1` -> `develop` = `v0.9.160-alpha.1`
+- `feature2` -> `develop` = `v0.9.160-alpha.0`
+
+The job will run when changes are pushed to `develop`
+#### Release Job
+
+To release changes a release job (can be found at `.github/workflows/release.yml`) will run to create final versions of npm packages and docker images. 
+
+Following the example above:
+
+- `develop` -> `master` = `v0.9.160`
+
+The job will run when changes are pushed to `master`
+
+#### Release Self Host Job
+
+To release the self hosted version of docker images, an additional job (can be found at `.github/workflows/release-selfhost.yml`) must be ran manually. This will releaae docker images to docker hub under the tag `latest` to be picked up by self hosted installations. 
 
 ### Troubleshooting
 
-Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again: 
+Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above. You should have a fresh Budibase installation.
-
-```
-rm -rf ~/.budibase
-```
-Follow from **Step 3. Install and Build** in the setup guide above. You should have a fresh Budibase installation.
-
 ### Running tests
 
 #### End-to-end Tests

.github/workflows/release-develop.yml

@@ -9,7 +9,7 @@ env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}
   INTERCOM_TOKEN: ${{ secrets.INTERCOM_TOKEN }}
   POSTHOG_URL: ${{ secrets.POSTHOG_URL }}
-  SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+  SENTRY_DSN: ${{ secrets.SENTRY_DSN_SELF_HOST }}
       
 jobs:
   release:

.github/workflows/release-selfhost.yml

@@ -1,4 +1,4 @@
-name: Budibase Release
+name: Budibase Release Selfhost
 
 on:
  workflow_dispatch:
@@ -7,7 +7,7 @@ env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}
   INTERCOM_TOKEN: ${{ secrets.INTERCOM_TOKEN }}
   POSTHOG_URL: ${{ secrets.POSTHOG_URL }}
-  SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+  SENTRY_DSN: ${{ secrets.SENTRY_DSN_SELF_HOST }}
 
 jobs:
   release:
@@ -28,12 +28,7 @@ jobs:
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: eu-west-1
 
-      - name: 'Get Previous tag'
-        id: previoustag
-        uses: "WyriHaximus/github-action-get-previous-tag@v1"
-
       - name: Build/release Docker images (Self Host)
-        if: ${{ github.event.inputs.release_self_host == 'Y' }}
         run: | 
           docker login -u $DOCKER_USER -p $DOCKER_PASSWORD
           yarn build
@@ -41,7 +36,7 @@ jobs:
         env:
           DOCKER_USER: ${{ secrets.DOCKER_USERNAME }}
           DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }}
-          BUDIBASE_RELEASE_VERSION: ${{ steps.previoustag.outputs.tag }}
+          BUDIBASE_RELEASE_VERSION: latest
       
       - uses: azure/setup-helm@v1
         id: install

docs/index.yaml

@@ -1,9 +1,35 @@
 apiVersion: v1
 entries:
   budibase:
+  - apiVersion: v2
+    appVersion: 0.9.163
+    created: "2021-10-12T21:58:00.515555+01:00"
+    dependencies:
+    - condition: services.couchdb.enabled
+      name: couchdb
+      repository: https://apache.github.io/couchdb-helm
+      version: 3.3.4
+    - condition: ingress.nginx
+      name: ingress-nginx
+      repository: https://github.com/kubernetes/ingress-nginx
+      version: 3.35.0
+    description: Budibase is an open source low-code platform, helping thousands of teams build apps for their workplace in minutes.
+    digest: f369536c0eac1f6959d51e8ce6d74a87a7a9df29ae84fb9cbed0a273ab77429b
+    keywords:
+    - low-code
+    - database
+    - cluster
+    name: budibase
+    sources:
+    - https://github.com/Budibase/budibase
+    - https://budibase.com
+    type: application
+    urls:
+    - https://budibase.github.io/budibase/budibase-0.2.0.tgz
+    version: 0.2.0
   - apiVersion: v2
     appVersion: 0.9.56
-    created: "2021-08-18T18:41:52.640176+01:00"
+    created: "2021-10-12T21:58:00.512062+01:00"
     dependencies:
     - condition: services.couchdb.enabled
       name: couchdb
@@ -28,7 +54,7 @@ entries:
     version: 0.1.1
   - apiVersion: v2
     appVersion: 0.9.56
-    created: "2021-08-18T18:41:52.635603+01:00"
+    created: "2021-10-12T21:58:00.507257+01:00"
     dependencies:
     - condition: services.couchdb.enabled
       name: couchdb
@@ -51,4 +77,4 @@ entries:
     urls:
     - https://budibase.github.io/budibase/budibase-0.1.0.tgz
     version: 0.1.0
-generated: "2021-08-18T18:41:52.629415+01:00"
+generated: "2021-10-12T21:58:00.503447+01:00"

hosting/docker-compose.yaml

@@ -21,7 +21,7 @@ services:
       PORT: 4002
       JWT_SECRET: ${JWT_SECRET}
       LOG_LEVEL: info
-      SENTRY_DSN: https://a34ae347621946bf8acded18e5b7d4b8@o420233.ingest.sentry.io/5338131
+      SENTRY_DSN: https://cc54bb0358fd4300ae97ef2273fbaf9f@o420233.ingest.sentry.io/6007553
       ENABLE_ANALYTICS: "true"
       REDIS_URL: redis-service:6379
       REDIS_PASSWORD: ${REDIS_PASSWORD}
@@ -51,7 +51,6 @@ services:
       INTERNAL_API_KEY: ${INTERNAL_API_KEY}
       REDIS_URL: redis-service:6379
       REDIS_PASSWORD: ${REDIS_PASSWORD}
-      ACCOUNT_PORTAL_URL: https://portal.budi.live
     volumes:
       - ./logs:/logs
     depends_on:

hosting/kubernetes/budibase/Chart.yaml

@@ -22,13 +22,13 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.1
+version: 0.2.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "0.9.56"
+appVersion: "0.9.163"
 
 dependencies:
   - name: couchdb
@@ -37,5 +37,5 @@ dependencies:
     condition: services.couchdb.enabled
   - name: ingress-nginx
     version: 3.35.0 
-    repository: https://kubernetes.github.io/ingress-nginx
+    repository: https://github.com/kubernetes/ingress-nginx
-    condition: services.ingress.nginx
+    condition: ingress.nginx

hosting/kubernetes/budibase/templates/alb-ingress.yaml

@@ -7,6 +7,8 @@ metadata:
     kubernetes.io/ingress.class: alb
     alb.ingress.kubernetes.io/scheme: internet-facing
     alb.ingress.kubernetes.io/target-type: ip
+    alb.ingress.kubernetes.io/success-codes: 200,301
+    alb.ingress.kubernetes.io/healthcheck-path: /
     {{- if .Values.ingress.certificateArn }}
     alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
     alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'

hosting/kubernetes/budibase/templates/app-service-deployment.yaml

@@ -14,7 +14,7 @@ spec:
     matchLabels:
       io.kompose.service: app-service
   strategy:
-    type: Recreate
+    type: RollingUpdate
   template:
     metadata:
       annotations:
@@ -73,13 +73,11 @@ spec:
               name: {{ template "budibase.fullname" . }}
               key: objectStoreSecret
         - name: MINIO_URL
-          {{ if .Values.services.objectStore.url }}
           value: {{ .Values.services.objectStore.url }}
-          {{ else }}
-          value: http://minio-service:{{ .Values.services.objectStore.port }}
-          {{ end }}
         - name: PORT
           value: {{ .Values.services.apps.port | quote }}
+        - name: MULTI_TENANCY
+          value: "1"
         - name: REDIS_PASSWORD
           value: {{ .Values.services.redis.password }}
         - name: REDIS_URL
@@ -92,14 +90,20 @@ spec:
           value: {{ .Values.globals.selfHosted | quote }}
         - name: SENTRY_DSN
           value: {{ .Values.globals.sentryDSN }}
+        - name: POSTHOG_TOKEN
+          value: {{ .Values.globals.posthogToken }}
         - name: WORKER_URL
-          value: worker-service:{{ .Values.services.worker.port }}
+          value: http://worker-service:{{ .Values.services.worker.port }}
-        - name: COOKIE_DOMAIN
+        - name: PLATFORM_URL
-          value: {{ .Values.globals.cookieDomain | quote }}
+          value: {{ .Values.globals.platformUrl | quote }}
+        - name: USE_QUOTAS
+          value: "1" 
         - name: ACCOUNT_PORTAL_URL
           value: {{ .Values.globals.accountPortalUrl | quote }}
         - name: ACCOUNT_PORTAL_API_KEY
           value: {{ .Values.globals.accountPortalApiKey | quote }}
+        - name: COOKIE_DOMAIN
+          value: {{ .Values.globals.cookieDomain | quote }}
         image: budibase/apps
         imagePullPolicy: Always
         name: bbapps

hosting/kubernetes/budibase/templates/proxy-service-deployment.yaml

@@ -14,7 +14,7 @@ spec:
     matchLabels:
       app.kubernetes.io/name: budibase-proxy
   strategy:
-    type: Recreate
+    type: RollingUpdate
   template:
     metadata:
       annotations:
@@ -26,7 +26,7 @@ spec:
     spec:
       containers:
       - image: budibase/proxy
-        imagePullPolicy: ""
+        imagePullPolicy: Always
         name: proxy-service
         ports:
         - containerPort: {{ .Values.services.proxy.port }}

hosting/kubernetes/budibase/templates/worker-service-deployment.yaml

@@ -15,7 +15,7 @@ spec:
     matchLabels:
       io.kompose.service: worker-service
   strategy:
-    type: Recreate
+    type: RollingUpdate
   template:
     metadata:
       annotations:
@@ -70,13 +70,11 @@ spec:
               name: {{ template "budibase.fullname" . }}
               key: objectStoreSecret
         - name: MINIO_URL
-          {{ if .Values.services.objectStore.url }}
           value: {{ .Values.services.objectStore.url }}
-          {{ else }}
-          value: http://minio-service:{{ .Values.services.objectStore.port }}
-          {{ end }}
         - name: PORT
           value: {{ .Values.services.worker.port | quote }}
+        - name: MULTI_TENANCY
+          value: "1"
         - name: REDIS_PASSWORD
           value: {{ .Values.services.redis.password | quote }}
         - name: REDIS_URL
@@ -91,8 +89,22 @@ spec:
           value: {{ .Values.globals.accountPortalUrl | quote }}
         - name: ACCOUNT_PORTAL_API_KEY
           value: {{ .Values.globals.accountPortalApiKey | quote }}
+        - name: PLATFORM_URL
+          value: {{ .Values.globals.platformUrl | quote }}
         - name: COOKIE_DOMAIN
           value: {{ .Values.globals.cookieDomain | quote }}
+        - name: SMTP_FALLBACK_ENABLED
+          value: {{ .Values.globals.smtp.enabled | quote }}
+        - name: SMTP_USER
+          value: {{ .Values.globals.smtp.user | quote }}
+        - name: SMTP_PASSWORD
+          value: {{ .Values.globals.smtp.password | quote }}
+        - name: SMTP_HOST
+          value: {{ .Values.globals.smtp.host | quote }}
+        - name: SMTP_PORT
+          value: {{ .Values.globals.smtp.port | quote }}
+        - name: SMTP_FROM_ADDRESS
+          value: {{ .Values.globals.smtp.from | quote }}
         image: budibase/worker
         imagePullPolicy: Always
         name: bbworker

hosting/kubernetes/budibase/values.yaml

@@ -40,11 +40,12 @@ service:
   port: 10000
 
 ingress:
-  enabled: true
+  enabled: false
-  nginx: true
+  aws: false
-  certificateArn: ""
+  nginx: true 
+  certificateArn: ""  
   className: ""
-  annotations:
+  annotations: 
     kubernetes.io/ingress.class: nginx
   hosts:
     - host: # change if using custom domain
@@ -55,7 +56,7 @@ ingress:
           service:
             name: proxy-service
             port:
-              number: 10000
+              number: 10000 
 
 resources: {}
   # We usually recommend not to specify default resources and to leave this as a conscious
@@ -84,20 +85,24 @@ affinity: {}
 
 globals:
   budibaseEnv: PRODUCTION
-  enableAnalytics: false
+  enableAnalytics: true
-  posthogToken: ""
   sentryDSN: ""
+  posthogToken: ""
   logLevel: info
-  selfHosted: 1
+  selfHosted: ""
-  accountPortalUrL: ""
+  accountPortalUrl: "" 
   accountPortalApiKey: ""
-  cookieDomain: ""
+  cookieDomain: "" 
+  platformUrl: ""
+
   createSecrets: true # creates an internal API key, JWT secrets and redis password for you
 
   # if createSecrets is set to false, you can hard-code your secrets here
   internalApiKey: ""
   jwtSecret: ""
 
+  smtp:
+    enabled: false
 
 services:
   dns: cluster.local
@@ -118,12 +123,12 @@ services:
   couchdb:
     enabled: true
     replicaCount: 3
-    url: "" # only change if pointing to existing couch server
+    # url: "" # only change if pointing to existing couch server
-    user: "" # only change if pointing to existing couch server
+    # user: "" # only change if pointing to existing couch server
-    password: "" # only change if pointing to existing couch server
+    # password: "" # only change if pointing to existing couch server
     port: 5984
     storage: 100Mi
-
+  
   redis:
     enabled: true # disable if using external redis
     port: 6379
@@ -131,7 +136,7 @@ services:
     url: "" # only change if pointing to existing redis cluster and enabled: false
     password: "budibase" # recommended to override if using built-in redis
     storage: 100Mi
-
+  
   objectStore:
     minio: true
     browser: true

lerna.json

@@ -1,5 +1,5 @@
 {
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "npmClient": "yarn",
   "packages": [
     "packages/*"

package.json

@@ -32,6 +32,7 @@
     "kill-port": "kill-port 4001",
     "dev": "yarn run kill-port && lerna link && lerna run --parallel dev:builder --concurrency 1",
     "dev:noserver": "lerna link && lerna run dev:stack:up && lerna run --parallel dev:builder --concurrency 1 --ignore @budibase/server --ignore @budibase/worker",
+    "dev:server": "lerna run --parallel dev:builder --concurrency 1 --scope @budibase/worker --scope @budibase/server",
     "test": "lerna run test",
     "lint:eslint": "eslint packages",
     "lint:prettier": "prettier --check \"packages/**/*.{js,svelte}\"",
@@ -46,12 +47,17 @@
     "build:docker:production": "lerna run build:docker && cd hosting/scripts/linux/ && ./release-to-docker-hub.sh $BUDIBASE_RELEASE_VERSION release && cd -",
     "build:docker:develop": "node scripts/pinVersions && lerna run build:docker && cd hosting/scripts/linux/ && ./release-to-docker-hub.sh develop && cd -",
     "release:helm": "./scripts/release_helm_chart.sh",
-    "multi:enable": "lerna run multi:enable",
+    "env:multi:enable": "lerna run env:multi:enable",
-    "multi:disable": "lerna run multi:disable",
+    "env:multi:disable": "lerna run env:multi:disable",
-    "selfhost:enable": "lerna run selfhost:enable",
+    "env:selfhost:enable": "lerna run env:selfhost:enable",
-    "selfhost:disable": "lerna run selfhost:disable",
+    "env:selfhost:disable": "lerna run env:selfhost:disable",
-    "localdomain:enable": "lerna run localdomain:enable",
+    "env:localdomain:enable": "lerna run env:localdomain:enable",
-    "localdomain:disable": "lerna run localdomain:disable",
+    "env:localdomain:disable": "lerna run env:localdomain:disable",
+    "env:account:enable": "lerna run env:account:enable",
+    "env:account:disable": "lerna run env:account:disable",
+    "mode:self": "yarn env:selfhost:enable && yarn env:multi:disable && yarn env:account:disable",
+    "mode:cloud": "yarn env:selfhost:disable && yarn env:multi:enable && yarn env:account:disable",
+    "mode:account": "yarn mode:cloud && yarn env:account:enable",
     "postinstall": "husky install"
   }
 }

packages/auth/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/auth",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "description": "Authentication middlewares for budibase builder and apps",
   "main": "src/index.js",
   "author": "Budibase",

packages/auth/src/middleware/authenticated.js

@@ -42,8 +42,9 @@ module.exports = (
         internal = false
       if (authCookie) {
         let error = null
-        const sessionId = authCookie.sessionId,
+        const sessionId = authCookie.sessionId
-          userId = authCookie.userId
+        const userId = authCookie.userId
+
         const session = await getSession(userId, sessionId)
         if (!session) {
           error = "No session found"

packages/auth/src/security/sessions.js

@@ -24,17 +24,24 @@ exports.createASession = async (userId, session) => {
   await client.store(makeSessionID(userId, sessionId), session, EXPIRY_SECONDS)
 }
 
-exports.invalidateSessions = async (userId, sessionId = null) => {
+exports.invalidateSessions = async (userId, sessionIds = null) => {
   let sessions = []
-  if (sessionId) {
+
-    sessions.push({ key: makeSessionID(userId, sessionId) })
+  // If no sessionIds, get all the sessions for the user
-  } else {
+  if (!sessionIds) {
     sessions = await getSessionsForUser(userId)
     sessions.forEach(
       session =>
         (session.key = makeSessionID(session.userId, session.sessionId))
     )
+  } else {
+    // use the passed array of sessionIds
+    sessions = Array.isArray(sessionIds) ? sessionIds : [sessionIds]
+    sessions = sessions.map(sessionId => ({
+      key: makeSessionID(userId, sessionId),
+    }))
   }
+
   const client = await redis.getSessionClient()
   const promises = []
   for (let session of sessions) {

packages/auth/src/utils.js

@@ -7,7 +7,7 @@ const {
 const jwt = require("jsonwebtoken")
 const { options } = require("./middleware/passport/jwt")
 const { createUserEmailView } = require("./db/views")
-const { Headers, UserStatus } = require("./constants")
+const { Headers, UserStatus, Cookies } = require("./constants")
 const {
   getGlobalDB,
   updateTenantId,
@@ -19,6 +19,7 @@ const accounts = require("./cloud/accounts")
 const { hash } = require("./hashing")
 const userCache = require("./cache/user")
 const env = require("./environment")
+const { getUserSessions, invalidateSessions } = require("./security/sessions")
 
 const APP_PREFIX = DocumentTypes.APP + SEPARATOR
 
@@ -235,3 +236,28 @@ exports.saveUser = async (
     }
   }
 }
+
+/**
+ * Logs a user out from budibase. Re-used across account portal and builder.
+ */
+exports.platformLogout = async ({ ctx, userId, keepActiveSession }) => {
+  if (!ctx) throw new Error("Koa context must be supplied to logout.")
+
+  const currentSession = this.getCookie(ctx, Cookies.Auth)
+  let sessions = await getUserSessions(userId)
+
+  if (keepActiveSession) {
+    sessions = sessions.filter(
+      session => session.sessionId !== currentSession.sessionId
+    )
+  } else {
+    // clear cookies
+    this.clearCookie(ctx, Cookies.Auth)
+    this.clearCookie(ctx, Cookies.CurrentApp)
+  }
+
+  await invalidateSessions(
+    userId,
+    sessions.map(({ sessionId }) => sessionId)
+  )
+}

packages/bbui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/bbui",
   "description": "A UI solution used in the different Budibase projects.",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "license": "AGPL-3.0",
   "svelte": "src/index.js",
   "module": "dist/bbui.es.js",

packages/bbui/src/Table/CellRenderer.svelte

@@ -5,11 +5,14 @@
   import RelationshipRenderer from "./RelationshipRenderer.svelte"
   import AttachmentRenderer from "./AttachmentRenderer.svelte"
   import ArrayRenderer from "./ArrayRenderer.svelte"
+  import InternalRenderer from "./InternalRenderer.svelte"
+
   export let row
   export let schema
   export let value
   export let customRenderers = []
 
+  let renderer
   const typeMap = {
     boolean: BooleanRenderer,
     datetime: DateTimeRenderer,
@@ -20,7 +23,9 @@
     number: StringRenderer,
     longform: StringRenderer,
     array: ArrayRenderer,
+    internal: InternalRenderer,
   }
+
   $: type = schema?.type ?? "string"
   $: customRenderer = customRenderers?.find(x => x.column === schema?.name)
   $: renderer = customRenderer?.component ?? typeMap[type] ?? StringRenderer

packages/bbui/src/Table/InternalRenderer.svelte

@@ -0,0 +1,28 @@
+<script>
+  import Icon from "../Icon/Icon.svelte"
+  import { notifications } from "../Stores/notifications"
+  export let value
+
+  const onClick = e => {
+    e.stopPropagation()
+    copyToClipboard(value)
+  }
+
+  function copyToClipboard(value) {
+    navigator.clipboard.writeText(value).then(() => {
+      notifications.success("Copied")
+    })
+  }
+</script>
+
+<div on:click|stopPropagation={onClick}>
+  <Icon size="S" name="Copy" />
+</div>
+
+<style>
+  div {
+    overflow: hidden;
+    text-overflow: ellipsis;
+    width: 150px;
+  }
+</style>

packages/bbui/src/Tabs/Tab.svelte

@@ -8,11 +8,19 @@
   const selected = getContext("tab")
   let tab
   let tabInfo
+
   const setTabInfo = () => {
-    tabInfo = tab.getBoundingClientRect()
+    // If the tabs are being rendered inside a component which uses
-    if ($selected.title === title) {
+    // a svelte transition to enter, then this initial getBoundingClientRect
-      $selected.info = tabInfo
+    // will return an incorrect position.
-    }
+    // We just need to get this off the main thread to fix this, by using
+    // a 0ms timeout.
+    setTimeout(() => {
+      tabInfo = tab.getBoundingClientRect()
+      if ($selected.title === title) {
+        $selected.info = tabInfo
+      }
+    }, 0)
   }
 
   onMount(() => {

packages/builder/cypress/integration/createTable.spec.js

@@ -31,15 +31,16 @@ context("Create a Table", () => {
     cy.contains("nameupdated ").should("contain", "nameupdated")
   })
 
+  
   it("edits a row", () => {
     cy.contains("button", "Edit").click({ force: true })
     cy.wait(1000)
     cy.get(".spectrum-Modal input").clear()
-    cy.get(".spectrum-Modal input").type("RoverUpdated")
+    cy.get(".spectrum-Modal input").type("Updated")
     cy.contains("Save").click()
     cy.contains("Updated").should("have.text", "Updated")
   })
-
+  
   it("deletes a row", () => {
     cy.get(".spectrum-Checkbox-input").check({ force: true })
     cy.contains("Delete 1 row(s)").click()

packages/builder/cypress/support/commands.js

@@ -36,18 +36,11 @@ Cypress.Commands.add("createApp", name => {
   cy.visit(`localhost:${Cypress.env("PORT")}/builder`)
   cy.wait(500)
   cy.contains(/Start from scratch/).click()
-  cy.get(".spectrum-Modal")
+  cy.get(".spectrum-Modal").within(() => {
-    .within(() => {
+    cy.get("input").eq(0).type(name).should("have.value", name).blur()
-      cy.get("input").eq(0).type(name).should("have.value", name).blur()
+    cy.get(".spectrum-ButtonGroup").contains("Create app").click()
-      cy.get(".spectrum-ButtonGroup").contains("Create app").click()
+    cy.wait(7000)
-      cy.wait(7000)
+  })
-    })
-    .then(() => {
-      // Because we show the datasource modal on entry, we need to create a table to get rid of the modal in the future
-      cy.createInitialDatasource("initialTable")
-      cy.expandBudibaseConnection()
-      cy.get(".nav-item.selected > .content").should("be.visible")
-    })
 })
 
 Cypress.Commands.add("deleteApp", () => {
@@ -77,22 +70,6 @@ Cypress.Commands.add("createTestTableWithData", () => {
   cy.addColumn("dog", "age", "Number")
 })
 
-Cypress.Commands.add("createInitialDatasource", tableName => {
-  // Enter table name
-  cy.get(".spectrum-Modal").within(() => {
-    cy.contains("Budibase DB").trigger("mouseover").click().click()
-    cy.wait(1000)
-    cy.contains("Continue").click()
-  })
-
-  cy.get(".spectrum-Modal").within(() => {
-    cy.wait(1000)
-    cy.get("input").first().type(tableName).blur()
-    cy.get(".spectrum-ButtonGroup").contains("Create").click()
-  })
-  cy.contains(tableName).should("be.visible")
-})
-
 Cypress.Commands.add("createTable", tableName => {
   cy.contains("Budibase DB").click()
   cy.contains("Create new table").click()

packages/builder/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/builder",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "license": "AGPL-3.0",
   "private": true,
   "scripts": {
@@ -65,10 +65,10 @@
     }
   },
   "dependencies": {
-    "@budibase/bbui": "^0.9.160-alpha.1",
+    "@budibase/bbui": "^0.9.167-alpha.8",
-    "@budibase/client": "^0.9.160-alpha.1",
+    "@budibase/client": "^0.9.167-alpha.8",
     "@budibase/colorpicker": "1.1.2",
-    "@budibase/string-templates": "^0.9.160-alpha.1",
+    "@budibase/string-templates": "^0.9.167-alpha.8",
     "@sentry/browser": "5.19.1",
     "@spectrum-css/page": "^3.0.1",
     "@spectrum-css/vars": "^3.0.1",

packages/builder/src/builderStore/dataBinding.js

@@ -7,11 +7,17 @@ import {
 } from "./storeUtils"
 import { store } from "builderStore"
 import { queries as queriesStores, tables as tablesStore } from "stores/backend"
-import { makePropSafe } from "@budibase/string-templates"
+import {
+  makePropSafe,
+  isJSBinding,
+  decodeJSBinding,
+  encodeJSBinding,
+} from "@budibase/string-templates"
 import { TableNames } from "../constants"
 
 // Regex to match all instances of template strings
 const CAPTURE_VAR_INSIDE_TEMPLATE = /{{([^}]+)}}/g
+const CAPTURE_VAR_INSIDE_JS = /\$\("([^")]+)"\)/g
 const CAPTURE_HBS_TEMPLATE = /{{[\S\s]*?}}/g
 
 /**
@@ -430,6 +436,15 @@ function replaceBetween(string, start, end, replacement) {
  * utility function for the readableToRuntimeBinding and runtimeToReadableBinding.
  */
 function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
+  // Decide from base64 if using JS
+  const isJS = isJSBinding(textWithBindings)
+  if (isJS) {
+    textWithBindings = decodeJSBinding(textWithBindings)
+  }
+
+  // Determine correct regex to find bindings to replace
+  const regex = isJS ? CAPTURE_VAR_INSIDE_JS : CAPTURE_VAR_INSIDE_TEMPLATE
+
   const convertFrom =
     convertTo === "runtimeBinding" ? "readableBinding" : "runtimeBinding"
   if (typeof textWithBindings !== "string") {
@@ -441,7 +456,7 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     .sort((a, b) => {
       return b.length - a.length
     })
-  const boundValues = textWithBindings.match(CAPTURE_VAR_INSIDE_TEMPLATE) || []
+  const boundValues = textWithBindings.match(regex) || []
   let result = textWithBindings
   for (let boundValue of boundValues) {
     let newBoundValue = boundValue
@@ -449,7 +464,7 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     // in the search, working from longest to shortest so always use best match first
     let searchString = newBoundValue
     for (let from of convertFromProps) {
-      if (shouldReplaceBinding(newBoundValue, from, convertTo)) {
+      if (isJS || shouldReplaceBinding(newBoundValue, from, convertTo)) {
         const binding = bindableProperties.find(el => el[convertFrom] === from)
         let idx
         do {
@@ -472,6 +487,12 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     }
     result = result.replace(boundValue, newBoundValue)
   }
+
+  // Re-encode to base64 if using JS
+  if (isJS) {
+    result = encodeJSBinding(result)
+  }
+
   return result
 }
 

packages/builder/src/builderStore/store/automation/Automation.js

@@ -14,7 +14,7 @@ export default class Automation {
   }
 
   addTestData(data) {
-    this.automation.testData = data
+    this.automation.testData = { ...this.automation.testData, ...data }
   }
 
   addBlock(block, idx) {

packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte

@@ -103,7 +103,7 @@
           <Detail size="S">{block?.name?.toUpperCase() || ""}</Detail>
         </div>
       </div>
-      {#if testResult}
+      {#if testResult && testResult[0]}
         <span on:click={() => resultsModal.show()}>
           <StatusLight
             positive={isTrigger || testResult[0].outputs?.success}

packages/builder/src/components/automation/AutomationBuilder/FlowChart/TestDataModal.svelte

@@ -5,20 +5,24 @@
   import { cloneDeep } from "lodash/fp"
 
   let failedParse = null
+  let trigger = {}
+  let schemaProperties = {}
+
   // clone the trigger so we're not mutating the reference
-  let trigger = cloneDeep(
+  $: trigger = cloneDeep(
     $automationStore.selectedAutomation.automation.definition.trigger
   )
-  let schemaProperties = Object.entries(trigger.schema.outputs.properties || {})
+
+  // get the outputs so we can define the fields
+  $: schemaProperties = Object.entries(trigger?.schema?.outputs?.properties)
 
   if (!$automationStore.selectedAutomation.automation.testData) {
     $automationStore.selectedAutomation.automation.testData = {}
   }
 
-  // get the outputs so we can define the fields
-
   // check to see if there is existing test data in the store
-  $: testData = $automationStore.selectedAutomation.automation.testData
+  $: testData = $automationStore.selectedAutomation.automation.testData || {}
+
   // Check the schema to see if required fields have been entered
   $: isError = !trigger.schema.outputs.required.every(
     required => testData[required]
@@ -41,7 +45,6 @@
   showConfirmButton={true}
   disabled={isError}
   onConfirm={() => {
-    automationStore.actions.addTestDataToAutomation(testData)
     automationStore.actions.test(
       $automationStore.selectedAutomation?.automation,
       testData
@@ -53,7 +56,7 @@
     ><Tab icon="Form" title="Form">
       <div class="tab-content-padding">
         <AutomationBlockSetup
-          bind:testData
+          {testData}
           {schemaProperties}
           isTestModal
           block={trigger}

packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte

@@ -9,7 +9,10 @@
     Label,
     ActionButton,
     Drawer,
+    Modal,
   } from "@budibase/bbui"
+  import CreateWebhookModal from "components/automation/Shared/CreateWebhookModal.svelte"
+
   import { automationStore } from "builderStore"
   import { tables } from "stores/backend"
   import WebhookDisplay from "../Shared/WebhookDisplay.svelte"
@@ -27,13 +30,14 @@
   import { buildLuceneQuery } from "helpers/lucene"
 
   export let block
-  export let webhookModal
   export let testData
   export let schemaProperties
   export let isTestModal = false
+  let webhookModal
   let drawer
   let tempFilters = lookForFilters(schemaProperties) || []
   let fillWidth = true
+
   $: stepId = block.stepId
   $: bindings = getAvailableBindings(
     block || $automationStore.selectedBlock,
@@ -50,6 +54,18 @@
   const onChange = debounce(
     async function (e, key) {
       if (isTestModal) {
+        // Special case for webhook, as it requires a body, but the schema already brings back the body's contents
+        if (stepId === "WEBHOOK") {
+          automationStore.actions.addTestDataToAutomation({
+            body: {
+              [key]: e.detail,
+              ...$automationStore.selectedAutomation.automation.testData.body,
+            },
+          })
+        }
+        automationStore.actions.addTestDataToAutomation({
+          [key]: e.detail,
+        })
         testData[key] = e.detail
       } else {
         block.inputs[key] = e.detail
@@ -178,6 +194,7 @@
             value={inputData[key]}
             on:change={e => onChange(e, key)}
             {bindings}
+            allowJS={false}
           />
         {/if}
       {:else if value.customType === "query"}
@@ -205,7 +222,10 @@
           {bindings}
         />
       {:else if value.customType === "webhookUrl"}
-        <WebhookDisplay value={inputData[key]} />
+        <WebhookDisplay
+          on:change={e => onChange(e, key)}
+          value={inputData[key]}
+        />
       {:else if value.customType === "triggerSchema"}
         <SchemaSetup on:change={e => onChange(e, key)} value={inputData[key]} />
       {:else if value.customType === "code"}
@@ -240,6 +260,7 @@
               value={inputData[key]}
               on:change={e => onChange(e, key)}
               {bindings}
+              allowJS={false}
             />
           </div>
         {/if}
@@ -247,6 +268,10 @@
     </div>
   {/each}
 </div>
+<Modal bind:this={webhookModal} width="30%">
+  <CreateWebhookModal />
+</Modal>
+
 {#if stepId === "WEBHOOK"}
   <Button secondary on:click={() => webhookModal.show()}>Set Up Webhook</Button>
 {/if}

packages/builder/src/components/automation/SetupPanel/QueryParamSelector.svelte

@@ -39,6 +39,7 @@
         type="string"
         {bindings}
         fillWidth={true}
+        allowJS={false}
       />
     {/each}
   </div>

packages/builder/src/components/automation/SetupPanel/RowSelector.svelte

@@ -1,20 +1,39 @@
 <script>
   import { tables } from "stores/backend"
-  import { Select, Toggle, DatePicker, Multiselect } from "@budibase/bbui"
+  import {
+    Select,
+    Toggle,
+    DatePicker,
+    Multiselect,
+    TextArea,
+  } from "@budibase/bbui"
   import DrawerBindableInput from "../../common/bindings/DrawerBindableInput.svelte"
   import AutomationBindingPanel from "../../common/bindings/ServerBindingPanel.svelte"
   import { createEventDispatcher } from "svelte"
   import ModalBindableInput from "components/common/bindings/ModalBindableInput.svelte"
+  import LinkedRowSelector from "components/common/LinkedRowSelector.svelte"
   import { automationStore } from "builderStore"
 
   const dispatch = createEventDispatcher()
 
   export let value
   export let bindings
-  $: table = $tables.list.find(table => table._id === value?.tableId)
+  let table
-  $: schemaFields = Object.entries(table?.schema ?? {})
+  let schemaFields
+
+  $: {
+    table = $tables.list.find(table => table._id === value?.tableId)
+    schemaFields = Object.entries(table?.schema ?? {})
+    // surface the schema so the user can see it in the json
+    schemaFields.map(([, schema]) => {
+      if (!schema.autocolumn && !value[schema.name]) {
+        value[schema.name] = ""
+      }
+    })
+  }
+
   const onChangeTable = e => {
-    value = { tableId: e.detail }
+    value["tableId"] = e.detail
     dispatch("change", value)
   }
 
@@ -39,7 +58,6 @@
   getOptionLabel={table => table.name}
   getOptionValue={table => table._id}
 />
-
 {#if schemaFields.length}
   <div class="schema-fields">
     {#each schemaFields as [field, schema]}
@@ -69,6 +87,10 @@
             label={field}
             options={schema.constraints.inclusion}
           />
+        {:else if schema.type === "longform"}
+          <TextArea label={field} bind:value={value[field]} />
+        {:else if schema.type === "link"}
+          <LinkedRowSelector bind:linkedRows={value[field]} {schema} />
         {:else if schema.type === "string" || schema.type === "number"}
           {#if $automationStore.selectedAutomation.automation.testData}
             <ModalBindableInput
@@ -88,6 +110,7 @@
               type="string"
               {bindings}
               fillWidth={true}
+              allowJS={false}
             />
           {/if}
         {/if}

packages/builder/src/components/automation/Shared/CreateWebhookModal.svelte

@@ -1,7 +1,6 @@
 <script>
   import { Icon } from "@budibase/bbui"
   import { automationStore } from "builderStore"
-  import { database } from "stores/backend"
   import WebhookDisplay from "./WebhookDisplay.svelte"
   import { ModalContent } from "@budibase/bbui"
   import { onMount, onDestroy } from "svelte"
@@ -12,7 +11,6 @@
   let schemaURL
   let propCount = 0
 
-  $: instanceId = $database._id
   $: automation = $automationStore.selectedAutomation?.automation
 
   onMount(async () => {

packages/builder/src/components/backend/DataTable/DataTable.svelte

@@ -16,11 +16,29 @@
   import { Pagination } from "@budibase/bbui"
 
   let hideAutocolumns = true
-
+  let schema
   $: isUsersTable = $tables.selected?._id === TableNames.USERS
-  $: schema = $tables.selected?.schema
   $: type = $tables.selected?.type
   $: isInternal = type !== "external"
+  $: {
+    schema = $tables.selected?.schema
+
+    // Manually add these as we don't want them to be 'real' auto-columns
+    schema._id = {
+      type: "internal",
+      editable: false,
+      displayName: "ID",
+      autocolumn: true,
+    }
+    if (isInternal) {
+      schema._rev = {
+        type: "internal",
+        editable: false,
+        displayName: "Revision",
+        autocolumn: true,
+      }
+    }
+  }
   $: id = $tables.selected?._id
   $: search = searchTable(id)
   $: columnOptions = Object.keys($search.schema || {})

packages/builder/src/components/backend/DataTable/modals/CreateEditColumn.svelte

@@ -32,6 +32,7 @@
   const FORMULA_TYPE = FIELDS.FORMULA.type
   const LINK_TYPE = FIELDS.LINK.type
   const dispatch = createEventDispatcher()
+  const PROHIBITED_COLUMN_NAMES = ["type", "_id", "_rev", "tableId"]
   const { hide } = getContext(Context.Modal)
   let fieldDefinitions = cloneDeep(FIELDS)
 
@@ -66,7 +67,11 @@
     (field.type === LINK_TYPE && !field.tableId) ||
     Object.keys($tables.draft?.schema ?? {}).some(
       key => key !== originalName && key === field.name
-    )
+    ) ||
+    columnNameInvalid
+  $: columnNameInvalid = PROHIBITED_COLUMN_NAMES.some(
+    name => field.name === name
+  )
 
   // used to select what different options can be displayed for column type
   $: canBeSearched =
@@ -200,6 +205,9 @@
     label="Name"
     bind:value={field.name}
     disabled={uneditable || (linkEditDisabled && field.type === LINK_TYPE)}
+    error={columnNameInvalid
+      ? `${PROHIBITED_COLUMN_NAMES.join(", ")} are not allowed as column names`
+      : ""}
   />
 
   <Select

packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/IntegrationConfigForm.svelte

@@ -1,10 +1,18 @@
 <script>
-  import { Label, Input, Layout, Toggle, Button } from "@budibase/bbui"
+  import {
+    Label,
+    Input,
+    Layout,
+    Toggle,
+    Button,
+    TextArea,
+  } from "@budibase/bbui"
   import KeyValueBuilder from "components/integration/KeyValueBuilder.svelte"
   import { capitalise } from "helpers"
 
   export let integration
   export let schema
+
   let addButton
 </script>
 
@@ -29,6 +37,15 @@
           <Label>{capitalise(configKey)}</Label>
           <Toggle text="" bind:value={integration[configKey]} />
         </div>
+      {:else if schema[configKey].type === "longForm"}
+        <div class="form-row">
+          <Label>{capitalise(configKey)}</Label>
+          <TextArea
+            type={schema[configKey].type}
+            on:change
+            bind:value={integration[configKey]}
+          />
+        </div>
       {:else}
         <div class="form-row">
           <Label>{capitalise(configKey)}</Label>

packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte

@@ -60,7 +60,7 @@
 </Modal>
 
 <Modal bind:this={externalDatasourceModal}>
-  <DatasourceConfigModal {integration} />
+  <DatasourceConfigModal {integration} {modal} />
 </Modal>
 
 <Modal bind:this={modal}>

packages/builder/src/components/backend/DatasourceNavigator/modals/DatasourceConfigModal.svelte

@@ -64,7 +64,7 @@
     ? "Fetch tables from database"
     : "Save and continue to query"}
   cancelText="Back"
-  size="M"
+  size="L"
 >
   <Layout noPadding>
     <Body size="XS"

packages/builder/src/components/common/CodeMirrorEditor.svelte

@@ -0,0 +1,159 @@
+<script context="module">
+  import { Label } from "@budibase/bbui"
+
+  export const EditorModes = {
+    JS: {
+      name: "javascript",
+      json: false,
+    },
+    JSON: {
+      name: "javascript",
+      json: true,
+    },
+    SQL: {
+      name: "sql",
+    },
+    Handlebars: {
+      name: "handlebars",
+      base: "text/html",
+    },
+  }
+</script>
+
+<script>
+  import CodeMirror from "components/integration/codemirror"
+  import { themeStore } from "builderStore"
+  import { createEventDispatcher, onMount } from "svelte"
+
+  export let mode = EditorModes.JS
+  export let value = ""
+  export let height = 300
+  export let resize = "none"
+  export let readonly = false
+  export let hints = []
+  export let label
+
+  const dispatch = createEventDispatcher()
+  let textarea
+  let editor
+
+  // Keep editor up to date with value
+  $: editor?.setValue(value || "")
+
+  // Creates an instance of a code mirror editor
+  async function createEditor(mode, value) {
+    if (!CodeMirror || !textarea || editor) {
+      return
+    }
+
+    // Configure CM options
+    const lightTheme = $themeStore.theme.includes("light")
+    const options = {
+      mode,
+      value: value || "",
+      readOnly: readonly,
+      theme: lightTheme ? "default" : "tomorrow-night-eighties",
+
+      // Style
+      lineNumbers: true,
+      lineWrapping: true,
+      indentWithTabs: true,
+      indentUnit: 2,
+      tabSize: 2,
+
+      // QOL addons
+      extraKeys: { "Ctrl-Space": "autocomplete" },
+      styleActiveLine: { nonEmpty: true },
+      autoCloseBrackets: true,
+      matchBrackets: true,
+    }
+
+    // Register hints plugin if desired
+    if (hints?.length) {
+      CodeMirror.registerHelper("hint", "dictionaryHint", function (editor) {
+        const cursor = editor.getCursor()
+        return {
+          list: hints,
+          from: CodeMirror.Pos(cursor.line, cursor.ch),
+          to: CodeMirror.Pos(cursor.line, cursor.ch),
+        }
+      })
+      CodeMirror.commands.autocomplete = function (cm) {
+        CodeMirror.showHint(cm, CodeMirror.hint.dictionaryHint)
+      }
+    }
+
+    // Construct CM instance
+    editor = CodeMirror.fromTextArea(textarea, options)
+
+    // Use a blur handler to update the value
+    editor.on("blur", instance => {
+      dispatch("change", instance.getValue())
+    })
+  }
+
+  // Export a function to expose caret position
+  export const getCaretPosition = () => {
+    const cursor = editor.getCursor()
+    return {
+      start: cursor.ch,
+      end: cursor.ch,
+    }
+  }
+
+  onMount(() => {
+    // Create the editor with initial value
+    createEditor(mode, value)
+
+    // Clean up editor on unmount
+    return () => {
+      if (editor) {
+        editor.toTextArea()
+      }
+    }
+  })
+</script>
+
+{#if label}
+  <div style="margin-bottom: var(--spacing-s)">
+    <Label small>{label}</Label>
+  </div>
+{/if}
+<div
+  style={`--code-mirror-height: ${height}px; --code-mirror-resize: ${resize}`}
+>
+  <textarea tabindex="0" bind:this={textarea} readonly {value} />
+</div>
+
+<style>
+  div :global(.CodeMirror) {
+    height: var(--code-mirror-height);
+    min-height: var(--code-mirror-height);
+    font-family: monospace;
+    line-height: 1.3;
+    border: var(--spectrum-alias-border-size-thin) solid;
+    border-color: var(--spectrum-alias-border-color);
+    border-radius: var(--border-radius-s);
+    resize: var(--code-mirror-resize);
+    overflow: hidden;
+  }
+
+  /* Override default active line highlight colour in dark theme */
+  div
+    :global(.CodeMirror-focused.cm-s-tomorrow-night-eighties
+      .CodeMirror-activeline-background) {
+    background: rgba(255, 255, 255, 0.075);
+  }
+
+  /* Remove active line styling when not focused */
+  div
+    :global(.CodeMirror:not(.CodeMirror-focused)
+      .CodeMirror-activeline-background) {
+    background: unset;
+  }
+
+  /* Add a spectrum themed border when focused */
+  div :global(.CodeMirror-focused) {
+    border-color: var(--spectrum-alias-border-color-mouse-focus);
+  }
+</style>

packages/builder/src/components/common/bindings/BindingPanel.svelte

@@ -1,32 +1,98 @@
 <script>
   import groupBy from "lodash/fp/groupBy"
-  import { Search, TextArea, DrawerContent } from "@budibase/bbui"
+  import {
-  import { createEventDispatcher } from "svelte"
+    Search,
-  import { isValid } from "@budibase/string-templates"
+    TextArea,
+    DrawerContent,
+    Tabs,
+    Tab,
+    Body,
+    Layout,
+  } from "@budibase/bbui"
+  import { createEventDispatcher, onMount } from "svelte"
+  import {
+    isValid,
+    decodeJSBinding,
+    encodeJSBinding,
+  } from "@budibase/string-templates"
   import { readableToRuntimeBinding } from "builderStore/dataBinding"
   import { handlebarsCompletions } from "constants/completions"
-  import { addToText } from "./utils"
+  import { addHBSBinding, addJSBinding } from "./utils"
+  import CodeMirrorEditor from "components/common/CodeMirrorEditor.svelte"
 
   const dispatch = createEventDispatcher()
 
   export let bindableProperties
   export let value = ""
   export let valid
+  export let allowJS = false
 
   let helpers = handlebarsCompletions()
   let getCaretPosition
   let search = ""
+  let initialValueJS = value?.startsWith("{{ js ")
+  let mode = initialValueJS ? "JavaScript" : "Handlebars"
+  let jsValue = initialValueJS ? value : null
+  let hbsValue = initialValueJS ? null : value
 
-  $: valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+  $: usingJS = mode === "JavaScript"
-  $: dispatch("change", value)
   $: ({ context } = groupBy("type", bindableProperties))
   $: searchRgx = new RegExp(search, "ig")
-  $: filteredColumns = context?.filter(context => {
+  $: filteredBindings = context?.filter(context => {
     return context.readableBinding.match(searchRgx)
   })
   $: filteredHelpers = helpers?.filter(helper => {
     return helper.label.match(searchRgx) || helper.description.match(searchRgx)
   })
+
+  const updateValue = value => {
+    valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+    if (valid) {
+      dispatch("change", value)
+    }
+  }
+
+  // Adds a HBS helper to the expression
+  const addHelper = helper => {
+    hbsValue = addHBSBinding(value, getCaretPosition(), helper.text)
+    updateValue(hbsValue)
+  }
+
+  // Adds a data binding to the expression
+  const addBinding = binding => {
+    if (usingJS) {
+      let js = decodeJSBinding(jsValue)
+      js = addJSBinding(js, getCaretPosition(), binding.readableBinding)
+      jsValue = encodeJSBinding(js)
+      updateValue(jsValue)
+    } else {
+      hbsValue = addHBSBinding(
+        hbsValue,
+        getCaretPosition(),
+        binding.readableBinding
+      )
+      updateValue(hbsValue)
+    }
+  }
+
+  const onChangeMode = e => {
+    mode = e.detail
+    updateValue(mode === "JavaScript" ? jsValue : hbsValue)
+  }
+
+  const onChangeHBSValue = e => {
+    hbsValue = e.detail
+    updateValue(hbsValue)
+  }
+
+  const onChangeJSValue = e => {
+    jsValue = encodeJSBinding(e.detail)
+    updateValue(jsValue)
+  }
+
+  onMount(() => {
+    valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+  })
 </script>
 
 <DrawerContent>
@@ -36,32 +102,24 @@
         <div class="heading">Search</div>
         <Search placeholder="Search" bind:value={search} />
       </section>
-      {#if filteredColumns?.length}
+      {#if filteredBindings?.length}
         <section>
           <div class="heading">Bindable Values</div>
           <ul>
-            {#each filteredColumns as { readableBinding }}
+            {#each filteredBindings as binding}
-              <li
+              <li on:click={() => addBinding(binding)}>
-                on:click={() => {
+                {binding.readableBinding}
-                  value = addToText(value, getCaretPosition(), readableBinding)
-                }}
-              >
-                {readableBinding}
               </li>
             {/each}
           </ul>
         </section>
       {/if}
-      {#if filteredHelpers?.length}
+      {#if filteredHelpers?.length && !usingJS}
         <section>
           <div class="heading">Helpers</div>
           <ul>
             {#each filteredHelpers as helper}
-              <li
+              <li on:click={() => addHelper(helper)}>
-                on:click={() => {
-                  value = addToText(value, getCaretPosition(), helper.text)
-                }}
-              >
                 <div class="helper">
                   <div class="helper__name">{helper.displayText}</div>
                   <div class="helper__description">
@@ -77,24 +135,56 @@
     </div>
   </svelte:fragment>
   <div class="main">
-    <TextArea
+    <Tabs selected={mode} on:select={onChangeMode}>
-      bind:getCaretPosition
+      <Tab title="Handlebars">
-      bind:value
+        <div class="main-content">
-      placeholder="Add text, or click the objects on the left to add them to the textbox."
+          <TextArea
-    />
+            bind:getCaretPosition
-    {#if !valid}
+            value={hbsValue}
-      <p class="syntax-error">
+            on:change={onChangeHBSValue}
-        Current Handlebars syntax is invalid, please check the guide
+            placeholder="Add text, or click the objects on the left to add them to the textbox."
-        <a href="https://handlebarsjs.com/guide/">here</a>
+          />
-        for more details.
+          {#if !valid}
-      </p>
+            <p class="syntax-error">
-    {/if}
+              Current Handlebars syntax is invalid, please check the guide
+              <a href="https://handlebarsjs.com/guide/">here</a>
+              for more details.
+            </p>
+          {/if}
+        </div>
+      </Tab>
+      {#if allowJS}
+        <Tab title="JavaScript">
+          <div class="main-content">
+            <Layout noPadding gap="XS">
+              <CodeMirrorEditor
+                bind:getCaretPosition
+                height={200}
+                value={decodeJSBinding(jsValue)}
+                on:change={onChangeJSValue}
+                hints={context?.map(x => `$("${x.readableBinding}")`)}
+              />
+              <Body size="S">
+                JavaScript expressions are executed as functions, so ensure that
+                your expression returns a value.
+              </Body>
+            </Layout>
+          </div>
+        </Tab>
+      {/if}
+    </Tabs>
   </div>
 </DrawerContent>
 
 <style>
   .main :global(textarea) {
-    min-height: 150px !important;
+    min-height: 202px !important;
+  }
+  .main {
+    margin: calc(-1 * var(--spacing-xl));
+  }
+  .main-content {
+    padding: var(--spacing-s) var(--spacing-xl);
   }
 
   .container {

packages/builder/src/components/common/bindings/DrawerBindableCombobox.svelte

@@ -6,6 +6,7 @@
   } from "builderStore/dataBinding"
   import BindingPanel from "components/common/bindings/BindingPanel.svelte"
   import { createEventDispatcher } from "svelte"
+  import { isJSBinding } from "@budibase/string-templates"
 
   export let panel = BindingPanel
   export let value = ""
@@ -15,11 +16,14 @@
   export let label
   export let disabled = false
   export let options
+  export let allowJS = true
 
   const dispatch = createEventDispatcher()
   let bindingDrawer
+
   $: readableValue = runtimeToReadableBinding(bindings, value)
   $: tempValue = readableValue
+  $: isJS = isJSBinding(value)
 
   const handleClose = () => {
     onChange(tempValue)
@@ -35,7 +39,7 @@
   <Combobox
     {label}
     {disabled}
-    value={readableValue}
+    value={isJS ? "(JavaScript function)" : readableValue}
     on:change={event => onChange(event.detail)}
     {placeholder}
     {options}
@@ -58,6 +62,7 @@
     close={handleClose}
     on:change={event => (tempValue = event.detail)}
     bindableProperties={bindings}
+    {allowJS}
   />
 </Drawer>
 

packages/builder/src/components/common/bindings/DrawerBindableInput.svelte

@@ -6,6 +6,7 @@
   } from "builderStore/dataBinding"
   import BindingPanel from "components/common/bindings/BindingPanel.svelte"
   import { createEventDispatcher } from "svelte"
+  import { isJSBinding } from "@budibase/string-templates"
 
   export let panel = BindingPanel
   export let value = ""
@@ -15,12 +16,15 @@
   export let label
   export let disabled = false
   export let fillWidth
+  export let allowJS = true
 
   const dispatch = createEventDispatcher()
   let bindingDrawer
   let valid = true
+
   $: readableValue = runtimeToReadableBinding(bindings, value)
   $: tempValue = readableValue
+  $: isJS = isJSBinding(value)
 
   const saveBinding = () => {
     onChange(tempValue)
@@ -36,7 +40,7 @@
   <Input
     {label}
     {disabled}
-    value={readableValue}
+    value={isJS ? "(JavaScript function)" : readableValue}
     on:change={event => onChange(event.detail)}
     {placeholder}
   />
@@ -60,6 +64,7 @@
     value={readableValue}
     on:change={event => (tempValue = event.detail)}
     bindableProperties={bindings}
+    {allowJS}
   />
 </Drawer>
 

packages/builder/src/components/common/bindings/ServerBindingPanel.svelte

@@ -5,7 +5,7 @@
   import { isValid } from "@budibase/string-templates"
   import { handlebarsCompletions } from "constants/completions"
   import { readableToRuntimeBinding } from "builderStore/dataBinding"
-  import { addToText } from "./utils"
+  import { addHBSBinding } from "./utils"
 
   const dispatch = createEventDispatcher()
 
@@ -47,7 +47,7 @@
               {#each bindings as binding}
                 <li
                   on:click={() => {
-                    value = addToText(value, getCaretPosition(), binding)
+                    value = addHBSBinding(value, getCaretPosition(), binding)
                   }}
                 >
                   <span class="binding__label">{binding.label}</span>
@@ -71,7 +71,7 @@
             {#each filteredHelpers as helper}
               <li
                 on:click={() => {
-                  value = addToText(value, getCaretPosition(), helper.text)
+                  value = addHBSBinding(value, getCaretPosition(), helper.text)
                 }}
               >
                 <div class="helper">

packages/builder/src/components/common/bindings/utils.js

@@ -1,4 +1,4 @@
-export function addToText(value, caretPos, binding) {
+export function addHBSBinding(value, caretPos, binding) {
   binding = typeof binding === "string" ? binding : binding.path
   value = value == null ? "" : value
   if (!value.includes("{{") && !value.includes("}}")) {
@@ -14,3 +14,18 @@ export function addToText(value, caretPos, binding) {
   }
   return value
 }
+
+export function addJSBinding(value, caretPos, binding) {
+  binding = typeof binding === "string" ? binding : binding.path
+  value = value == null ? "" : value
+  binding = `$("${binding}")`
+  if (caretPos.start) {
+    value =
+      value.substring(0, caretPos.start) +
+      binding +
+      value.substring(caretPos.end, value.length)
+  } else {
+    value += binding
+  }
+  return value
+}

packages/builder/src/components/design/PropertiesPanel/PropertyControls/Input.svelte

@@ -0,0 +1,15 @@
+<script>
+  import { Input } from "@budibase/bbui"
+  import { isJSBinding } from "@budibase/string-templates"
+
+  export let value
+
+  $: isJS = isJSBinding(value)
+</script>
+
+<Input
+  {...$$props}
+  value={isJS ? "(JavaScript function)" : value}
+  readonly={isJS}
+  on:change
+/>

packages/builder/src/components/design/PropertiesPanel/PropertyControls/PropertyControl.svelte

@@ -105,6 +105,7 @@
           value={safeValue}
           on:change={e => (tempValue = e.detail)}
           bindableProperties={bindings}
+          allowJS
         />
       </Drawer>
     {/if}

packages/builder/src/components/design/PropertiesPanel/PropertyControls/componentSettings.js

@@ -1,4 +1,4 @@
-import { Checkbox, Input, Select, Stepper } from "@budibase/bbui"
+import { Checkbox, Select, Stepper } from "@budibase/bbui"
 import DataSourceSelect from "./DataSourceSelect.svelte"
 import DataProviderSelect from "./DataProviderSelect.svelte"
 import EventsEditor from "./EventsEditor"
@@ -15,6 +15,7 @@ import URLSelect from "./URLSelect.svelte"
 import OptionsEditor from "./OptionsEditor/OptionsEditor.svelte"
 import FormFieldSelect from "./FormFieldSelect.svelte"
 import ValidationEditor from "./ValidationEditor/ValidationEditor.svelte"
+import Input from "./Input.svelte"
 
 const componentMap = {
   text: Input,

packages/builder/src/components/integration/QueryViewer.svelte

@@ -21,12 +21,15 @@
   import ParameterBuilder from "components/integration/QueryParameterBuilder.svelte"
   import { datasources, integrations, queries } from "stores/backend"
   import { capitalise } from "../../helpers"
+  import CodeMirrorEditor from "components/common/CodeMirrorEditor.svelte"
 
   export let query
   export let fields = []
 
   let parameters
   let data = []
+  const transformerDocs =
+    "https://docs.budibase.com/building-apps/data/transformers"
   const typeOptions = [
     { label: "Text", value: "STRING" },
     { label: "Number", value: "NUMBER" },
@@ -52,6 +55,11 @@
   $: readQuery = query.queryVerb === "read" || query.readable
   $: queryInvalid = !query.name || (readQuery && data.length === 0)
 
+  // seed the transformer
+  if (query && !query.transformer) {
+    query.transformer = "return data"
+  }
+
   function newField() {
     fields = [...fields, {}]
   }
@@ -74,6 +82,7 @@
       const response = await api.post(`/api/queries/preview`, {
         fields: query.fields,
         queryVerb: query.queryVerb,
+        transformer: query.transformer,
         parameters: query.parameters.reduce(
           (acc, next) => ({
             ...acc,
@@ -160,12 +169,34 @@
       <IntegrationQueryEditor
         {datasource}
         {query}
-        height={300}
+        height={200}
         schema={queryConfig[query.queryVerb]}
         bind:parameters
       />
       <Divider />
     </div>
+    <div class="config">
+      <div class="help-heading">
+        <Heading size="S">Transformer</Heading>
+        <Icon
+          on:click={() => window.open(transformerDocs)}
+          hoverable
+          name="Help"
+          size="L"
+        />
+      </div>
+      <Body size="S"
+        >Add a JavaScript function to transform the query result.</Body
+      >
+      <CodeMirrorEditor
+        height={200}
+        label="Transformer"
+        value={query.transformer}
+        resize="vertical"
+        on:change={e => (query.transformer = e.detail)}
+      />
+      <Divider />
+    </div>
     <div class="viewer-controls">
       <Heading size="S">Results</Heading>
       <ButtonGroup>
@@ -220,6 +251,7 @@
     display: grid;
     grid-gap: var(--spacing-s);
   }
+
   .config-field {
     display: grid;
     grid-template-columns: 20% 1fr;
@@ -227,6 +259,11 @@
     align-items: center;
   }
 
+  .help-heading {
+    display: flex;
+    justify-content: space-between;
+  }
+
   .field {
     display: grid;
     grid-template-columns: 1fr 1fr 5%;

packages/builder/src/components/integration/codemirror.js

@@ -1,12 +1,22 @@
 import CodeMirror from "codemirror"
 import "codemirror/lib/codemirror.css"
-import "codemirror/theme/tomorrow-night-eighties.css"
+
-import "codemirror/addon/hint/show-hint.css"
+// Modes
-import "codemirror/theme/neo.css"
+import "codemirror/mode/javascript/javascript"
 import "codemirror/mode/sql/sql"
 import "codemirror/mode/css/css"
 import "codemirror/mode/handlebars/handlebars"
-import "codemirror/mode/javascript/javascript"
+
+// Hints
 import "codemirror/addon/hint/show-hint"
+import "codemirror/addon/hint/show-hint.css"
+
+// Theming
+import "codemirror/theme/tomorrow-night-eighties.css"
+
+// Functional addons
+import "codemirror/addon/selection/active-line"
+import "codemirror/addon/edit/closebrackets"
+import "codemirror/addon/edit/matchbrackets"
 
 export default CodeMirror

packages/builder/src/components/integration/index.svelte

@@ -17,9 +17,9 @@
 
   $: urlDisplay =
     schema.urlDisplay &&
-    `${datasource.config.url}${query.fields.path ?? ""}${
+    `${datasource.config.url}${
-      query.fields.queryString ?? ""
+      query.fields.path ? "/" + query.fields.path : ""
-    }`
+    }${query.fields.queryString ? "?" + query.fields.queryString : ""}`
 
   function updateQuery({ detail }) {
     query.fields[schema.type] = detail.value

packages/builder/src/components/start/CreateAppModal.svelte

@@ -150,7 +150,6 @@
     showCancelButton={false}
     showCloseIcon={false}
   >
-    <Body size="M">Select a template below, or start from scratch.</Body>
     <TemplateList
       onSelect={selected => {
         if (!selected) {

packages/builder/src/components/start/TemplateList.svelte

@@ -1,5 +1,5 @@
 <script>
-  import { Heading, Layout, Icon } from "@budibase/bbui"
+  import { Heading, Layout, Icon, Body } from "@budibase/bbui"
   import Spinner from "components/common/Spinner.svelte"
   import api from "builderStore/api"
 
@@ -7,6 +7,7 @@
 
   async function fetchTemplates() {
     const response = await api.get("/api/templates?type=app")
+    console.log("Responded")
     return await response.json()
   }
 
@@ -19,6 +20,11 @@
       <Spinner size="30" />
     </div>
   {:then templates}
+    {#if templates?.length > 0}
+      <Body size="M">Select a template below, or start from scratch.</Body>
+    {:else}
+      <Body size="M">Start your app from scratch below.</Body>
+    {/if}
     <div class="templates">
       {#each templates as template}
         <div class="template" on:click={() => onSelect(template)}>

packages/builder/src/helpers/fetchTableData.js

@@ -48,7 +48,6 @@ export const fetchTableData = opts => {
   const fetchPage = async bookmark => {
     lastBookmark = bookmark
     const { tableId, limit, sortColumn, sortOrder, paginate } = options
-    store.update($store => ({ ...$store, loading: true }))
     const res = await API.post(`/api/${options.tableId}/search`, {
       tableId,
       query,
@@ -59,7 +58,6 @@ export const fetchTableData = opts => {
       paginate,
       bookmark,
     })
-    store.update($store => ({ ...$store, loading: false, loaded: true }))
     return await res.json()
   }
 
@@ -103,7 +101,7 @@ export const fetchTableData = opts => {
     if (!schema) {
       return
     }
-    store.update($store => ({ ...$store, schema }))
+    store.update($store => ({ ...$store, schema, loading: true }))
 
     // Work out what sort type to use
     if (!sortColumn || !schema[sortColumn]) {
@@ -135,6 +133,7 @@ export const fetchTableData = opts => {
     }
 
     // Fetch next page
+    store.update($store => ({ ...$store, loading: true }))
     const page = await fetchPage(state.bookmarks[state.pageNumber + 1])
 
     // Update state
@@ -148,6 +147,7 @@ export const fetchTableData = opts => {
         pageNumber: pageNumber + 1,
         rows: page.rows,
         bookmarks,
+        loading: false,
       }
     })
   }
@@ -160,6 +160,7 @@ export const fetchTableData = opts => {
     }
 
     // Fetch previous page
+    store.update($store => ({ ...$store, loading: true }))
     const page = await fetchPage(state.bookmarks[state.pageNumber - 1])
 
     // Update state
@@ -168,6 +169,7 @@ export const fetchTableData = opts => {
         ...$store,
         pageNumber: $store.pageNumber - 1,
         rows: page.rows,
+        loading: false,
       }
     })
   }

packages/builder/src/pages/builder/admin/_components/ImportAppsModal.svelte

@@ -1,6 +1,7 @@
 <script>
   import { notifications, ModalContent, Dropzone, Body } from "@budibase/bbui"
   import { post } from "builderStore/api"
+  import { admin } from "stores/portal"
 
   let submitting = false
 
@@ -20,8 +21,8 @@
       if (!importResp.ok) {
         throw new Error(importJson.message)
       }
-      // now reload to get to login
+      await admin.checkImportComplete()
-      window.location.reload()
+      notifications.success("Import complete, please finish registration!")
     } catch (error) {
       notifications.error(error)
       submitting = false

packages/builder/src/pages/builder/admin/index.svelte

@@ -15,6 +15,7 @@
   import PasswordRepeatInput from "components/common/users/PasswordRepeatInput.svelte"
   import ImportAppsModal from "./_components/ImportAppsModal.svelte"
   import Logo from "assets/bb-emblem.svg"
+  import { onMount } from "svelte"
 
   let adminUser = {}
   let error
@@ -23,6 +24,7 @@
   $: tenantId = $auth.tenantId
   $: multiTenancyEnabled = $admin.multiTenancy
   $: cloud = $admin.cloud
+  $: imported = $admin.importComplete
 
   async function save() {
     try {
@@ -40,6 +42,12 @@
       notifications.error(`Failed to create admin user`)
     }
   }
+
+  onMount(async () => {
+    if (!cloud) {
+      await admin.checkImportComplete()
+    }
+  })
 </script>
 
 <Modal bind:this={modal} padding={false} width="600px">
@@ -73,7 +81,7 @@
           >
             Change organisation
           </ActionButton>
-        {:else if !cloud}
+        {:else if !cloud && !imported}
           <ActionButton
             quiet
             on:click={() => {

packages/builder/src/pages/builder/app/[application]/data/index.svelte

@@ -1,6 +1,7 @@
 <script>
   import { goto } from "@roxi/routify"
   import { onMount } from "svelte"
+  import { admin } from "stores/portal"
   import CreateDatasourceModal from "components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte"
   import { datasources } from "stores/backend"
 
@@ -10,7 +11,7 @@
     $datasources.list.length > 1
 
   onMount(() => {
-    if (!setupComplete) {
+    if (!setupComplete && !$admin.isDev) {
       modal.show()
     } else {
       $goto("./table")

packages/builder/src/stores/portal/admin.js

@@ -7,8 +7,10 @@ export function createAdminStore() {
     loaded: false,
     multiTenancy: false,
     cloud: false,
+    isDev: false,
     disableAccountPortal: false,
     accountPortalUrl: "",
+    importComplete: false,
     onboardingProgress: 0,
     checklist: {
       apps: { checked: false },
@@ -45,11 +47,23 @@ export function createAdminStore() {
     }
   }
 
+  async function checkImportComplete() {
+    const response = await api.get(`/api/cloud/import/complete`)
+    if (response.status === 200) {
+      const json = await response.json()
+      admin.update(store => {
+        store.importComplete = json ? json.imported : false
+        return store
+      })
+    }
+  }
+
   async function getEnvironment() {
     let multiTenancyEnabled = false
     let cloud = false
     let disableAccountPortal = false
     let accountPortalUrl = ""
+    let isDev = false
     try {
       const response = await api.get(`/api/system/environment`)
       const json = await response.json()
@@ -57,6 +71,7 @@ export function createAdminStore() {
       cloud = json.cloud
       disableAccountPortal = json.disableAccountPortal
       accountPortalUrl = json.accountPortalUrl
+      isDev = json.isDev
     } catch (err) {
       // just let it stay disabled
     }
@@ -65,6 +80,7 @@ export function createAdminStore() {
       store.cloud = cloud
       store.disableAccountPortal = disableAccountPortal
       store.accountPortalUrl = accountPortalUrl
+      store.isDev = isDev
       return store
     })
   }
@@ -79,6 +95,7 @@ export function createAdminStore() {
   return {
     subscribe: admin.subscribe,
     init,
+    checkImportComplete,
     unload,
   }
 }

packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/cli",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "description": "Budibase CLI, for developers, self hosting and migrations.",
   "main": "src/index.js",
   "bin": {

packages/client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/client",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "license": "MPL-2.0",
   "module": "dist/budibase-client.js",
   "main": "dist/budibase-client.js",
@@ -19,9 +19,9 @@
     "dev:builder": "rollup -cw"
   },
   "dependencies": {
-    "@budibase/bbui": "^0.9.160-alpha.1",
+    "@budibase/bbui": "^0.9.167-alpha.8",
     "@budibase/standard-components": "^0.9.139",
-    "@budibase/string-templates": "^0.9.160-alpha.1",
+    "@budibase/string-templates": "^0.9.167-alpha.8",
     "regexparam": "^1.3.0",
     "shortid": "^2.2.15",
     "svelte-spa-router": "^3.0.5"

packages/client/src/components/CustomThemeWrapper.svelte

@@ -16,7 +16,10 @@
     /*  Buttons */
     --spectrum-semantic-cta-color-background-default: var(--primaryColor);
     --spectrum-semantic-cta-color-background-hover: var(--primaryColorHover);
+    --spectrum-button-primary-s-border-radius: var(--buttonBorderRadius);
     --spectrum-button-primary-m-border-radius: var(--buttonBorderRadius);
+    --spectrum-button-primary-l-border-radius: var(--buttonBorderRadius);
+    --spectrum-button-primary-xl-border-radius: var(--buttonBorderRadius);
 
     /* Loading spinners */
     --spectrum-progresscircle-medium-track-fill-color: var(--primaryColor);

packages/server/nodemon.json

@@ -1,5 +1,5 @@
 {
-  "watch": ["src"],
+  "watch": ["src", "../auth"],
   "ext": "js,ts,json",
   "ignore": ["src/**/*.spec.ts", "src/**/*.spec.js"],
   "exec": "ts-node src/index.ts"

packages/server/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/server",
   "email": "hi@budibase.com",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "description": "Budibase Web Server",
   "main": "src/index.js",
   "repository": {
@@ -24,12 +24,14 @@
     "lint": "eslint --fix src/",
     "lint:fix": "yarn run format && yarn run lint",
     "initialise": "node scripts/initialise.js",
-    "multi:enable": "node scripts/multiTenancy.js enable",
+    "env:multi:enable": "node scripts/multiTenancy.js enable",
-    "multi:disable": "node scripts/multiTenancy.js disable",
+    "env:multi:disable": "node scripts/multiTenancy.js disable",
-    "selfhost:enable": "node scripts/selfhost.js enable",
+    "env:selfhost:enable": "node scripts/selfhost.js enable",
-    "selfhost:disable": "node scripts/selfhost.js disable",
+    "env:selfhost:disable": "node scripts/selfhost.js disable",
-    "localdomain:enable": "node scripts/localdomain.js enable",
+    "env:localdomain:enable": "node scripts/localdomain.js enable",
-    "localdomain:disable": "node scripts/localdomain.js disable"
+    "env:localdomain:disable": "node scripts/localdomain.js disable",
+    "env:account:enable": "node scripts/account.js enable",
+    "env:account:disable": "node scripts/account.js disable"
   },
   "jest": {
     "preset": "ts-jest",
@@ -66,9 +68,9 @@
   "author": "Budibase",
   "license": "AGPL-3.0-or-later",
   "dependencies": {
-    "@budibase/auth": "^0.9.160-alpha.1",
+    "@budibase/auth": "^0.9.167-alpha.8",
-    "@budibase/client": "^0.9.160-alpha.1",
+    "@budibase/client": "^0.9.167-alpha.8",
-    "@budibase/string-templates": "^0.9.160-alpha.1",
+    "@budibase/string-templates": "^0.9.167-alpha.8",
     "@elastic/elasticsearch": "7.10.0",
     "@koa/router": "8.0.0",
     "@sendgrid/mail": "7.1.1",
@@ -117,6 +119,7 @@
     "to-json-schema": "0.2.5",
     "uuid": "3.3.2",
     "validate.js": "0.13.1",
+    "vm2": "^3.9.3",
     "yargs": "13.2.4",
     "zlib": "1.0.5"
   },

packages/server/scripts/account.js

@@ -0,0 +1,8 @@
+#!/usr/bin/env node
+const updateDotEnv = require("update-dotenv")
+
+const arg = process.argv.slice(2)[0]
+
+updateDotEnv({
+  DISABLE_ACCOUNT_PORTAL: arg === "enable" ? "" : "1",
+}).then(() => console.log("Updated server!"))

packages/server/src/api/controllers/application.js

@@ -86,6 +86,7 @@ async function getAppUrlIfNotInUse(ctx) {
   if (
     url &&
     deployedApps[url] != null &&
+    ctx.params != null &&
     deployedApps[url].appId !== ctx.params.appId
   ) {
     ctx.throw(400, "App name/URL is already in use.")

packages/server/src/api/controllers/cloud.js

@@ -28,15 +28,18 @@ exports.exportApps = async ctx => {
     ctx.throw(400, "Exporting only allowed in multi-tenant cloud environments.")
   }
   const apps = await getAllApps(CouchDB, { all: true })
-  const globalDBString = await exportDB(getGlobalDBName())
+  const globalDBString = await exportDB(getGlobalDBName(), {
+    filter: doc => !doc._id.startsWith(DocumentTypes.USER),
+  })
   let allDBs = {
     global: globalDBString,
   }
   for (let app of apps) {
+    const appId = app.appId || app._id
     // only export the dev apps as they will be the latest, the user can republish the apps
     // in their self hosted environment
-    if (isDevAppID(app._id)) {
+    if (isDevAppID(appId)) {
-      allDBs[app.name] = await exportDB(app._id)
+      allDBs[app.name] = await exportDB(appId)
     }
   }
   const filename = `cloud-export-${new Date().getTime()}.txt`
@@ -53,16 +56,26 @@ async function getAllDocType(db, docType) {
   return response.rows.map(row => row.doc)
 }
 
+async function hasBeenImported() {
+  if (!env.SELF_HOSTED || env.MULTI_TENANCY) {
+    return true
+  }
+  const apps = await getAllApps(CouchDB, { all: true })
+  return apps.length !== 0
+}
+
+exports.hasBeenImported = async ctx => {
+  ctx.body = {
+    imported: await hasBeenImported(),
+  }
+}
+
 exports.importApps = async ctx => {
   if (!env.SELF_HOSTED || env.MULTI_TENANCY) {
     ctx.throw(400, "Importing only allowed in self hosted environments.")
   }
-  const apps = await getAllApps(CouchDB, { all: true })
+  const beenImported = await hasBeenImported()
-  if (
+  if (beenImported || !ctx.request.files || !ctx.request.files.importFile) {
-    apps.length !== 0 ||
-    !ctx.request.files ||
-    !ctx.request.files.importFile
-  ) {
     ctx.throw(
       400,
       "Import file is required and environment must be fresh to import apps."
@@ -80,11 +93,17 @@ exports.importApps = async ctx => {
   for (let [appName, appImport] of Object.entries(dbs)) {
     await createApp(appName, appImport)
   }
-  // once apps are created clean up the global db
+
+  // if there are any users make sure to remove them
   let users = await getAllDocType(globalDb, DocumentTypes.USER)
+  let userDeletionPromises = []
   for (let user of users) {
-    delete user.tenantId
+    userDeletionPromises.push(globalDb.remove(user._id, user._rev))
   }
+  if (userDeletionPromises.length > 0) {
+    await Promise.all(userDeletionPromises)
+  }
+
   await globalDb.bulkDocs(users)
   ctx.body = {
     message: "Apps successfully imported.",

packages/server/src/api/controllers/dev.js

@@ -7,11 +7,13 @@ const { clearLock } = require("../../utilities/redis")
 const { Replication } = require("@budibase/auth").db
 const { DocumentTypes } = require("../../db/utils")
 
-async function redirect(ctx, method) {
+async function redirect(ctx, method, path = "global") {
   const { devPath } = ctx.params
   const queryString = ctx.originalUrl.split("?")[1] || ""
   const response = await fetch(
-    checkSlashesInUrl(`${env.WORKER_URL}/api/global/${devPath}?${queryString}`),
+    checkSlashesInUrl(
+      `${env.WORKER_URL}/api/${path}/${devPath}?${queryString}`
+    ),
     request(
       ctx,
       {
@@ -41,16 +43,22 @@ async function redirect(ctx, method) {
   ctx.cookies
 }
 
-exports.redirectGet = async ctx => {
+exports.buildRedirectGet = path => {
-  await redirect(ctx, "GET")
+  return async ctx => {
+    await redirect(ctx, "GET", path)
+  }
 }
 
-exports.redirectPost = async ctx => {
+exports.buildRedirectPost = path => {
-  await redirect(ctx, "POST")
+  return async ctx => {
+    await redirect(ctx, "POST", path)
+  }
 }
 
-exports.redirectDelete = async ctx => {
+exports.buildRedirectDelete = path => {
-  await redirect(ctx, "DELETE")
+  return async ctx => {
+    await redirect(ctx, "DELETE", path)
+  }
 }
 
 exports.clearLock = async ctx => {

packages/server/src/api/controllers/query.js

@@ -4,6 +4,7 @@ const { generateQueryID, getQueryParams } = require("../../db/utils")
 const { integrations } = require("../../integrations")
 const { BaseQueryVerbs } = require("../../constants")
 const env = require("../../environment")
+const ScriptRunner = require("../../utilities/scriptRunner")
 
 // simple function to append "readable" to all read queries
 function enrichQueries(input) {
@@ -28,12 +29,39 @@ function formatResponse(resp) {
       resp = { response: resp }
     }
   }
-  if (!Array.isArray(resp)) {
-    resp = [resp]
-  }
   return resp
 }
 
+async function runAndTransform(
+  integration,
+  queryVerb,
+  enrichedQuery,
+  transformer
+) {
+  let rows = formatResponse(await integration[queryVerb](enrichedQuery))
+
+  // transform as required
+  if (transformer) {
+    const runner = new ScriptRunner(transformer, { data: rows })
+    rows = runner.execute()
+  }
+
+  // needs to an array for next step
+  if (!Array.isArray(rows)) {
+    rows = [rows]
+  }
+
+  // map into JSON if just raw primitive here
+  if (rows.find(row => typeof row !== "object")) {
+    rows = rows.map(value => ({ value }))
+  }
+
+  // get all the potential fields in the schema
+  let keys = rows.flatMap(Object.keys)
+
+  return { rows, keys }
+}
+
 exports.fetch = async function (ctx) {
   const db = new CouchDB(ctx.appId)
 
@@ -122,15 +150,16 @@ exports.preview = async function (ctx) {
     ctx.throw(400, "Integration type does not exist.")
   }
 
-  const { fields, parameters, queryVerb } = ctx.request.body
+  const { fields, parameters, queryVerb, transformer } = ctx.request.body
-
   const enrichedQuery = await enrichQueryFields(fields, parameters)
-
   const integration = new Integration(datasource.config)
-  const rows = formatResponse(await integration[queryVerb](enrichedQuery))
 
-  // get all the potential fields in the schema
+  const { rows, keys } = await runAndTransform(
-  const keys = rows.flatMap(Object.keys)
+    integration,
+    queryVerb,
+    enrichedQuery,
+    transformer
+  )
 
   ctx.body = {
     rows,
@@ -158,10 +187,16 @@ exports.execute = async function (ctx) {
     query.fields,
     ctx.request.body.parameters
   )
-
   const integration = new Integration(datasource.config)
+
   // call the relevant CRUD method on the integration class
-  ctx.body = formatResponse(await integration[query.queryVerb](enrichedQuery))
+  const { rows } = await runAndTransform(
+    integration,
+    query.queryVerb,
+    enrichedQuery,
+    query.transformer
+  )
+  ctx.body = rows
   // cleanup
   if (integration.end) {
     integration.end()

packages/server/src/api/controllers/script.js

@@ -1,24 +1,9 @@
-const fetch = require("node-fetch")
+const ScriptRunner = require("../../utilities/scriptRunner")
-const vm = require("vm")
-
-class ScriptExecutor {
-  constructor(body) {
-    const code = `let fn = () => {\n${body.script}\n}; out = fn();`
-    this.script = new vm.Script(code)
-    this.context = vm.createContext(body.context)
-    this.context.fetch = fetch
-  }
-
-  execute() {
-    this.script.runInContext(this.context)
-    return this.context.out
-  }
-}
 
 exports.execute = async function (ctx) {
-  const executor = new ScriptExecutor(ctx.request.body)
+  const { script, context } = ctx.request.body
-
+  const runner = new ScriptRunner(script, context)
-  ctx.body = executor.execute()
+  ctx.body = runner.execute()
 }
 
 exports.save = async function (ctx) {

packages/server/src/api/controllers/table/utils.js

@@ -1,5 +1,4 @@
 const CouchDB = require("../../../db")
-const linkRows = require("../../../db/linkedRows")
 const csvParser = require("../../../utilities/csvParser")
 const {
   getRowParams,
@@ -93,19 +92,10 @@ exports.handleDataImport = async (appId, user, table, dataImport) => {
         }
       }
 
-      // make sure link rows are up to date
+      finalData.push(row)
-      finalData.push(
-        linkRows.updateLinks({
-          appId,
-          eventType: linkRows.EventType.ROW_SAVE,
-          row,
-          tableId: row.tableId,
-          table,
-        })
-      )
     }
 
-    await db.bulkDocs(await Promise.all(finalData))
+    await db.bulkDocs(finalData)
     let response = await db.put(table)
     table._rev = response._rev
   }

packages/server/src/api/controllers/templates.js

@@ -7,11 +7,23 @@ const DEFAULT_TEMPLATES_BUCKET =
 
 exports.fetch = async function (ctx) {
   const { type = "app" } = ctx.query
-  const response = await fetch(
+  let response,
-    `https://${DEFAULT_TEMPLATES_BUCKET}/manifest.json`
+    error = false
-  )
+  try {
-  const json = await response.json()
+    response = await fetch(`https://${DEFAULT_TEMPLATES_BUCKET}/manifest.json`)
-  ctx.body = Object.values(json.templates[type])
+    if (response.status !== 200) {
+      error = true
+    }
+  } catch (err) {
+    error = true
+  }
+  // if there is an error, simply return no templates
+  if (!error && response) {
+    const json = await response.json()
+    ctx.body = Object.values(json.templates[type])
+  } else {
+    ctx.body = []
+  }
 }
 
 // can't currently test this, have to ignore from coverage

packages/server/src/api/routes/cloud.js

@@ -9,5 +9,6 @@ router
   .get("/api/cloud/export", authorized(BUILDER), controller.exportApps)
   // has to be public, only run if apps don't exist
   .post("/api/cloud/import", controller.importApps)
+  .get("/api/cloud/import/complete", controller.hasBeenImported)
 
 module.exports = router

packages/server/src/api/routes/dev.js

@@ -6,11 +6,16 @@ const { BUILDER } = require("@budibase/auth/permissions")
 
 const router = Router()
 
-if (env.isDev() || env.isTest()) {
+function redirectPath(path) {
   router
-    .get("/api/global/:devPath(.*)", controller.redirectGet)
+    .get(`/api/${path}/:devPath(.*)`, controller.buildRedirectGet(path))
-    .post("/api/global/:devPath(.*)", controller.redirectPost)
+    .post(`/api/${path}/:devPath(.*)`, controller.buildRedirectPost(path))
-    .delete("/api/global/:devPath(.*)", controller.redirectDelete)
+    .delete(`/api/${path}/:devPath(.*)`, controller.buildRedirectDelete(path))
+}
+
+if (env.isDev() || env.isTest()) {
+  redirectPath("global")
+  redirectPath("system")
 }
 
 router

packages/server/src/api/routes/query.js

@@ -31,7 +31,8 @@ function generateQueryValidation() {
     })),
     queryVerb: Joi.string().allow().required(),
     extra: Joi.object().optional(),
-    schema: Joi.object({}).required().unknown(true)
+    schema: Joi.object({}).required().unknown(true),
+    transformer: Joi.string().optional(),
   }))
 }
 
@@ -42,6 +43,7 @@ function generateQueryPreviewValidation() {
     queryVerb: Joi.string().allow().required(),
     extra: Joi.object().optional(),
     datasourceId: Joi.string().required(),
+    transformer: Joi.string().optional(),
     parameters: Joi.object({}).required().unknown(true)
   }))
 }

packages/server/src/automations/actions.js

@@ -3,7 +3,6 @@ const createRow = require("./steps/createRow")
 const updateRow = require("./steps/updateRow")
 const deleteRow = require("./steps/deleteRow")
 const executeScript = require("./steps/executeScript")
-const bash = require("./steps/bash")
 const executeQuery = require("./steps/executeQuery")
 const outgoingWebhook = require("./steps/outgoingWebhook")
 const serverLog = require("./steps/serverLog")
@@ -14,6 +13,7 @@ const integromat = require("./steps/integromat")
 let filter = require("./steps/filter")
 let delay = require("./steps/delay")
 let queryRow = require("./steps/queryRows")
+const env = require("../environment")
 
 const ACTION_IMPLS = {
   SEND_EMAIL_SMTP: sendSmtpEmail.run,
@@ -22,7 +22,6 @@ const ACTION_IMPLS = {
   DELETE_ROW: deleteRow.run,
   OUTGOING_WEBHOOK: outgoingWebhook.run,
   EXECUTE_SCRIPT: executeScript.run,
-  EXECUTE_BASH: bash.run,
   EXECUTE_QUERY: executeQuery.run,
   SERVER_LOG: serverLog.run,
   DELAY: delay.run,
@@ -42,7 +41,6 @@ const ACTION_DEFINITIONS = {
   OUTGOING_WEBHOOK: outgoingWebhook.definition,
   EXECUTE_SCRIPT: executeScript.definition,
   EXECUTE_QUERY: executeQuery.definition,
-  EXECUTE_BASH: bash.definition,
   SERVER_LOG: serverLog.definition,
   DELAY: delay.definition,
   FILTER: filter.definition,
@@ -54,6 +52,15 @@ const ACTION_DEFINITIONS = {
   integromat: integromat.definition,
 }
 
+// don't add the bash script/definitions unless in self host
+// the fact this isn't included in any definitions means it cannot be
+// ran at all
+if (env.SELF_HOSTED) {
+  const bash = require("./steps/bash")
+  ACTION_IMPLS["EXECUTE_BASH"] = bash.run
+  ACTION_DEFINITIONS["EXECUTE_BASH"] = bash.definition
+}
+
 /* istanbul ignore next */
 exports.getAction = async function (actionName) {
   if (ACTION_IMPLS[actionName] != null) {

packages/server/src/automations/steps/discord.js

@@ -77,7 +77,7 @@ exports.run = async function ({ inputs }) {
   const { status, message } = await getFetchResponse(response)
   return {
     httpStatus: status,
-    success: status === 200,
+    success: status === 200 || status === 204,
     response: message,
   }
 }

packages/server/src/automations/tests/automation.spec.js

@@ -1,5 +1,12 @@
 jest.mock("../../utilities/usageQuota")
 jest.mock("../thread")
+jest.mock("../../utilities/redis", () => ({
+  init: jest.fn(),
+  checkTestFlag: () => {
+    return false
+  },
+}))
+
 jest.spyOn(global.console, "error")
 
 require("../../environment")

packages/server/src/automations/thread.js

@@ -10,6 +10,7 @@ const env = require("../environment")
 const usage = require("../utilities/usageQuota")
 
 const FILTER_STEP_ID = actions.ACTION_DEFINITIONS.FILTER.stepId
+const STOPPED_STATUS = { success: false, status: "STOPPED" }
 
 /**
  * The automation orchestrator is a class responsible for executing automations.
@@ -68,7 +69,13 @@ class Orchestrator {
   async execute() {
     let automation = this._automation
     const app = await this.getApp()
+    let stopped = false
     for (let step of automation.definition.steps) {
+      // execution stopped, record state for that
+      if (stopped) {
+        this.updateExecutionOutput(step.id, step.stepId, {}, STOPPED_STATUS)
+        continue
+      }
       let stepFn = await this.getStepFunctionality(step.stepId)
       step.inputs = await processObject(step.inputs, this._context)
       step.inputs = automationUtils.cleanInputValues(
@@ -86,10 +93,17 @@ class Orchestrator {
             context: this._context,
           })
         })
-        if (step.stepId === FILTER_STEP_ID && !outputs.success) {
-          break
-        }
         this._context.steps.push(outputs)
+        // if filter causes us to stop execution don't break the loop, set a var
+        // so that we can finish iterating through the steps and record that it stopped
+        if (step.stepId === FILTER_STEP_ID && !outputs.success) {
+          stopped = true
+          this.updateExecutionOutput(step.id, step.stepId, step.inputs, {
+            ...outputs,
+            ...STOPPED_STATUS,
+          })
+          continue
+        }
         this.updateExecutionOutput(step.id, step.stepId, step.inputs, outputs)
       } catch (err) {
         console.error(`Automation error - ${step.stepId} - ${err}`)
@@ -99,7 +113,7 @@ class Orchestrator {
 
     // Increment quota for automation runs
     if (!env.SELF_HOSTED && !isDevAppID(this._appId)) {
-      usage.update(usage.Properties.AUTOMATION, 1)
+      await usage.update(usage.Properties.AUTOMATION, 1)
     }
     return this.executionOutput
   }

packages/server/src/automations/triggerInfo/app.js

@@ -22,6 +22,7 @@ exports.definition = {
         fields: {
           type: "object",
           description: "Fields submitted from the app frontend",
+          customType: "triggerSchema",
         },
       },
       required: ["fields"],

packages/server/src/automations/triggers.js

@@ -81,16 +81,20 @@ exports.externalTrigger = async function (
   params,
   { getResponses } = {}
 ) {
-  if (automation.definition != null && automation.definition.trigger != null) {
+  if (
-    if (automation.definition.trigger.stepId === "APP") {
+    automation.definition != null &&
-      // values are likely to be submitted as strings, so we shall convert to correct type
+    automation.definition.trigger != null &&
-      const coercedFields = {}
+    automation.definition.trigger.stepId === definitions.APP.stepId &&
-      const fields = automation.definition.trigger.inputs.fields
+    automation.definition.trigger.stepId === "APP" &&
-      for (let key of Object.keys(fields)) {
+    !checkTestFlag(automation._id)
-        coercedFields[key] = coerce(params.fields[key], fields[key])
+  ) {
-      }
+    // values are likely to be submitted as strings, so we shall convert to correct type
-      params.fields = coercedFields
+    const coercedFields = {}
+    const fields = automation.definition.trigger.inputs.fields
+    for (let key of Object.keys(fields)) {
+      coercedFields[key] = coerce(params.fields[key], fields[key])
     }
+    params.fields = coercedFields
   }
   const data = { automation, event: params }
   if (getResponses) {

packages/server/src/definitions/datasource.ts

@@ -20,12 +20,14 @@ export enum QueryTypes {
 
 export enum DatasourceFieldTypes {
   STRING = "string",
+  LONGFORM = "longForm",
   BOOLEAN = "boolean",
   NUMBER = "number",
   PASSWORD = "password",
   LIST = "list",
   OBJECT = "object",
   JSON = "json",
+  FILE = "file",
 }
 
 export enum SourceNames {

packages/server/src/environment.js

@@ -74,9 +74,10 @@ module.exports = {
   },
 }
 
-// convert any strings to numbers if required, like "0" would be true otherwise
+// clean up any environment variable edge cases
 for (let [key, value] of Object.entries(module.exports)) {
-  if (typeof value === "string" && !isNaN(parseInt(value))) {
+  // handle the edge case of "0" to disable an environment variable
-    module.exports[key] = parseInt(value)
+  if (value === "0") {
+    module.exports[key] = 0
   }
 }

packages/server/src/integrations/postgres.ts

@@ -28,6 +28,8 @@ module PostgresModule {
     user: string
     password: string
     ssl?: boolean
+    ca?: string
+    rejectUnauthorized?: boolean
   }
 
   const SCHEMA: Integration = {
@@ -67,6 +69,16 @@ module PostgresModule {
         default: false,
         required: false,
       },
+      rejectUnauthorized: {
+        type: DatasourceFieldTypes.BOOLEAN,
+        default: false,
+        required: false,
+      },
+      ca: {
+        type: DatasourceFieldTypes.LONGFORM,
+        default: false,
+        required: false,
+      },
     },
     query: {
       create: {
@@ -144,7 +156,12 @@ module PostgresModule {
 
       let newConfig = {
         ...this.config,
-        ssl: this.config.ssl ? { rejectUnauthorized: true } : undefined,
+        ssl: this.config.ssl
+          ? {
+              rejectUnauthorized: this.config.rejectUnauthorized,
+              ca: this.config.ca,
+            }
+          : undefined,
       }
       if (!this.pool) {
         this.pool = new Pool(newConfig)

packages/server/src/integrations/rest.ts

@@ -152,13 +152,17 @@ module RestModule {
       }
     }
 
+    getUrl(path: string, queryString: string): string {
+      return `${this.config.url}/${path}?${queryString}`
+    }
+
     async create({ path = "", queryString = "", headers = {}, json = {} }) {
       this.headers = {
         ...this.config.defaultHeaders,
         ...headers,
       }
 
-      const response = await fetch(this.config.url + path + queryString, {
+      const response = await fetch(this.getUrl(path, queryString), {
         method: "POST",
         headers: this.headers,
         body: JSON.stringify(json),
@@ -173,7 +177,7 @@ module RestModule {
         ...headers,
       }
 
-      const response = await fetch(this.config.url + path + queryString, {
+      const response = await fetch(this.getUrl(path, queryString), {
         headers: this.headers,
       })
 
@@ -186,7 +190,7 @@ module RestModule {
         ...headers,
       }
 
-      const response = await fetch(this.config.url + path + queryString, {
+      const response = await fetch(this.getUrl(path, queryString), {
         method: "POST",
         headers: this.headers,
         body: JSON.stringify(json),
@@ -201,7 +205,7 @@ module RestModule {
         ...headers,
       }
 
-      const response = await fetch(this.config.url + path + queryString, {
+      const response = await fetch(this.getUrl(path, queryString), {
         method: "PATCH",
         headers: this.headers,
         body: JSON.stringify(json),
@@ -216,7 +220,7 @@ module RestModule {
         ...headers,
       }
 
-      const response = await fetch(this.config.url + path + queryString, {
+      const response = await fetch(this.getUrl(path, queryString), {
         method: "DELETE",
         headers: this.headers,
       })

packages/server/src/integrations/tests/mongo.spec.js

@@ -27,7 +27,7 @@ describe("MongoDB Integration", () => {
     const body = {
       name: "Hello"
     }
-    const response = await config.integration.create({ 
+    await config.integration.create({
       index: indexName,
       json: body,
       extra: { collection: 'testCollection', actionTypes: 'insertOne'}
@@ -54,7 +54,7 @@ describe("MongoDB Integration", () => {
       },
       extra: { collection: 'testCollection', actionTypes: 'deleteOne'}
     }
-    const response = await config.integration.delete(query)
+    await config.integration.delete(query)
     expect(config.integration.client.deleteOne).toHaveBeenCalledWith(query.json)
   })
 
@@ -65,7 +65,7 @@ describe("MongoDB Integration", () => {
       },
       extra: { collection: 'testCollection', actionTypes: 'updateOne'}
     }
-    const response = await config.integration.update(query)
+    await config.integration.update(query)
     expect(config.integration.client.updateOne).toHaveBeenCalledWith(query.json)
   })
 
@@ -75,10 +75,14 @@ describe("MongoDB Integration", () => {
     const query = {
       extra: { collection: 'testCollection', actionTypes: 'deleteOne'}
     }
-    // Weird, need to do an IIFE for jest to recognize that it throws
-    expect(() => config.integration.read(query)()).toThrow(expect.any(Object))
 
+    let error = null
+    try {
+      await config.integration.read(query)
+    } catch (err) {
+      error = err
+    }
+    expect(error).toBeDefined()
     restore()
   })
-
 })

packages/server/src/integrations/tests/rest.spec.js

@@ -1,98 +1,100 @@
-jest.mock("node-fetch", () => jest.fn(() => ({ json: jest.fn(), text: jest.fn() })))
+jest.mock("node-fetch", () =>
+  jest.fn(() => ({ json: jest.fn(), text: jest.fn() }))
+)
 const fetch = require("node-fetch")
 const RestIntegration = require("../rest")
 
 class TestConfiguration {
   constructor(config = {}) {
-    this.integration = new RestIntegration.integration(config) 
+    this.integration = new RestIntegration.integration(config)
   }
 }
 
 describe("REST Integration", () => {
   const BASE_URL = "https://myapi.com"
-  let config 
+  let config
 
   beforeEach(() => {
     config = new TestConfiguration({
-      url: BASE_URL
+      url: BASE_URL,
     })
   })
 
   it("calls the create method with the correct params", async () => {
     const query = {
-      path: "/api",
+      path: "api",
-      queryString: "?test=1",
+      queryString: "test=1",
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
       },
       json: {
-        name: "test"
+        name: "test",
-      }
+      },
     }
     const response = await config.integration.create(query)
     expect(fetch).toHaveBeenCalledWith(`${BASE_URL}/api?test=1`, {
       method: "POST",
-      body: "{\"name\":\"test\"}",
+      body: '{"name":"test"}',
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
-      }
+      },
     })
   })
 
   it("calls the read method with the correct params", async () => {
     const query = {
-      path: "/api",
+      path: "api",
-      queryString: "?test=1",
+      queryString: "test=1",
       headers: {
-        Accept: "text/html"
+        Accept: "text/html",
-      }
+      },
     }
     const response = await config.integration.read(query)
     expect(fetch).toHaveBeenCalledWith(`${BASE_URL}/api?test=1`, {
       headers: {
-        Accept: "text/html"
+        Accept: "text/html",
-      }
+      },
     })
   })
 
   it("calls the update method with the correct params", async () => {
     const query = {
-      path: "/api",
+      path: "api",
-      queryString: "?test=1",
+      queryString: "test=1",
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
       },
       json: {
-        name: "test"
+        name: "test",
-      }
+      },
     }
     const response = await config.integration.update(query)
     expect(fetch).toHaveBeenCalledWith(`${BASE_URL}/api?test=1`, {
       method: "POST",
-      body: "{\"name\":\"test\"}",
+      body: '{"name":"test"}',
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
-      }
+      },
     })
   })
 
   it("calls the delete method with the correct params", async () => {
     const query = {
-      path: "/api",
+      path: "api",
-      queryString: "?test=1",
+      queryString: "test=1",
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
       },
       json: {
-        name: "test"
+        name: "test",
-      }
+      },
     }
     const response = await config.integration.delete(query)
     expect(fetch).toHaveBeenCalledWith(`${BASE_URL}/api?test=1`, {
       method: "DELETE",
       headers: {
-        Accept: "application/json"
+        Accept: "application/json",
-      }
+      },
     })
   })
-})
+})

packages/server/src/utilities/rowProcessor/index.js

@@ -150,6 +150,10 @@ exports.processAutoColumn = processAutoColumn
  * @returns {object} The coerced value
  */
 exports.coerce = (row, type) => {
+  // no coercion specified for type, skip it
+  if (!TYPE_TRANSFORM_MAP[type]) {
+    return row
+  }
   // eslint-disable-next-line no-prototype-builtins
   if (TYPE_TRANSFORM_MAP[type].hasOwnProperty(row)) {
     return TYPE_TRANSFORM_MAP[type][row]

packages/server/src/utilities/scriptRunner.js

@@ -0,0 +1,21 @@
+const fetch = require("node-fetch")
+const { VM, VMScript } = require("vm2")
+
+class ScriptRunner {
+  constructor(script, context) {
+    const code = `let fn = () => {\n${script}\n}; results.out = fn();`
+    this.vm = new VM()
+    this.results = { out: "" }
+    this.vm.setGlobals(context)
+    this.vm.setGlobal("fetch", fetch)
+    this.vm.setGlobal("results", this.results)
+    this.script = new VMScript(code)
+  }
+
+  execute() {
+    this.vm.run(this.script)
+    return this.results.out
+  }
+}
+
+module.exports = ScriptRunner

packages/server/yarn.lock

@@ -11006,6 +11006,11 @@ verror@1.10.0:
     core-util-is "1.0.2"
     extsprintf "^1.2.0"
 
+vm2@^3.9.3:
+  version "3.9.3"
+  resolved "https://registry.yarnpkg.com/vm2/-/vm2-3.9.3.tgz#29917f6cc081cc43a3f580c26c5b553fd3c91f40"
+  integrity sha512-smLS+18RjXYMl9joyJxMNI9l4w7biW8ilSDaVRvFBDwOH8P0BK1ognFQTpg0wyQ6wIKLTblHJvROW692L/E53Q==
+
 vuvuzela@1.0.3:
   version "1.0.3"
   resolved "https://registry.yarnpkg.com/vuvuzela/-/vuvuzela-1.0.3.tgz#3be145e58271c73ca55279dd851f12a682114b0b"

packages/string-templates/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/string-templates",
-  "version": "0.9.160-alpha.1",
+  "version": "0.9.167-alpha.8",
   "description": "Handlebars wrapper for Budibase templating.",
   "main": "src/index.cjs",
   "module": "dist/bundle.mjs",
@@ -24,7 +24,8 @@
     "dayjs": "^1.10.4",
     "handlebars": "^4.7.6",
     "handlebars-utils": "^1.0.6",
-    "lodash": "^4.17.20"
+    "lodash": "^4.17.20",
+    "vm2": "^3.9.4"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^17.1.0",

packages/string-templates/rollup.config.js

@@ -7,18 +7,6 @@ import globals from "rollup-plugin-node-globals"
 
 const production = !process.env.ROLLUP_WATCH
 
-const plugins = [
-  resolve({
-    preferBuiltins: true,
-    browser: true,
-  }),
-  commonjs(),
-  globals(),
-  builtins(),
-  json(),
-  production && terser(),
-]
-
 export default [
   {
     input: "src/index.mjs",
@@ -27,18 +15,16 @@ export default [
       format: "esm",
       file: "./dist/bundle.mjs",
     },
-    plugins,
+    plugins: [
+      resolve({
+        preferBuiltins: true,
+        browser: true,
+      }),
+      commonjs(),
+      globals(),
+      builtins(),
+      json(),
+      production && terser(),
+    ],
   },
-  // This is the valid configuration for a CommonJS bundle, but since we have
-  // no use for this, it's better to leave it out.
-  // {
-  //   input: "src/index.cjs",
-  //   output: {
-  //     sourcemap: !production,
-  //     format: "cjs",
-  //     file: "./dist/bundle.cjs",
-  //     exports: "named",
-  //   },
-  //   plugins,
-  // },
 ]

packages/string-templates/src/helpers/Helper.js

@@ -1,13 +1,23 @@
 class Helper {
-  constructor(name, fn) {
+  constructor(name, fn, useValueFallback = true) {
     this.name = name
     this.fn = fn
+    this.useValueFallback = useValueFallback
   }
 
   register(handlebars) {
     // wrap the function so that no helper can cause handlebars to break
-    handlebars.registerHelper(this.name, value => {
+    handlebars.registerHelper(this.name, (value, info) => {
-      return this.fn(value) || value
+      let context = {}
+      if (info && info.data && info.data.root) {
+        context = info.data.root
+      }
+      const result = this.fn(value, context)
+      if (result == null) {
+        return this.useValueFallback ? value : null
+      } else {
+        return result
+      }
     })
   }
 

packages/string-templates/src/helpers/constants.js

@@ -19,6 +19,7 @@ module.exports.HelperFunctionNames = {
   OBJECT: "object",
   ALL: "all",
   LITERAL: "literal",
+  JS: "js",
 }
 
 module.exports.LITERAL_MARKER = "%LITERAL%"

packages/string-templates/src/helpers/index.js

@@ -1,6 +1,7 @@
 const Helper = require("./Helper")
 const { SafeString } = require("handlebars")
 const externalHandlebars = require("./external")
+const { processJS } = require("./javascript")
 const {
   HelperFunctionNames,
   HelperFunctionBuiltin,
@@ -17,6 +18,8 @@ const HELPERS = [
   new Helper(HelperFunctionNames.OBJECT, value => {
     return new SafeString(JSON.stringify(value))
   }),
+  // javascript helper
+  new Helper(HelperFunctionNames.JS, processJS, false),
   // this help is applied to all statements
   new Helper(HelperFunctionNames.ALL, value => {
     if (

packages/string-templates/src/helpers/javascript.js

@@ -0,0 +1,49 @@
+const { atob } = require("../utilities")
+
+// The method of executing JS scripts depends on the bundle being built.
+// This setter is used in the entrypoint (either index.cjs or index.mjs).
+let runJS
+module.exports.setJSRunner = runner => (runJS = runner)
+
+// Helper utility to strip square brackets from a value
+const removeSquareBrackets = value => {
+  if (!value || typeof value !== "string") {
+    return value
+  }
+  const regex = /\[+(.+)]+/
+  const matches = value.match(regex)
+  if (matches && matches[1]) {
+    return matches[1]
+  }
+  return value
+}
+
+// Our context getter function provided to JS code as $.
+// Extracts a value from context.
+const getContextValue = (path, context) => {
+  let data = context
+  path.split(".").forEach(key => {
+    if (data == null || typeof data !== "object") {
+      return null
+    }
+    data = data[removeSquareBrackets(key)]
+  })
+  return data
+}
+
+// Evaluates JS code against a certain context
+module.exports.processJS = (handlebars, context) => {
+  try {
+    // Wrap JS in a function and immediately invoke it.
+    // This is required to allow the final `return` statement to be valid.
+    const js = `function run(){${atob(handlebars)}};run();`
+
+    // Our $ context function gets a value from context
+    const sandboxContext = { $: path => getContextValue(path, context) }
+
+    // Create a sandbox with out context and run the JS
+    return runJS(js, sandboxContext)
+  } catch (error) {
+    return "Error while executing JS"
+  }
+}

packages/string-templates/src/index.cjs

@@ -1,161 +1,28 @@
-const handlebars = require("handlebars")
+const { VM } = require("vm2")
-const { registerAll } = require("./helpers/index")
+const templates = require("./index.js")
-const processors = require("./processors")
+const { setJSRunner } = require("./helpers/javascript")
-const { removeHandlebarsStatements } = require("./utilities")
-const manifest = require("../manifest.json")
-
-const hbsInstance = handlebars.create()
-registerAll(hbsInstance)
 
 /**
- * utility function to check if the object is valid
+ * CJS entrypoint for rollup
  */
-function testObject(object) {
+module.exports.isValid = templates.isValid
-  // JSON stringify will fail if there are any cycles, stops infinite recursion
+module.exports.makePropSafe = templates.makePropSafe
-  try {
+module.exports.getManifest = templates.getManifest
-    JSON.stringify(object)
+module.exports.isJSBinding = templates.isJSBinding
-  } catch (err) {
+module.exports.encodeJSBinding = templates.encodeJSBinding
-    throw "Unable to process inputs to JSON, cannot recurse"
+module.exports.decodeJSBinding = templates.decodeJSBinding
-  }
+module.exports.processStringSync = templates.processStringSync
-}
+module.exports.processObjectSync = templates.processObjectSync
+module.exports.processString = templates.processString
+module.exports.processObject = templates.processObject
 
 /**
- * Given an input object this will recurse through all props to try and update any handlebars statements within.
+ * Use vm2 to run JS scripts in a node env
- * @param {object|array} object The input structure which is to be recursed, it is important to note that
- * if the structure contains any cycles then this will fail.
- * @param {object} context The context that handlebars should fill data from.
- * @returns {Promise<object|array>} The structure input, as fully updated as possible.
  */
-module.exports.processObject = async (object, context) => {
+setJSRunner((js, context) => {
-  testObject(object)
+  const vm = new VM({
-  for (let key of Object.keys(object || {})) {
+    sandbox: context,
-    if (object[key] != null) {
+    timeout: 1000
-      let val = object[key]
+  })
-      if (typeof val === "string") {
+  return vm.run(js)
-        object[key] = await module.exports.processString(object[key], context)
+})
-      } else if (typeof val === "object") {
-        object[key] = await module.exports.processObject(object[key], context)
-      }
-    }
-  }
-  return object
-}
-
-/**
- * This will process a single handlebars containing string. If the string passed in has no valid handlebars statements
- * then nothing will occur.
- * @param {string} string The template string which is the filled from the context object.
- * @param {object} context An object of information which will be used to enrich the string.
- * @returns {Promise<string>} The enriched string, all templates should have been replaced if they can be.
- */
-module.exports.processString = async (string, context) => {
-  // TODO: carry out any async calls before carrying out async call
-  return module.exports.processStringSync(string, context)
-}
-
-/**
- * Given an input object this will recurse through all props to try and update any handlebars statements within. This is
- * a pure sync call and therefore does not have the full functionality of the async call.
- * @param {object|array} object The input structure which is to be recursed, it is important to note that
- * if the structure contains any cycles then this will fail.
- * @param {object} context The context that handlebars should fill data from.
- * @returns {object|array} The structure input, as fully updated as possible.
- */
-module.exports.processObjectSync = (object, context) => {
-  testObject(object)
-  for (let key of Object.keys(object || {})) {
-    let val = object[key]
-    if (typeof val === "string") {
-      object[key] = module.exports.processStringSync(object[key], context)
-    } else if (typeof val === "object") {
-      object[key] = module.exports.processObjectSync(object[key], context)
-    }
-  }
-  return object
-}
-
-/**
- * This will process a single handlebars containing string. If the string passed in has no valid handlebars statements
- * then nothing will occur. This is a pure sync call and therefore does not have the full functionality of the async call.
- * @param {string} string The template string which is the filled from the context object.
- * @param {object} context An object of information which will be used to enrich the string.
- * @returns {string} The enriched string, all templates should have been replaced if they can be.
- */
-module.exports.processStringSync = (string, context) => {
-  if (!exports.isValid(string)) {
-    return string
-  }
-  // take a copy of input incase error
-  const input = string
-  if (typeof string !== "string") {
-    throw "Cannot process non-string types."
-  }
-  try {
-    string = processors.preprocess(string)
-    // this does not throw an error when template can't be fulfilled, have to try correct beforehand
-    const template = hbsInstance.compile(string, {
-      strict: false,
-    })
-    return processors.postprocess(template({
-      now: new Date().toISOString(),
-      ...context,
-    }))
-  } catch (err) {
-    return removeHandlebarsStatements(input)
-  }
-}
-
-/**
- * Simple utility function which makes sure that a templating property has been wrapped in literal specifiers correctly.
- * @param {string} property The property which is to be wrapped.
- * @returns {string} The wrapped property ready to be added to a templating string.
- */
-module.exports.makePropSafe = property => {
-  return `[${property}]`.replace("[[", "[").replace("]]", "]")
-}
-
-/**
- * Checks whether or not a template string contains totally valid syntax (simply tries running it)
- * @param string The string to test for valid syntax - this may contain no templates and will be considered valid.
- * @returns {boolean} Whether or not the input string is valid.
- */
-module.exports.isValid = string => {
-  const validCases = [
-    "string",
-    "number",
-    "object",
-    "array",
-    "cannot read property",
-    "undefined",
-  ]
-  // this is a portion of a specific string always output by handlebars in the case of a syntax error
-  const invalidCases = [`expecting '`]
-  // don't really need a real context to check if its valid
-  const context = {}
-  try {
-    hbsInstance.compile(processors.preprocess(string, false))(context)
-    return true
-  } catch (err) {
-    const msg = err && err.message ? err.message : err
-    if (!msg) {
-      return false
-    }
-    const invalidCase = invalidCases.some(invalidCase =>
-      msg.toLowerCase().includes(invalidCase)
-    )
-    const validCase = validCases.some(validCase =>
-      msg.toLowerCase().includes(validCase)
-    )
-    // special case for maths functions - don't have inputs yet
-    return validCase && !invalidCase
-  }
-}
-
-/**
- * We have generated a static manifest file from the helpers that this string templating package makes use of.
- * This manifest provides information about each of the helpers and how it can be used.
- * @returns The manifest JSON which has been generated from the helpers.
- */
-module.exports.getManifest = () => {
-  return manifest
-}

packages/string-templates/src/index.js

@@ -0,0 +1,204 @@
+const handlebars = require("handlebars")
+const { registerAll } = require("./helpers/index")
+const processors = require("./processors")
+const { removeHandlebarsStatements, atob, btoa } = require("./utilities")
+const manifest = require("../manifest.json")
+
+const hbsInstance = handlebars.create()
+registerAll(hbsInstance)
+
+/**
+ * utility function to check if the object is valid
+ */
+function testObject(object) {
+  // JSON stringify will fail if there are any cycles, stops infinite recursion
+  try {
+    JSON.stringify(object)
+  } catch (err) {
+    throw "Unable to process inputs to JSON, cannot recurse"
+  }
+}
+
+/**
+ * Given an input object this will recurse through all props to try and update any handlebars statements within.
+ * @param {object|array} object The input structure which is to be recursed, it is important to note that
+ * if the structure contains any cycles then this will fail.
+ * @param {object} context The context that handlebars should fill data from.
+ * @returns {Promise<object|array>} The structure input, as fully updated as possible.
+ */
+module.exports.processObject = async (object, context) => {
+  testObject(object)
+  for (let key of Object.keys(object || {})) {
+    if (object[key] != null) {
+      let val = object[key]
+      if (typeof val === "string") {
+        object[key] = await module.exports.processString(object[key], context)
+      } else if (typeof val === "object") {
+        object[key] = await module.exports.processObject(object[key], context)
+      }
+    }
+  }
+  return object
+}
+
+/**
+ * This will process a single handlebars containing string. If the string passed in has no valid handlebars statements
+ * then nothing will occur.
+ * @param {string} string The template string which is the filled from the context object.
+ * @param {object} context An object of information which will be used to enrich the string.
+ * @returns {Promise<string>} The enriched string, all templates should have been replaced if they can be.
+ */
+module.exports.processString = async (string, context) => {
+  // TODO: carry out any async calls before carrying out async call
+  return module.exports.processStringSync(string, context)
+}
+
+/**
+ * Given an input object this will recurse through all props to try and update any handlebars statements within. This is
+ * a pure sync call and therefore does not have the full functionality of the async call.
+ * @param {object|array} object The input structure which is to be recursed, it is important to note that
+ * if the structure contains any cycles then this will fail.
+ * @param {object} context The context that handlebars should fill data from.
+ * @returns {object|array} The structure input, as fully updated as possible.
+ */
+module.exports.processObjectSync = (object, context) => {
+  testObject(object)
+  for (let key of Object.keys(object || {})) {
+    let val = object[key]
+    if (typeof val === "string") {
+      object[key] = module.exports.processStringSync(object[key], context)
+    } else if (typeof val === "object") {
+      object[key] = module.exports.processObjectSync(object[key], context)
+    }
+  }
+  return object
+}
+
+/**
+ * This will process a single handlebars containing string. If the string passed in has no valid handlebars statements
+ * then nothing will occur. This is a pure sync call and therefore does not have the full functionality of the async call.
+ * @param {string} string The template string which is the filled from the context object.
+ * @param {object} context An object of information which will be used to enrich the string.
+ * @returns {string} The enriched string, all templates should have been replaced if they can be.
+ */
+module.exports.processStringSync = (string, context) => {
+  if (!exports.isValid(string)) {
+    return string
+  }
+  // take a copy of input incase error
+  const input = string
+  if (typeof string !== "string") {
+    throw "Cannot process non-string types."
+  }
+  try {
+    string = processors.preprocess(string)
+    // this does not throw an error when template can't be fulfilled, have to try correct beforehand
+    const template = hbsInstance.compile(string, {
+      strict: false,
+    })
+    return processors.postprocess(
+      template({
+        now: new Date().toISOString(),
+        ...context,
+      })
+    )
+  } catch (err) {
+    return removeHandlebarsStatements(input)
+  }
+}
+
+/**
+ * Simple utility function which makes sure that a templating property has been wrapped in literal specifiers correctly.
+ * @param {string} property The property which is to be wrapped.
+ * @returns {string} The wrapped property ready to be added to a templating string.
+ */
+module.exports.makePropSafe = property => {
+  return `[${property}]`.replace("[[", "[").replace("]]", "]")
+}
+
+/**
+ * Checks whether or not a template string contains totally valid syntax (simply tries running it)
+ * @param string The string to test for valid syntax - this may contain no templates and will be considered valid.
+ * @returns {boolean} Whether or not the input string is valid.
+ */
+module.exports.isValid = string => {
+  const validCases = [
+    "string",
+    "number",
+    "object",
+    "array",
+    "cannot read property",
+    "undefined",
+  ]
+  // this is a portion of a specific string always output by handlebars in the case of a syntax error
+  const invalidCases = [`expecting '`]
+  // don't really need a real context to check if its valid
+  const context = {}
+  try {
+    hbsInstance.compile(processors.preprocess(string, false))(context)
+    return true
+  } catch (err) {
+    const msg = err && err.message ? err.message : err
+    if (!msg) {
+      return false
+    }
+    const invalidCase = invalidCases.some(invalidCase =>
+      msg.toLowerCase().includes(invalidCase)
+    )
+    const validCase = validCases.some(validCase =>
+      msg.toLowerCase().includes(validCase)
+    )
+    // special case for maths functions - don't have inputs yet
+    return validCase && !invalidCase
+  }
+}
+
+/**
+ * We have generated a static manifest file from the helpers that this string templating package makes use of.
+ * This manifest provides information about each of the helpers and how it can be used.
+ * @returns The manifest JSON which has been generated from the helpers.
+ */
+module.exports.getManifest = () => {
+  return manifest
+}
+
+/**
+ * Checks if a HBS expression is a valid JS HBS expression
+ * @param handlebars the HBS expression to check
+ * @returns {boolean} whether the expression is JS or not
+ */
+module.exports.isJSBinding = handlebars => {
+  return module.exports.decodeJSBinding(handlebars) != null
+}
+
+/**
+ * Encodes a raw JS string as a JS HBS expression
+ * @param javascript the JS code to encode
+ * @returns {string} the JS HBS expression
+ */
+module.exports.encodeJSBinding = javascript => {
+  return `{{ js "${btoa(javascript)}" }}`
+}
+
+/**
+ * Decodes a JS HBS expression to the raw JS code
+ * @param handlebars the JS HBS expression
+ * @returns {string|null} the raw JS code
+ */
+module.exports.decodeJSBinding = handlebars => {
+  if (!handlebars || typeof handlebars !== "string") {
+    return null
+  }
+
+  // JS is only valid if it is the only HBS expression
+  if (!handlebars.trim().startsWith("{{ js ")) {
+    return null
+  }
+
+  const captureJSRegex = new RegExp(/{{ js "(.*)" }}/)
+  const match = handlebars.match(captureJSRegex)
+  if (!match || match.length < 2) {
+    return null
+  }
+  return atob(match[1])
+}

packages/string-templates/src/index.mjs

@@ -1,12 +1,31 @@
-import templates from "./index.cjs"
+import vm from "vm"
+import templates from "./index.js"
+import { setJSRunner } from "./helpers/javascript"
 
 /**
- * This file is simply an entrypoint for rollup - makes a lot of cjs problems go away
+ * ES6 entrypoint for rollup
  */
 export const isValid = templates.isValid
 export const makePropSafe = templates.makePropSafe
 export const getManifest = templates.getManifest
+export const isJSBinding = templates.isJSBinding
+export const encodeJSBinding = templates.encodeJSBinding
+export const decodeJSBinding = templates.decodeJSBinding
 export const processStringSync = templates.processStringSync
 export const processObjectSync = templates.processObjectSync
 export const processString = templates.processString
 export const processObject = templates.processObject
+
+/**
+ * Use polyfilled vm to run JS scripts in a browser Env
+ */
+setJSRunner((js, context) => {
+  context = {
+    ...context,
+    alert: undefined,
+    setInterval: undefined,
+    setTimeout: undefined,
+  }
+  vm.createContext(context)
+  return vm.runInNewContext(js, context, { timeout: 1000 })
+})

packages/string-templates/src/utilities.js

@@ -22,3 +22,11 @@ module.exports.removeHandlebarsStatements = string => {
   }
   return string
 }
+
+module.exports.btoa = plainText => {
+  return Buffer.from(plainText, "utf-8").toString("base64")
+}
+
+module.exports.atob = base64 => {
+  return Buffer.from(base64, "base64").toString("utf-8")
+}

packages/string-templates/test/javascript.spec.js

@@ -0,0 +1,85 @@
+const { processStringSync, encodeJSBinding } = require("../src/index.cjs")
+
+const processJS = (js, context) => {
+  return processStringSync(encodeJSBinding(js), context)
+}
+
+describe("Test the JavaScript helper", () => {
+  it("should execute a simple expression", () => {
+    const output = processJS(`return 1 + 2`)
+    expect(output).toBe("3")
+  })
+
+  it("should be able to use primitive bindings", () => {
+    const output = processJS(`return $("foo")`, {
+      foo: "bar",
+    })
+    expect(output).toBe("bar")
+  })
+
+  it("should be able to use an object binding", () => {
+    const output = processJS(`return $("foo").bar`, {
+      foo: {
+        bar: "baz",
+      },
+    })
+    expect(output).toBe("baz")
+  })
+
+  it("should be able to use a complex object binding", () => {
+    const output = processJS(`return $("foo").bar[0].baz`, {
+      foo: {
+        bar: [
+          {
+            baz: "shazbat",
+          },
+        ],
+      },
+    })
+    expect(output).toBe("shazbat")
+  })
+
+  it("should be able to use a deep binding", () => {
+    const output = processJS(`return $("foo.bar.baz")`, {
+      foo: {
+        bar: {
+          baz: "shazbat",
+        },
+      },
+    })
+    expect(output).toBe("shazbat")
+  })
+
+  it("should be able to use a deep array binding", () => {
+    const output = processJS(`return $("foo.0.bar")`, {
+      foo: [
+        {
+          bar: "baz",
+        },
+      ],
+    })
+    expect(output).toBe("baz")
+  })
+
+  it("should handle errors", () => {
+    const output = processJS(`throw "Error"`)
+    expect(output).toBe("Error while executing JS")
+  })
+
+  it("should timeout after one second", () => {
+    const output = processJS(`while (true) {}`)
+    expect(output).toBe("Error while executing JS")
+  })
+
+  it("should prevent access to the process global", () => {
+    const output = processJS(`return process`)
+    expect(output).toBe("Error while executing JS")
+  })
+
+  it("should prevent sandbox escape", () => {
+    const output = processJS(
+      `return this.constructor.constructor("return process")()`
+    )
+    expect(output).toBe("Error while executing JS")
+  })
+})