Merge branch 'develop' into fix/postreleasefixes2

.github/workflows/release-develop.yml

@@ -9,7 +9,6 @@ env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}
   INTERCOM_TOKEN: ${{ secrets.INTERCOM_TOKEN }}
   POSTHOG_URL: ${{ secrets.POSTHOG_URL }}
-  SENTRY_DSN: ${{ secrets.SENTRY_DSN_SELF_HOST }}
       
 jobs:
   release:

.github/workflows/release-selfhost.yml

@@ -7,7 +7,6 @@ env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}
   INTERCOM_TOKEN: ${{ secrets.INTERCOM_TOKEN }}
   POSTHOG_URL: ${{ secrets.POSTHOG_URL }}
-  SENTRY_DSN: ${{ secrets.SENTRY_DSN_SELF_HOST }}
 
 jobs:
   release:

README.md

@@ -8,18 +8,19 @@
 </h1>
 
 <h3 align="center">
-  Build, automate and self-host internal tools in minutes
+  The low code platform you'll enjoy using
 </h3>
 <p align="center">
-  Budibase is an open-source low-code platform, helping developers and IT professionals build, automate, and ship internal tools on their own infrastructure in minutes.
+  Budibase is an open source low-code platform, and the easiest way to build internal tools that improve productivity.
 </p>
 
 <h3 align="center">
  🤖 🎨 🚀
 </h3>
+<br>
 
 <p align="center">
-  <img alt="Budibase design ui" src="https://i.imgur.com/5BnXPsN.png">
+  <img alt="Budibase design ui" src="https://res.cloudinary.com/daog6scxm/image/upload/v1633524049/ui/design-ui-wide-mobile_gdaveq.jpg">
 </p>
 
 <p align="center">
@@ -65,68 +66,25 @@
 
 - **Admin paradise.** Budibase is made to scale. With Budibase, you can self-host on your own infrastructure and globally manage users, onboarding, SMTP, apps, groups, theming and more. You can also provide users/groups with an app portal and disseminate user-management to the group manager.
 
-<br />
+<br /><br /><br />
-
----
-
-<br />
 
 ## 🏁 Get started
-Currently there are two ways to get started with Budibase; Digital Ocean, and Docker.
+
+<img src="https://res.cloudinary.com/daog6scxm/image/upload/v1634808888/logo/deploy_npl9za.png" />
+
+Deploy Budibase self-Hosted in your existing infrastructure, using Docker, Kubernetes, and Digital Ocean.
+Or use Budibase Cloud if you don't need to self-host, and would like to get started quickly.
+
+### [Get started with Budibase](https://budibase.com)
+
+
 <br /><br />
 
-### Get started with Digital Ocean
-The easiest and quickest way to get started, is to use Digital Ocean:
-<a href="https://marketplace.digitalocean.com/apps/budibase">1-click Digital Ocean deploy</a>
-
-<a href="https://marketplace.digitalocean.com/apps/budibase">
-  <img src="https://user-images.githubusercontent.com/552074/87779219-5c3b7600-c824-11ea-9898-981a8ba94f6c.png" alt="digital ocean badge">
-</a>  
-<br /><br />
-
-### Get started with Docker
-To get started, you must have docker and docker compose installed on your machine. 
-Once you have Docker installed, the process takes 5 minutes, with these four steps:
-
-1. Install the Budibase CLI.
-
-```
-$ npm i -g @budibase/cli 
-```
-
-
-2. Setup Budibase (select where to store Budibase, and the port to run it on)
-
-```
-budi hosting --init 
-```
-
-
-3. Run Budibase
-
-```
-budi hosting --start 
-```
-
-
-4. Create your admin user
-
-Enter the email and password for the new admin user.
-
-Done! You are now ready to build powerful internal tools in minutes. For additional information on how to get started and learn Budibase, visit our [docs](https://docs.budibase.com/getting-started).
-
-<br />
-
----
-
-<br />
-
 ## 🎓 Learning Budibase
 
 The Budibase documentation [lives here](https://docs.budibase.com).
 <br />
 
----
 
 <br /><br />
 
@@ -134,22 +92,17 @@ The Budibase documentation [lives here](https://docs.budibase.com).
 
 If you have a question or would like to talk with other Budibase users and join our community, please hop over to [Github discussions](https://github.com/Budibase/budibase/discussions)
 
-<img src="https://d33wubrfki0l68.cloudfront.net/e9241201fd89f9abbbdaac4fe44bb16312752abe/84013/img/hero-images/community.webp" />
+<br /><br /><br />
 
-<br /><br />
-
----
-
-<br />
 
 ## ❗ Code of conduct
 
 Budibase is dedicated to providing a welcoming, diverse, and harrassment-free experience for everyone. We expect everyone in the Budibase community to abide by our [**Code of Conduct**](https://github.com/Budibase/budibase/blob/HEAD/.github/CODE_OF_CONDUCT.md). Please read it.
 <br />
 
----
 
-<br />
+<br /><br />
+
 
 ## 🙌 Contributing to Budibase
 
@@ -168,32 +121,22 @@ Budibase is a monorepo managed by lerna. Lerna manages the building and publishi
 - [packages/server](https://github.com/Budibase/budibase/tree/HEAD/packages/server) - The budibase server. This Koa app is responsible for serving the JS for the builder and budibase apps, as well as providing the API for interaction with the database and file system.
 
 For more information, see [CONTRIBUTING.md](https://github.com/Budibase/budibase/blob/HEAD/.github/CONTRIBUTING.md)
-<br /><br />
-
----
 
 <br /><br />
 
+
 ## 📝 License
 
 Budibase is open-source, licensed as [GPL v3](https://www.gnu.org/licenses/gpl-3.0.en.html). The client and component libraries are licensed as [MPL](https://directory.fsf.org/wiki/License:MPL-2.0) - so the apps that you build can be licensed however you like.
+
 <br /><br />
 
----
-
-
-<br />
-
 ## ⭐ Stargazers over time
 
 [![Stargazers over time](https://starchart.cc/Budibase/budibase.svg)](https://starchart.cc/Budibase/budibase)
 
 If you are having issues between updates of the builder, please use the guide [here](https://github.com/Budibase/budibase/blob/HEAD/.github/CONTRIBUTING.md#troubleshooting) to clear down your environment.
 
-<br />
-
----
-
 <br /><br />
 
 ## Contributors ✨

docs/index.yaml

@@ -3,7 +3,7 @@ entries:
   budibase:
   - apiVersion: v2
     appVersion: 0.9.163
-    created: "2021-10-12T21:58:00.515555+01:00"
+    created: "2021-10-18T16:25:04.374924613+01:00"
     dependencies:
     - condition: services.couchdb.enabled
       name: couchdb
@@ -13,7 +13,35 @@ entries:
       name: ingress-nginx
       repository: https://github.com/kubernetes/ingress-nginx
       version: 3.35.0
-    description: Budibase is an open source low-code platform, helping thousands of teams build apps for their workplace in minutes.
+    description: Budibase is an open source low-code platform, helping thousands of
+      teams build apps for their workplace in minutes.
+    digest: ebac6d8631cc38b266c3689508b5123f5afc395f23bdb02738be26c7cae0b0b5
+    keywords:
+    - low-code
+    - database
+    - cluster
+    name: budibase
+    sources:
+    - https://github.com/Budibase/budibase
+    - https://budibase.com
+    type: application
+    urls:
+    - https://budibase.github.io/budibase/budibase-0.2.1.tgz
+    version: 0.2.1
+  - apiVersion: v2
+    appVersion: 0.9.163
+    created: "2021-10-18T16:25:04.36936805+01:00"
+    dependencies:
+    - condition: services.couchdb.enabled
+      name: couchdb
+      repository: https://apache.github.io/couchdb-helm
+      version: 3.3.4
+    - condition: ingress.nginx
+      name: ingress-nginx
+      repository: https://github.com/kubernetes/ingress-nginx
+      version: 3.35.0
+    description: Budibase is an open source low-code platform, helping thousands of
+      teams build apps for their workplace in minutes.
     digest: f369536c0eac1f6959d51e8ce6d74a87a7a9df29ae84fb9cbed0a273ab77429b
     keywords:
     - low-code
@@ -29,7 +57,7 @@ entries:
     version: 0.2.0
   - apiVersion: v2
     appVersion: 0.9.56
-    created: "2021-10-12T21:58:00.512062+01:00"
+    created: "2021-10-18T16:25:04.36360616+01:00"
     dependencies:
     - condition: services.couchdb.enabled
       name: couchdb
@@ -38,7 +66,8 @@ entries:
     - name: ingress-nginx
       repository: https://github.com/kubernetes/ingress-nginx
       version: 3.35.0
-    description: Budibase is an open source low-code platform, helping thousands of teams build apps for their workplace in minutes.
+    description: Budibase is an open source low-code platform, helping thousands of
+      teams build apps for their workplace in minutes.
     digest: 8dc4f2ed4d98cad5adf25936aefea680042d3e4e17832f846b961fd8708ad192
     keywords:
     - low-code
@@ -54,7 +83,7 @@ entries:
     version: 0.1.1
   - apiVersion: v2
     appVersion: 0.9.56
-    created: "2021-10-12T21:58:00.507257+01:00"
+    created: "2021-10-18T16:25:04.354504201+01:00"
     dependencies:
     - condition: services.couchdb.enabled
       name: couchdb
@@ -63,7 +92,8 @@ entries:
     - name: ingress-nginx
       repository: https://github.com/kubernetes/ingress-nginx
       version: 3.35.0
-    description: Budibase is an open source low-code platform, helping thousands of teams build apps for their workplace in minutes.
+    description: Budibase is an open source low-code platform, helping thousands of
+      teams build apps for their workplace in minutes.
     digest: 08031b0803cce0eff64472e569d454d9176119c8207aa9873a9c95ee66cc7d3f
     keywords:
     - low-code
@@ -77,4 +107,4 @@ entries:
     urls:
     - https://budibase.github.io/budibase/budibase-0.1.0.tgz
     version: 0.1.0
-generated: "2021-10-12T21:58:00.503447+01:00"
+generated: "2021-10-18T16:25:04.346266269+01:00"

hosting/docker-compose.yaml

@@ -21,7 +21,7 @@ services:
       PORT: 4002
       JWT_SECRET: ${JWT_SECRET}
       LOG_LEVEL: info
-      SENTRY_DSN: https://cc54bb0358fd4300ae97ef2273fbaf9f@o420233.ingest.sentry.io/6007553
+      SENTRY_DSN: https://a34ae347621946bf8acded18e5b7d4b8@o420233.ingest.sentry.io/5338131
       ENABLE_ANALYTICS: "true"
       REDIS_URL: redis-service:6379
       REDIS_PASSWORD: ${REDIS_PASSWORD}
@@ -48,10 +48,10 @@ services:
       COUCH_DB_USERNAME: ${COUCH_DB_USER}
       COUCH_DB_PASSWORD: ${COUCH_DB_PASSWORD}
       COUCH_DB_URL: http://${COUCH_DB_USER}:${COUCH_DB_PASSWORD}@couchdb-service:5984
+      SENTRY_DSN: https://a34ae347621946bf8acded18e5b7d4b8@o420233.ingest.sentry.io/5338131
       INTERNAL_API_KEY: ${INTERNAL_API_KEY}
       REDIS_URL: redis-service:6379
       REDIS_PASSWORD: ${REDIS_PASSWORD}
-      ACCOUNT_PORTAL_URL: https://portal.budi.live
     volumes:
       - ./logs:/logs
     depends_on:

hosting/envoy.dev.yaml.hbs

@@ -41,6 +41,7 @@ static_resources:
               - match: { prefix: "/api/" }
                 route:
                   cluster: server-dev
+                  timeout: 120s
 
               - match: { prefix: "/app_" }
                 route:

hosting/envoy.yaml

@@ -58,6 +58,7 @@ static_resources:
               - match: { prefix: "/api/" }
                 route:
                   cluster: app-service
+                  timeout: 120s
 
               - match: { prefix: "/worker/" }
                 route: 

hosting/kubernetes/budibase/Chart.yaml

@@ -22,7 +22,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.2.0
+version: 0.2.1
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

hosting/kubernetes/budibase/templates/app-service-deployment.yaml

@@ -73,11 +73,15 @@ spec:
               name: {{ template "budibase.fullname" . }}
               key: objectStoreSecret
         - name: MINIO_URL
+          {{ if .Values.services.objectStore.url }}
           value: {{ .Values.services.objectStore.url }}
+          {{ else }}
+          value: http://minio-service:{{ .Values.services.objectStore.port }}
+          {{ end }}
         - name: PORT
           value: {{ .Values.services.apps.port | quote }}
         - name: MULTI_TENANCY
-          value: "1"
+          value: {{ .Values.globals.multiTenancy | quote }}
         - name: REDIS_PASSWORD
           value: {{ .Values.services.redis.password }}
         - name: REDIS_URL

hosting/kubernetes/budibase/templates/worker-service-deployment.yaml

@@ -70,11 +70,15 @@ spec:
               name: {{ template "budibase.fullname" . }}
               key: objectStoreSecret
         - name: MINIO_URL
+          {{ if .Values.services.objectStore.url }}
           value: {{ .Values.services.objectStore.url }}
+          {{ else }}
+          value: http://minio-service:{{ .Values.services.objectStore.port }}
+          {{ end }}
         - name: PORT
           value: {{ .Values.services.worker.port | quote }}
         - name: MULTI_TENANCY
-          value: "1"
+          value: {{ .Values.globals.multiTenancy | quote }}
         - name: REDIS_PASSWORD
           value: {{ .Values.services.redis.password | quote }}
         - name: REDIS_URL
@@ -85,6 +89,8 @@ spec:
           {{ end }}
         - name: SELF_HOSTED
           value: {{ .Values.globals.selfHosted | quote }}
+        - name: SENTRY_DSN
+          value: {{ .Values.globals.sentryDSN }}
         - name: ACCOUNT_PORTAL_URL
           value: {{ .Values.globals.accountPortalUrl | quote }}
         - name: ACCOUNT_PORTAL_API_KEY

hosting/kubernetes/budibase/values.yaml

@@ -24,10 +24,12 @@ serviceAccount:
 
 podAnnotations: {}
 
-podSecurityContext: {}
+podSecurityContext:
+  {}
   # fsGroup: 2000
 
-securityContext: {}
+securityContext:
+  {}
   # capabilities:
   #   drop:
   #   - ALL
@@ -42,23 +44,24 @@ service:
 ingress:
   enabled: false
   aws: false
-  nginx: true 
+  nginx: true
-  certificateArn: ""  
+  certificateArn: ""
   className: ""
-  annotations: 
+  annotations:
     kubernetes.io/ingress.class: nginx
   hosts:
     - host: # change if using custom domain
       paths:
-      - path: /
+        - path: /
-        pathType: Prefix
+          pathType: Prefix
-        backend:
+          backend:
-          service:
+            service:
-            name: proxy-service
+              name: proxy-service
-            port:
+              port:
-              number: 10000 
+                number: 10000
 
-resources: {}
+resources:
+  {}
   # We usually recommend not to specify default resources and to leave this as a conscious
   # choice for the user. This also increases chances charts run on environments with little
   # resources, such as Minikube. If you do want to specify resources, uncomment the following
@@ -89,10 +92,11 @@ globals:
   sentryDSN: ""
   posthogToken: ""
   logLevel: info
-  selfHosted: ""
+  selfHosted: "1" # set to 0 for budibase cloud environment, set to 1 for self-hosted setup
-  accountPortalUrl: "" 
+  multiTenancy: "0" # set to 0 to disable multiple orgs, set to 1 to enable multiple orgs
+  accountPortalUrl: ""
   accountPortalApiKey: ""
-  cookieDomain: "" 
+  cookieDomain: ""
   platformUrl: ""
 
   createSecrets: true # creates an internal API key, JWT secrets and redis password for you
@@ -128,7 +132,7 @@ services:
     # password: "" # only change if pointing to existing couch server
     port: 5984
     storage: 100Mi
-  
+
   redis:
     enabled: true # disable if using external redis
     port: 6379
@@ -136,7 +140,7 @@ services:
     url: "" # only change if pointing to existing redis cluster and enabled: false
     password: "budibase" # recommended to override if using built-in redis
     storage: 100Mi
-  
+
   objectStore:
     minio: true
     browser: true
@@ -147,4 +151,3 @@ services:
     region: "" # AWS_REGION if using S3 or existing minio secret
     url: "" # only change if pointing to existing minio cluster and minio: false
     storage: 100Mi
-

lerna.json

@@ -1,5 +1,5 @@
 {
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "npmClient": "yarn",
   "packages": [
     "packages/*"

packages/auth/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/auth",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "description": "Authentication middlewares for budibase builder and apps",
   "main": "src/index.js",
   "author": "Budibase",

packages/bbui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/bbui",
   "description": "A UI solution used in the different Budibase projects.",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "license": "AGPL-3.0",
   "svelte": "src/index.js",
   "module": "dist/bbui.es.js",

packages/bbui/src/Form/Core/DatePicker.svelte

@@ -31,7 +31,11 @@
 
   const handleChange = event => {
     const [dates] = event.detail
-    dispatch("change", dates[0])
+    let newValue = dates[0]
+    if (newValue) {
+      newValue = newValue.toISOString()
+    }
+    dispatch("change", newValue)
   }
 
   const clearDateOnBackspace = event => {
@@ -57,11 +61,38 @@
     const els = document.querySelectorAll(`#${flatpickrId} input`)
     els.forEach(el => el.blur())
   }
+
+  const parseDate = val => {
+    if (!val) {
+      return null
+    }
+    let date
+    if (val instanceof Date) {
+      // Use real date obj if already parsed
+      date = val
+    } else if (isNaN(val)) {
+      // Treat as date string of some sort
+      date = new Date(val)
+    } else {
+      // Treat as numerical timestamp
+      date = new Date(parseInt(val))
+    }
+    const time = date.getTime()
+    if (isNaN(time)) {
+      return null
+    }
+    // By rounding to the nearest second we avoid locking up in an endless
+    // loop in the builder, caused by potentially enriching {{ now }} to every
+    // millisecond.
+    return new Date(Math.floor(time / 1000) * 1000)
+  }
+
+  $: console.log(value)
 </script>
 
 <Flatpickr
   bind:flatpickr
-  {value}
+  value={parseDate(value)}
   on:open={onOpen}
   on:close={onClose}
   options={flatpickrOptions}

packages/bbui/src/Form/DatePicker.svelte

@@ -13,10 +13,10 @@
   export let appendTo = undefined
 
   const dispatch = createEventDispatcher()
+
   const onChange = e => {
-    const isoString = e.detail.toISOString()
+    value = e.detail
-    value = isoString
+    dispatch("change", e.detail)
-    dispatch("change", isoString)
   }
 </script>
 

packages/bbui/src/Table/CellRenderer.svelte

@@ -25,7 +25,6 @@
     array: ArrayRenderer,
     internal: InternalRenderer,
   }
-
   $: type = schema?.type ?? "string"
   $: customRenderer = customRenderers?.find(x => x.column === schema?.name)
   $: renderer = customRenderer?.component ?? typeMap[type] ?? StringRenderer

packages/bbui/src/Tabs/Tab.svelte

@@ -8,11 +8,19 @@
   const selected = getContext("tab")
   let tab
   let tabInfo
+
   const setTabInfo = () => {
-    tabInfo = tab.getBoundingClientRect()
+    // If the tabs are being rendered inside a component which uses
-    if ($selected.title === title) {
+    // a svelte transition to enter, then this initial getBoundingClientRect
-      $selected.info = tabInfo
+    // will return an incorrect position.
-    }
+    // We just need to get this off the main thread to fix this, by using
+    // a 0ms timeout.
+    setTimeout(() => {
+      tabInfo = tab.getBoundingClientRect()
+      if ($selected.title === title) {
+        $selected.info = tabInfo
+      }
+    }, 0)
   }
 
   onMount(() => {

packages/builder/cypress/integration/createTable.spec.js

@@ -31,7 +31,7 @@ context("Create a Table", () => {
     cy.contains("nameupdated ").should("contain", "nameupdated")
   })
 
-  /*
+  
   it("edits a row", () => {
     cy.contains("button", "Edit").click({ force: true })
     cy.wait(1000)
@@ -40,7 +40,7 @@ context("Create a Table", () => {
     cy.contains("Save").click()
     cy.contains("Updated").should("have.text", "Updated")
   })
-  */
+  
   it("deletes a row", () => {
     cy.get(".spectrum-Checkbox-input").check({ force: true })
     cy.contains("Delete 1 row(s)").click()

packages/builder/cypress/integration/customThemingProperties.spec.js

@@ -5,12 +5,12 @@ context("Custom Theming Properties", () => {
       cy.navigateToFrontend()
     })
 
-    // Default Values
+    /* Default Values:
-    // Button roundness = Large
+    Button roundness = Large
-    // Accent colour = Blue 600
+    Accent colour = Blue 600
-    // Accent colour (hover) = Blue 500
+    Accent colour (hover) = Blue 500
-    // Navigation bar background colour = Gray 100
+    Navigation bar background colour = Gray 100
-    // Navigation bar text colour = Gray 800
+    Navigation bar text colour = Gray 800 */
     it("should reset the color property values", () => {
       // Open Theme modal and change colours  
       cy.get(".spectrum-ActionButton-label").contains("Theme").click()
@@ -24,6 +24,29 @@ context("Custom Theming Properties", () => {
         checkThemeColorDefaults()
     })
     
+    /* Button Roundness Values:
+    None = 0
+    Small = 4px
+    Medium = 8px
+    Large = 16px */
+    it.only("should test button roundness", () => {
+      const buttonRoundnessValues = ["0", "4px", "8px", "16px"]
+      cy.wait(1000)
+      // Add button, change roundness and confirm value
+      cy.addComponent("Button", null).then((componentId) => {
+        buttonRoundnessValues.forEach(function (item, index){
+          cy.get(".spectrum-ActionButton-label").contains("Theme").click()
+          cy.get(".setting").contains("Button roundness").parent()
+          .get(".select-wrapper").click()
+          cy.get(".spectrum-Popover").find('li').eq(index).click()
+          cy.get(".spectrum-Button").contains("View changes").click({force: true})
+          cy.reload()
+          cy.getComponent(componentId)
+        .parents(".svelte-xiqd1c").eq(0).should('have.attr', 'style').and('contains', `--buttonBorderRadius:${item}`)
+      })
+    })
+  })
+    
     const changeThemeColors = () => {
       // Changes the theme colours
       cy.get(".spectrum-FieldLabel").contains("Accent color")

packages/builder/cypress/integration/renameAnApplication.spec.js

@@ -0,0 +1,102 @@
+context("Rename an App", () => {
+    beforeEach(() => {
+      cy.login()
+      cy.createTestApp()
+    })
+
+it("should rename an unpublished application", () => {
+    const appRename = "Cypress Renamed"
+    // Rename app, Search for app, Confirm name was changed
+    cy.get(".home-logo").click()
+    renameApp(appRename)
+    cy.searchForApplication(appRename)
+    cy.get(".appGrid").find(".wrapper").should("have.length", 1)
+    })
+    
+it("Should rename a published application", () => {
+    // It is not possible to rename a published application
+    const appRename = "Cypress Renamed"
+    // Publish the app
+    cy.get(".toprightnav")
+    cy.get(".spectrum-Button").contains("Publish").click({force: true})
+    cy.get(".spectrum-Dialog-grid")
+        .within(() => {
+            // Click publish again within the modal
+            cy.get(".spectrum-Button").contains("Publish").click({force: true})
+        })
+    // Rename app, Search for app, Confirm name was changed
+    cy.get(".home-logo").click()
+    renameApp(appRename, true)
+    cy.searchForApplication(appRename)
+    cy.get(".appGrid").find(".wrapper").should("have.length", 1)
+})
+
+it("Should try to rename an application to have no name", () => {
+    cy.get(".home-logo").click()
+    renameApp(" ", false, true)
+    // Close modal and confirm name has not been changed
+    cy.get(".spectrum-Dialog-grid").contains("Cancel").click()
+    cy.searchForApplication("Cypress Tests")
+    cy.get(".appGrid").find(".wrapper").should("have.length", 1)
+})
+
+it("Should create two applications with the same name", () => {
+    // It is not possible to have applications with the same name
+    const appName = "Cypress Tests"
+    cy.visit(`localhost:${Cypress.env("PORT")}/builder`)
+    cy.wait(500)
+    cy.get(".spectrum-Button").contains("Create app").click({force: true})
+    cy.contains(/Start from scratch/).click()
+    cy.get(".spectrum-Modal")
+    .within(() => {
+      cy.get("input").eq(0).type(appName)
+      cy.get(".spectrum-ButtonGroup").contains("Create app").click({force: true})
+      cy.get(".error").should("have.text", "Another app with the same name already exists")      
+    })
+})
+
+it("should validate application names", () => {
+    // App name must be letters, numbers and spaces only
+    // This test checks numbers and special characters specifically
+    const numberName = 12345
+    const specialCharName = "£$%^"
+    cy.get(".home-logo").click()
+    renameApp(numberName)
+    cy.searchForApplication(numberName)
+    cy.get(".appGrid").find(".wrapper").should("have.length", 1)
+    renameApp(specialCharName)
+    cy.get(".error").should("have.text", "App name must be letters, numbers and spaces only")
+})
+    
+    const renameApp = (appName, published, noName) => {
+    cy.request(`localhost:${Cypress.env("PORT")}/api/applications?status=all`)
+    .its("body")
+    .then(val => {
+        if (val.length > 0) {
+            cy.get(".title > :nth-child(3) > .spectrum-Icon").click()
+            // Check for when an app is published
+            if (published == true){
+                // Should not have Edit as option, will unpublish app
+                cy.should("not.have.value", "Edit")
+                cy.get(".spectrum-Menu").contains("Unpublish").click()
+                cy.get(".spectrum-Dialog-grid").contains("Unpublish app").click()
+                cy.get(".title > :nth-child(3) > .spectrum-Icon").click()
+            }
+            cy.contains("Edit").click()
+            cy.get(".spectrum-Modal")
+            .within(() => {
+                if (noName == true){
+                    cy.get("input").clear()
+                    cy.get(".spectrum-Dialog-grid").click()
+                    .contains("App name must be letters, numbers and spaces only")
+                    return cy
+                }
+                cy.get("input").clear()
+                cy.get("input").eq(0).type(appName).should("have.value", appName).blur()
+                cy.get(".spectrum-ButtonGroup").contains("Save").click({force: true})
+                cy.wait(500)
+      })
+    }
+    })
+}
+})

packages/builder/cypress/support/commands.js

@@ -35,19 +35,12 @@ Cypress.Commands.add("login", () => {
 Cypress.Commands.add("createApp", name => {
   cy.visit(`localhost:${Cypress.env("PORT")}/builder`)
   cy.wait(500)
-  cy.contains(/Start from scratch/).click()
+  cy.contains(/Start from scratch/).dblclick()
-  cy.get(".spectrum-Modal")
+  cy.get(".spectrum-Modal").within(() => {
-    .within(() => {
+    cy.get("input").eq(0).type(name).should("have.value", name).blur()
-      cy.get("input").eq(0).type(name).should("have.value", name).blur()
+    cy.get(".spectrum-ButtonGroup").contains("Create app").click()
-      cy.get(".spectrum-ButtonGroup").contains("Create app").click()
+    cy.wait(7000)
-      cy.wait(7000)
+  })
-    })
-    .then(() => {
-      // Because we show the datasource modal on entry, we need to create a table to get rid of the modal in the future
-      cy.createInitialDatasource("initialTable")
-      cy.expandBudibaseConnection()
-      cy.get(".nav-item.selected > .content").should("be.visible")
-    })
 })
 
 Cypress.Commands.add("deleteApp", () => {
@@ -77,22 +70,6 @@ Cypress.Commands.add("createTestTableWithData", () => {
   cy.addColumn("dog", "age", "Number")
 })
 
-Cypress.Commands.add("createInitialDatasource", tableName => {
-  // Enter table name
-  cy.get(".spectrum-Modal").within(() => {
-    cy.contains("Budibase DB").trigger("mouseover").click().click()
-    cy.wait(1000)
-    cy.contains("Continue").click()
-  })
-
-  cy.get(".spectrum-Modal").within(() => {
-    cy.wait(1000)
-    cy.get("input").first().type(tableName).blur()
-    cy.get(".spectrum-ButtonGroup").contains("Create").click()
-  })
-  cy.contains(tableName).should("be.visible")
-})
-
 Cypress.Commands.add("createTable", tableName => {
   cy.contains("Budibase DB").click()
   cy.contains("Create new table").click()
@@ -247,3 +224,9 @@ Cypress.Commands.add("addCustomSourceOptions", totalOptions => {
       cy.get(".spectrum-Button").contains("Save").click({ force: true })
     })
 })
+
+Cypress.Commands.add("searchForApplication", appName => {
+  cy.get(".spectrum-Textfield").within(() => {
+    cy.get("input").eq(0).type(appName)
+  })
+})

packages/builder/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/builder",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "license": "AGPL-3.0",
   "private": true,
   "scripts": {
@@ -65,11 +65,11 @@
     }
   },
   "dependencies": {
-    "@budibase/bbui": "^0.9.167-alpha.2",
+    "@budibase/bbui": "^0.9.169-alpha.10",
-    "@budibase/client": "^0.9.167-alpha.2",
+    "@budibase/client": "^0.9.169-alpha.10",
     "@budibase/colorpicker": "1.1.2",
-    "@budibase/string-templates": "^0.9.167-alpha.2",
+    "@budibase/string-templates": "^0.9.169-alpha.10",
-    "@sentry/browser": "5.19.1",
+    "@sentry/browser": "6.0.0",
     "@spectrum-css/page": "^3.0.1",
     "@spectrum-css/vars": "^3.0.1",
     "codemirror": "^5.59.0",

packages/builder/src/builderStore/api.js

@@ -15,7 +15,7 @@ const apiCall =
     if (resp.status === 403) {
       removeCookie(Cookies.Auth)
       // reload after removing cookie, go to login
-      if (!url.includes("self")) {
+      if (!url.includes("self") && !url.includes("login")) {
         location.reload()
       }
     }

packages/builder/src/builderStore/dataBinding.js

@@ -7,11 +7,17 @@ import {
 } from "./storeUtils"
 import { store } from "builderStore"
 import { queries as queriesStores, tables as tablesStore } from "stores/backend"
-import { makePropSafe } from "@budibase/string-templates"
+import {
+  makePropSafe,
+  isJSBinding,
+  decodeJSBinding,
+  encodeJSBinding,
+} from "@budibase/string-templates"
 import { TableNames } from "../constants"
 
 // Regex to match all instances of template strings
 const CAPTURE_VAR_INSIDE_TEMPLATE = /{{([^}]+)}}/g
+const CAPTURE_VAR_INSIDE_JS = /\$\("([^")]+)"\)/g
 const CAPTURE_HBS_TEMPLATE = /{{[\S\s]*?}}/g
 
 /**
@@ -430,6 +436,15 @@ function replaceBetween(string, start, end, replacement) {
  * utility function for the readableToRuntimeBinding and runtimeToReadableBinding.
  */
 function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
+  // Decide from base64 if using JS
+  const isJS = isJSBinding(textWithBindings)
+  if (isJS) {
+    textWithBindings = decodeJSBinding(textWithBindings)
+  }
+
+  // Determine correct regex to find bindings to replace
+  const regex = isJS ? CAPTURE_VAR_INSIDE_JS : CAPTURE_VAR_INSIDE_TEMPLATE
+
   const convertFrom =
     convertTo === "runtimeBinding" ? "readableBinding" : "runtimeBinding"
   if (typeof textWithBindings !== "string") {
@@ -441,7 +456,7 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     .sort((a, b) => {
       return b.length - a.length
     })
-  const boundValues = textWithBindings.match(CAPTURE_VAR_INSIDE_TEMPLATE) || []
+  const boundValues = textWithBindings.match(regex) || []
   let result = textWithBindings
   for (let boundValue of boundValues) {
     let newBoundValue = boundValue
@@ -449,7 +464,7 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     // in the search, working from longest to shortest so always use best match first
     let searchString = newBoundValue
     for (let from of convertFromProps) {
-      if (shouldReplaceBinding(newBoundValue, from, convertTo)) {
+      if (isJS || shouldReplaceBinding(newBoundValue, from, convertTo)) {
         const binding = bindableProperties.find(el => el[convertFrom] === from)
         let idx
         do {
@@ -457,7 +472,7 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
           idx = searchString.indexOf(from)
           if (idx !== -1) {
             let end = idx + from.length,
-              searchReplace = Array(binding[convertTo].length).join("*")
+              searchReplace = Array(binding[convertTo].length + 1).join("*")
             // blank out parts of the search string
             searchString = replaceBetween(searchString, idx, end, searchReplace)
             newBoundValue = replaceBetween(
@@ -472,6 +487,12 @@ function bindingReplacement(bindableProperties, textWithBindings, convertTo) {
     }
     result = result.replace(boundValue, newBoundValue)
   }
+
+  // Re-encode to base64 if using JS
+  if (isJS) {
+    result = encodeJSBinding(result)
+  }
+
   return result
 }
 

packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte

@@ -103,7 +103,7 @@
           <Detail size="S">{block?.name?.toUpperCase() || ""}</Detail>
         </div>
       </div>
-      {#if testResult}
+      {#if testResult && testResult[0]}
         <span on:click={() => resultsModal.show()}>
           <StatusLight
             positive={isTrigger || testResult[0].outputs?.success}

packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte

@@ -194,6 +194,7 @@
             value={inputData[key]}
             on:change={e => onChange(e, key)}
             {bindings}
+            allowJS={false}
           />
         {/if}
       {:else if value.customType === "query"}
@@ -259,6 +260,7 @@
               value={inputData[key]}
               on:change={e => onChange(e, key)}
               {bindings}
+              allowJS={false}
             />
           </div>
         {/if}

packages/builder/src/components/automation/SetupPanel/QueryParamSelector.svelte

@@ -4,12 +4,23 @@
   import { Select } from "@budibase/bbui"
   import DrawerBindableInput from "../../common/bindings/DrawerBindableInput.svelte"
   import AutomationBindingPanel from "../../common/bindings/ServerBindingPanel.svelte"
+  import { createEventDispatcher } from "svelte"
 
   const dispatch = createEventDispatcher()
 
   export let value
   export let bindings
 
+  const onChangeQuery = e => {
+    value.queryId = e.detail
+    dispatch("change", value)
+  }
+
+  const onChange = (e, field) => {
+    value[field.name] = e.detail
+    dispatch("change", value)
+  }
+
   $: query = $queries.list.find(query => query._id === value?.queryId)
   $: parameters = query?.parameters ?? []
 </script>
@@ -17,11 +28,8 @@
 <div class="block-field">
   <Select
     label="Query"
+    on:change={onChangeQuery}
     value={value.queryId}
-    on:change={e => {
-      value = { queryId: e.detail }
-      dispatch("change", value)
-    }}
     options={$queries.list}
     getOptionValue={query => query._id}
     getOptionLabel={query => query.name}
@@ -35,14 +43,12 @@
         panel={AutomationBindingPanel}
         extraThin
         value={value[field.name]}
-        on:change={e => {
+        on:change={e => onChange(e, field)}
-          value[field.name] = e.detail
-          dispatch("change", value)
-        }}
         label={field.name}
         type="string"
         {bindings}
         fillWidth={true}
+        allowJS={false}
       />
     {/each}
   </div>

packages/builder/src/components/automation/SetupPanel/RowSelector.svelte

@@ -110,6 +110,7 @@
               type="string"
               {bindings}
               fillWidth={true}
+              allowJS={false}
             />
           {/if}
         {/if}

packages/builder/src/components/backend/DataTable/DataTable.svelte

@@ -16,33 +16,34 @@
   import { Pagination } from "@budibase/bbui"
 
   let hideAutocolumns = true
-  let schema
   $: isUsersTable = $tables.selected?._id === TableNames.USERS
   $: type = $tables.selected?.type
   $: isInternal = type !== "external"
-  $: {
+  $: schema = $tables.selected?.schema
-    schema = $tables.selected?.schema
+  $: enrichedSchema = enrichSchema($tables.selected?.schema)
+  $: id = $tables.selected?._id
+  $: search = searchTable(id)
+  $: columnOptions = Object.keys($search.schema || {})
 
-    // Manually add these as we don't want them to be 'real' auto-columns
+  const enrichSchema = schema => {
-    schema._id = {
+    let tempSchema = { ...schema }
+    tempSchema._id = {
       type: "internal",
       editable: false,
       displayName: "ID",
       autocolumn: true,
     }
     if (isInternal) {
-      schema._rev = {
+      tempSchema._rev = {
         type: "internal",
         editable: false,
         displayName: "Revision",
         autocolumn: true,
       }
     }
-  }
-  $: id = $tables.selected?._id
-  $: search = searchTable(id)
-  $: columnOptions = Object.keys($search.schema || {})
 
+    return tempSchema
+  }
   // Fetches new data whenever the table changes
   const searchTable = tableId => {
     return fetchTableData({
@@ -85,7 +86,7 @@
 <div>
   <Table
     title={$tables.selected?.name}
-    {schema}
+    schema={enrichedSchema}
     {type}
     tableId={id}
     data={$search.rows}

packages/builder/src/components/backend/DataTable/formula.js

@@ -4,10 +4,15 @@ import { get as svelteGet } from "svelte/store"
 
 // currently supported level of relationship depth (server side)
 const MAX_DEPTH = 1
+
+//https://github.com/Budibase/budibase/issues/3030
+const internalType = "internal"
+
 const TYPES_TO_SKIP = [
   FIELDS.FORMULA.type,
   FIELDS.LONGFORM.type,
   FIELDS.ATTACHMENT.type,
+  internalType,
 ]
 
 export function getBindings({
@@ -53,6 +58,7 @@ export function getBindings({
     const field = Object.values(FIELDS).find(
       field => field.type === schema.type
     )
+
     const label = path == null ? column : `${path}.0.${column}`
     // only supply a description for relationship paths
     const description =

packages/builder/src/components/backend/DataTable/modals/FilterModal.svelte

@@ -82,34 +82,30 @@
 
   function isMultipleChoice(field) {
     return (
-      (viewTable.schema[field].constraints &&
+      viewTable.schema[field]?.constraints?.inclusion?.length ||
-        viewTable.schema[field].constraints.inclusion &&
+      viewTable.schema[field]?.type === "boolean"
-        viewTable.schema[field].constraints.inclusion.length) ||
-      viewTable.schema[field].type === "boolean"
     )
   }
 
   function fieldOptions(field) {
-    return viewTable.schema[field].type === "options"
+    return viewTable.schema[field]?.type === "options"
-      ? viewTable.schema[field].constraints.inclusion
+      ? viewTable.schema[field]?.constraints.inclusion
       : [true, false]
   }
 
   function isDate(field) {
-    return viewTable.schema[field].type === "datetime"
+    return viewTable.schema[field]?.type === "datetime"
   }
 
   function isNumber(field) {
-    return viewTable.schema[field].type === "number"
+    return viewTable.schema[field]?.type === "number"
   }
 
   const fieldChanged = filter => ev => {
-    // reset if type changed
+    // Reset if type changed
-    if (
+    const oldType = viewTable.schema[filter.key]?.type
-      filter.key &&
+    const newType = viewTable.schema[ev.detail]?.type
-      ev.detail &&
+    if (filter.key && ev.detail && oldType !== newType) {
-      viewTable.schema[filter.key].type !== viewTable.schema[ev.detail].type
-    ) {
       filter.value = ""
     }
   }

packages/builder/src/components/common/CodeMirrorEditor.svelte

@@ -0,0 +1,159 @@
+<script context="module">
+  import { Label } from "@budibase/bbui"
+
+  export const EditorModes = {
+    JS: {
+      name: "javascript",
+      json: false,
+    },
+    JSON: {
+      name: "javascript",
+      json: true,
+    },
+    SQL: {
+      name: "sql",
+    },
+    Handlebars: {
+      name: "handlebars",
+      base: "text/html",
+    },
+  }
+</script>
+
+<script>
+  import CodeMirror from "components/integration/codemirror"
+  import { themeStore } from "builderStore"
+  import { createEventDispatcher, onMount } from "svelte"
+
+  export let mode = EditorModes.JS
+  export let value = ""
+  export let height = 300
+  export let resize = "none"
+  export let readonly = false
+  export let hints = []
+  export let label
+
+  const dispatch = createEventDispatcher()
+  let textarea
+  let editor
+
+  // Keep editor up to date with value
+  $: editor?.setValue(value || "")
+
+  // Creates an instance of a code mirror editor
+  async function createEditor(mode, value) {
+    if (!CodeMirror || !textarea || editor) {
+      return
+    }
+
+    // Configure CM options
+    const lightTheme = $themeStore.theme.includes("light")
+    const options = {
+      mode,
+      value: value || "",
+      readOnly: readonly,
+      theme: lightTheme ? "default" : "tomorrow-night-eighties",
+
+      // Style
+      lineNumbers: true,
+      lineWrapping: true,
+      indentWithTabs: true,
+      indentUnit: 2,
+      tabSize: 2,
+
+      // QOL addons
+      extraKeys: { "Ctrl-Space": "autocomplete" },
+      styleActiveLine: { nonEmpty: true },
+      autoCloseBrackets: true,
+      matchBrackets: true,
+    }
+
+    // Register hints plugin if desired
+    if (hints?.length) {
+      CodeMirror.registerHelper("hint", "dictionaryHint", function (editor) {
+        const cursor = editor.getCursor()
+        return {
+          list: hints,
+          from: CodeMirror.Pos(cursor.line, cursor.ch),
+          to: CodeMirror.Pos(cursor.line, cursor.ch),
+        }
+      })
+      CodeMirror.commands.autocomplete = function (cm) {
+        CodeMirror.showHint(cm, CodeMirror.hint.dictionaryHint)
+      }
+    }
+
+    // Construct CM instance
+    editor = CodeMirror.fromTextArea(textarea, options)
+
+    // Use a blur handler to update the value
+    editor.on("blur", instance => {
+      dispatch("change", instance.getValue())
+    })
+  }
+
+  // Export a function to expose caret position
+  export const getCaretPosition = () => {
+    const cursor = editor.getCursor()
+    return {
+      start: cursor.ch,
+      end: cursor.ch,
+    }
+  }
+
+  onMount(() => {
+    // Create the editor with initial value
+    createEditor(mode, value)
+
+    // Clean up editor on unmount
+    return () => {
+      if (editor) {
+        editor.toTextArea()
+      }
+    }
+  })
+</script>
+
+{#if label}
+  <div style="margin-bottom: var(--spacing-s)">
+    <Label small>{label}</Label>
+  </div>
+{/if}
+<div
+  style={`--code-mirror-height: ${height}px; --code-mirror-resize: ${resize}`}
+>
+  <textarea tabindex="0" bind:this={textarea} readonly {value} />
+</div>
+
+<style>
+  div :global(.CodeMirror) {
+    height: var(--code-mirror-height);
+    min-height: var(--code-mirror-height);
+    font-family: monospace;
+    line-height: 1.3;
+    border: var(--spectrum-alias-border-size-thin) solid;
+    border-color: var(--spectrum-alias-border-color);
+    border-radius: var(--border-radius-s);
+    resize: var(--code-mirror-resize);
+    overflow: hidden;
+  }
+
+  /* Override default active line highlight colour in dark theme */
+  div
+    :global(.CodeMirror-focused.cm-s-tomorrow-night-eighties
+      .CodeMirror-activeline-background) {
+    background: rgba(255, 255, 255, 0.075);
+  }
+
+  /* Remove active line styling when not focused */
+  div
+    :global(.CodeMirror:not(.CodeMirror-focused)
+      .CodeMirror-activeline-background) {
+    background: unset;
+  }
+
+  /* Add a spectrum themed border when focused */
+  div :global(.CodeMirror-focused) {
+    border-color: var(--spectrum-alias-border-color-mouse-focus);
+  }
+</style>

packages/builder/src/components/common/bindings/BindingPanel.svelte

@@ -1,32 +1,98 @@
 <script>
   import groupBy from "lodash/fp/groupBy"
-  import { Search, TextArea, DrawerContent } from "@budibase/bbui"
+  import {
-  import { createEventDispatcher } from "svelte"
+    Search,
-  import { isValid } from "@budibase/string-templates"
+    TextArea,
+    DrawerContent,
+    Tabs,
+    Tab,
+    Body,
+    Layout,
+  } from "@budibase/bbui"
+  import { createEventDispatcher, onMount } from "svelte"
+  import {
+    isValid,
+    decodeJSBinding,
+    encodeJSBinding,
+  } from "@budibase/string-templates"
   import { readableToRuntimeBinding } from "builderStore/dataBinding"
   import { handlebarsCompletions } from "constants/completions"
-  import { addToText } from "./utils"
+  import { addHBSBinding, addJSBinding } from "./utils"
+  import CodeMirrorEditor from "components/common/CodeMirrorEditor.svelte"
 
   const dispatch = createEventDispatcher()
 
   export let bindableProperties
   export let value = ""
   export let valid
+  export let allowJS = false
 
   let helpers = handlebarsCompletions()
   let getCaretPosition
   let search = ""
+  let initialValueJS = value?.startsWith("{{ js ")
+  let mode = initialValueJS ? "JavaScript" : "Handlebars"
+  let jsValue = initialValueJS ? value : null
+  let hbsValue = initialValueJS ? null : value
 
-  $: valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+  $: usingJS = mode === "JavaScript"
-  $: dispatch("change", value)
   $: ({ context } = groupBy("type", bindableProperties))
   $: searchRgx = new RegExp(search, "ig")
-  $: filteredColumns = context?.filter(context => {
+  $: filteredBindings = context?.filter(context => {
     return context.readableBinding.match(searchRgx)
   })
   $: filteredHelpers = helpers?.filter(helper => {
     return helper.label.match(searchRgx) || helper.description.match(searchRgx)
   })
+
+  const updateValue = value => {
+    valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+    if (valid) {
+      dispatch("change", value)
+    }
+  }
+
+  // Adds a HBS helper to the expression
+  const addHelper = helper => {
+    hbsValue = addHBSBinding(value, getCaretPosition(), helper.text)
+    updateValue(hbsValue)
+  }
+
+  // Adds a data binding to the expression
+  const addBinding = binding => {
+    if (usingJS) {
+      let js = decodeJSBinding(jsValue)
+      js = addJSBinding(js, getCaretPosition(), binding.readableBinding)
+      jsValue = encodeJSBinding(js)
+      updateValue(jsValue)
+    } else {
+      hbsValue = addHBSBinding(
+        hbsValue,
+        getCaretPosition(),
+        binding.readableBinding
+      )
+      updateValue(hbsValue)
+    }
+  }
+
+  const onChangeMode = e => {
+    mode = e.detail
+    updateValue(mode === "JavaScript" ? jsValue : hbsValue)
+  }
+
+  const onChangeHBSValue = e => {
+    hbsValue = e.detail
+    updateValue(hbsValue)
+  }
+
+  const onChangeJSValue = e => {
+    jsValue = encodeJSBinding(e.detail)
+    updateValue(jsValue)
+  }
+
+  onMount(() => {
+    valid = isValid(readableToRuntimeBinding(bindableProperties, value))
+  })
 </script>
 
 <DrawerContent>
@@ -36,32 +102,24 @@
         <div class="heading">Search</div>
         <Search placeholder="Search" bind:value={search} />
       </section>
-      {#if filteredColumns?.length}
+      {#if filteredBindings?.length}
         <section>
           <div class="heading">Bindable Values</div>
           <ul>
-            {#each filteredColumns as { readableBinding }}
+            {#each filteredBindings as binding}
-              <li
+              <li on:click={() => addBinding(binding)}>
-                on:click={() => {
+                {binding.readableBinding}
-                  value = addToText(value, getCaretPosition(), readableBinding)
-                }}
-              >
-                {readableBinding}
               </li>
             {/each}
           </ul>
         </section>
       {/if}
-      {#if filteredHelpers?.length}
+      {#if filteredHelpers?.length && !usingJS}
         <section>
           <div class="heading">Helpers</div>
           <ul>
             {#each filteredHelpers as helper}
-              <li
+              <li on:click={() => addHelper(helper)}>
-                on:click={() => {
-                  value = addToText(value, getCaretPosition(), helper.text)
-                }}
-              >
                 <div class="helper">
                   <div class="helper__name">{helper.displayText}</div>
                   <div class="helper__description">
@@ -77,24 +135,56 @@
     </div>
   </svelte:fragment>
   <div class="main">
-    <TextArea
+    <Tabs selected={mode} on:select={onChangeMode}>
-      bind:getCaretPosition
+      <Tab title="Handlebars">
-      bind:value
+        <div class="main-content">
-      placeholder="Add text, or click the objects on the left to add them to the textbox."
+          <TextArea
-    />
+            bind:getCaretPosition
-    {#if !valid}
+            value={hbsValue}
-      <p class="syntax-error">
+            on:change={onChangeHBSValue}
-        Current Handlebars syntax is invalid, please check the guide
+            placeholder="Add text, or click the objects on the left to add them to the textbox."
-        <a href="https://handlebarsjs.com/guide/">here</a>
+          />
-        for more details.
+          {#if !valid}
-      </p>
+            <p class="syntax-error">
-    {/if}
+              Current Handlebars syntax is invalid, please check the guide
+              <a href="https://handlebarsjs.com/guide/">here</a>
+              for more details.
+            </p>
+          {/if}
+        </div>
+      </Tab>
+      {#if allowJS}
+        <Tab title="JavaScript">
+          <div class="main-content">
+            <Layout noPadding gap="XS">
+              <CodeMirrorEditor
+                bind:getCaretPosition
+                height={200}
+                value={decodeJSBinding(jsValue)}
+                on:change={onChangeJSValue}
+                hints={context?.map(x => `$("${x.readableBinding}")`)}
+              />
+              <Body size="S">
+                JavaScript expressions are executed as functions, so ensure that
+                your expression returns a value.
+              </Body>
+            </Layout>
+          </div>
+        </Tab>
+      {/if}
+    </Tabs>
   </div>
 </DrawerContent>
 
 <style>
   .main :global(textarea) {
-    min-height: 150px !important;
+    min-height: 202px !important;
+  }
+  .main {
+    margin: calc(-1 * var(--spacing-xl));
+  }
+  .main-content {
+    padding: var(--spacing-s) var(--spacing-xl);
   }
 
   .container {

packages/builder/src/components/common/bindings/DrawerBindableCombobox.svelte

@@ -6,6 +6,7 @@
   } from "builderStore/dataBinding"
   import BindingPanel from "components/common/bindings/BindingPanel.svelte"
   import { createEventDispatcher } from "svelte"
+  import { isJSBinding } from "@budibase/string-templates"
 
   export let panel = BindingPanel
   export let value = ""
@@ -15,11 +16,14 @@
   export let label
   export let disabled = false
   export let options
+  export let allowJS = true
 
   const dispatch = createEventDispatcher()
   let bindingDrawer
+
   $: readableValue = runtimeToReadableBinding(bindings, value)
   $: tempValue = readableValue
+  $: isJS = isJSBinding(value)
 
   const handleClose = () => {
     onChange(tempValue)
@@ -35,7 +39,7 @@
   <Combobox
     {label}
     {disabled}
-    value={readableValue}
+    value={isJS ? "(JavaScript function)" : readableValue}
     on:change={event => onChange(event.detail)}
     {placeholder}
     {options}
@@ -58,6 +62,7 @@
     close={handleClose}
     on:change={event => (tempValue = event.detail)}
     bindableProperties={bindings}
+    {allowJS}
   />
 </Drawer>
 

packages/builder/src/components/common/bindings/DrawerBindableInput.svelte

@@ -6,6 +6,7 @@
   } from "builderStore/dataBinding"
   import BindingPanel from "components/common/bindings/BindingPanel.svelte"
   import { createEventDispatcher } from "svelte"
+  import { isJSBinding } from "@budibase/string-templates"
 
   export let panel = BindingPanel
   export let value = ""
@@ -15,12 +16,15 @@
   export let label
   export let disabled = false
   export let fillWidth
+  export let allowJS = true
 
   const dispatch = createEventDispatcher()
   let bindingDrawer
   let valid = true
+
   $: readableValue = runtimeToReadableBinding(bindings, value)
   $: tempValue = readableValue
+  $: isJS = isJSBinding(value)
 
   const saveBinding = () => {
     onChange(tempValue)
@@ -36,7 +40,7 @@
   <Input
     {label}
     {disabled}
-    value={readableValue}
+    value={isJS ? "(JavaScript function)" : readableValue}
     on:change={event => onChange(event.detail)}
     {placeholder}
   />
@@ -60,6 +64,7 @@
     value={readableValue}
     on:change={event => (tempValue = event.detail)}
     bindableProperties={bindings}
+    {allowJS}
   />
 </Drawer>
 

packages/builder/src/components/common/bindings/ServerBindingPanel.svelte

@@ -5,7 +5,7 @@
   import { isValid } from "@budibase/string-templates"
   import { handlebarsCompletions } from "constants/completions"
   import { readableToRuntimeBinding } from "builderStore/dataBinding"
-  import { addToText } from "./utils"
+  import { addHBSBinding } from "./utils"
 
   const dispatch = createEventDispatcher()
 
@@ -47,7 +47,7 @@
               {#each bindings as binding}
                 <li
                   on:click={() => {
-                    value = addToText(value, getCaretPosition(), binding)
+                    value = addHBSBinding(value, getCaretPosition(), binding)
                   }}
                 >
                   <span class="binding__label">{binding.label}</span>
@@ -71,7 +71,7 @@
             {#each filteredHelpers as helper}
               <li
                 on:click={() => {
-                  value = addToText(value, getCaretPosition(), helper.text)
+                  value = addHBSBinding(value, getCaretPosition(), helper.text)
                 }}
               >
                 <div class="helper">

packages/builder/src/components/common/bindings/utils.js

@@ -1,4 +1,4 @@
-export function addToText(value, caretPos, binding) {
+export function addHBSBinding(value, caretPos, binding) {
   binding = typeof binding === "string" ? binding : binding.path
   value = value == null ? "" : value
   if (!value.includes("{{") && !value.includes("}}")) {
@@ -14,3 +14,18 @@ export function addToText(value, caretPos, binding) {
   }
   return value
 }
+
+export function addJSBinding(value, caretPos, binding) {
+  binding = typeof binding === "string" ? binding : binding.path
+  value = value == null ? "" : value
+  binding = `$("${binding}")`
+  if (caretPos.start) {
+    value =
+      value.substring(0, caretPos.start) +
+      binding +
+      value.substring(caretPos.end, value.length)
+  } else {
+    value += binding
+  }
+  return value
+}

packages/builder/src/components/design/PropertiesPanel/PropertyControls/FormFieldSelect.svelte

@@ -20,12 +20,20 @@
   $: schema = getSchemaForDatasource($currentAsset, datasource, true).schema
   $: options = getOptions(schema, type)
 
-  const getOptions = (schema, fieldType) => {
+  const getOptions = (schema, type) => {
     let entries = Object.entries(schema ?? {})
-    if (fieldType) {
+
-      fieldType = fieldType.split("/")[1]
+    let types = []
-      entries = entries.filter(entry => entry[1].type === fieldType)
+    if (type === "field/options") {
+      // allow options to be used on both options and string fields
+      types = [type, "field/string"]
+    } else {
+      types = [type]
     }
+
+    types = types.map(type => type.split("/")[1])
+    entries = entries.filter(entry => types.includes(entry[1].type))
+
     return entries.map(entry => entry[0])
   }
 </script>

packages/builder/src/components/design/PropertiesPanel/PropertyControls/Input.svelte

@@ -0,0 +1,15 @@
+<script>
+  import { Input } from "@budibase/bbui"
+  import { isJSBinding } from "@budibase/string-templates"
+
+  export let value
+
+  $: isJS = isJSBinding(value)
+</script>
+
+<Input
+  {...$$props}
+  value={isJS ? "(JavaScript function)" : value}
+  readonly={isJS}
+  on:change
+/>

packages/builder/src/components/design/PropertiesPanel/PropertyControls/PropertyControl.svelte

@@ -105,6 +105,7 @@
           value={safeValue}
           on:change={e => (tempValue = e.detail)}
           bindableProperties={bindings}
+          allowJS
         />
       </Drawer>
     {/if}

packages/builder/src/components/design/PropertiesPanel/PropertyControls/componentSettings.js

@@ -1,4 +1,4 @@
-import { Checkbox, Input, Select, Stepper } from "@budibase/bbui"
+import { Checkbox, Select, Stepper } from "@budibase/bbui"
 import DataSourceSelect from "./DataSourceSelect.svelte"
 import DataProviderSelect from "./DataProviderSelect.svelte"
 import EventsEditor from "./EventsEditor"
@@ -15,6 +15,7 @@ import URLSelect from "./URLSelect.svelte"
 import OptionsEditor from "./OptionsEditor/OptionsEditor.svelte"
 import FormFieldSelect from "./FormFieldSelect.svelte"
 import ValidationEditor from "./ValidationEditor/ValidationEditor.svelte"
+import Input from "./Input.svelte"
 
 const componentMap = {
   text: Input,

packages/builder/src/components/integration/QueryViewer.svelte

@@ -21,12 +21,15 @@
   import ParameterBuilder from "components/integration/QueryParameterBuilder.svelte"
   import { datasources, integrations, queries } from "stores/backend"
   import { capitalise } from "../../helpers"
+  import CodeMirrorEditor from "components/common/CodeMirrorEditor.svelte"
 
   export let query
   export let fields = []
 
   let parameters
   let data = []
+  const transformerDocs =
+    "https://docs.budibase.com/building-apps/data/transformers"
   const typeOptions = [
     { label: "Text", value: "STRING" },
     { label: "Number", value: "NUMBER" },
@@ -52,6 +55,11 @@
   $: readQuery = query.queryVerb === "read" || query.readable
   $: queryInvalid = !query.name || (readQuery && data.length === 0)
 
+  // seed the transformer
+  if (query && !query.transformer) {
+    query.transformer = "return data"
+  }
+
   function newField() {
     fields = [...fields, {}]
   }
@@ -74,6 +82,7 @@
       const response = await api.post(`/api/queries/preview`, {
         fields: query.fields,
         queryVerb: query.queryVerb,
+        transformer: query.transformer,
         parameters: query.parameters.reduce(
           (acc, next) => ({
             ...acc,
@@ -160,12 +169,34 @@
       <IntegrationQueryEditor
         {datasource}
         {query}
-        height={300}
+        height={200}
         schema={queryConfig[query.queryVerb]}
         bind:parameters
       />
       <Divider />
     </div>
+    <div class="config">
+      <div class="help-heading">
+        <Heading size="S">Transformer</Heading>
+        <Icon
+          on:click={() => window.open(transformerDocs)}
+          hoverable
+          name="Help"
+          size="L"
+        />
+      </div>
+      <Body size="S"
+        >Add a JavaScript function to transform the query result.</Body
+      >
+      <CodeMirrorEditor
+        height={200}
+        label="Transformer"
+        value={query.transformer}
+        resize="vertical"
+        on:change={e => (query.transformer = e.detail)}
+      />
+      <Divider />
+    </div>
     <div class="viewer-controls">
       <Heading size="S">Results</Heading>
       <ButtonGroup>
@@ -220,6 +251,7 @@
     display: grid;
     grid-gap: var(--spacing-s);
   }
+
   .config-field {
     display: grid;
     grid-template-columns: 20% 1fr;
@@ -227,6 +259,11 @@
     align-items: center;
   }
 
+  .help-heading {
+    display: flex;
+    justify-content: space-between;
+  }
+
   .field {
     display: grid;
     grid-template-columns: 1fr 1fr 5%;

packages/builder/src/components/integration/codemirror.js

@@ -1,12 +1,22 @@
 import CodeMirror from "codemirror"
 import "codemirror/lib/codemirror.css"
-import "codemirror/theme/tomorrow-night-eighties.css"
+
-import "codemirror/addon/hint/show-hint.css"
+// Modes
-import "codemirror/theme/neo.css"
+import "codemirror/mode/javascript/javascript"
 import "codemirror/mode/sql/sql"
 import "codemirror/mode/css/css"
 import "codemirror/mode/handlebars/handlebars"
-import "codemirror/mode/javascript/javascript"
+
+// Hints
 import "codemirror/addon/hint/show-hint"
+import "codemirror/addon/hint/show-hint.css"
+
+// Theming
+import "codemirror/theme/tomorrow-night-eighties.css"
+
+// Functional addons
+import "codemirror/addon/selection/active-line"
+import "codemirror/addon/edit/closebrackets"
+import "codemirror/addon/edit/matchbrackets"
 
 export default CodeMirror

packages/builder/src/components/start/CreateAppModal.svelte

@@ -20,6 +20,7 @@
   import TemplateList from "./TemplateList.svelte"
 
   export let template
+  export let inline
 
   const values = writable({ name: null })
   const errors = writable({})
@@ -39,9 +40,10 @@
 
   let submitting = false
   let valid = false
+  let initialTemplateInfo = template?.fromFile || template?.key
 
   $: checkValidity($values, validator)
-  $: showTemplateSelection = !template?.fromFile && !template?.key
+  $: showTemplateSelection = !template && !initialTemplateInfo
 
   onMount(async () => {
     await hostingStore.actions.fetchDeployedApps()
@@ -64,6 +66,11 @@
   const checkValidity = async (values, validator) => {
     const obj = object().shape(validator)
     Object.keys(validator).forEach(key => ($errors[key] = null))
+    if (template?.fromFile && values.file == null) {
+      valid = false
+      return
+    }
+
     try {
       await obj.validate(values, { abortEarly: false })
     } catch (validationErrors) {
@@ -71,14 +78,17 @@
         $errors[error.path] = capitalise(error.message)
       })
     }
+
     valid = await obj.isValid(values)
   }
 
   async function createNewApp() {
+    const letTemplateToUse =
+      Object.keys(template).length === 0 ? null : template
     submitting = true
 
     // Check a template exists if we are important
-    if (template?.fromFile && !$values.file) {
+    if (letTemplateToUse?.fromFile && !$values.file) {
       $errors.file = "Please choose a file to import"
       valid = false
       submitting = false
@@ -89,10 +99,10 @@
       // Create form data to create app
       let data = new FormData()
       data.append("name", $values.name.trim())
-      data.append("useTemplate", template != null)
+      data.append("useTemplate", letTemplateToUse != null)
-      if (template) {
+      if (letTemplateToUse) {
-        data.append("templateName", template.name)
+        data.append("templateName", letTemplateToUse.name)
-        data.append("templateKey", template.key)
+        data.append("templateKey", letTemplateToUse.key)
         data.append("templateFile", $values.file)
       }
 
@@ -106,7 +116,7 @@
       analytics.captureEvent(Events.APP.CREATED, {
         name: $values.name,
         appId: appJson.instance._id,
-        template,
+        letTemplateToUse,
       })
 
       // Select Correct Application/DB in prep for creating user
@@ -144,19 +154,18 @@
     showConfirmButton={false}
     size="L"
     onConfirm={() => {
-      showTemplateSelection = false
+      template = {}
       return false
     }}
-    showCancelButton={false}
+    showCancelButton={!inline}
-    showCloseIcon={false}
+    showCloseIcon={!inline}
   >
     <TemplateList
-      onSelect={selected => {
+      onSelect={(selected, { useImport } = {}) => {
         if (!selected) {
-          showTemplateSelection = false
+          template = useImport ? { fromFile: true } : {}
           return
         }
-
         template = selected
       }}
     />
@@ -166,6 +175,9 @@
     title={template?.fromFile ? "Import app" : "Create app"}
     confirmText={template?.fromFile ? "Import app" : "Create app"}
     onConfirm={createNewApp}
+    onCancel={inline ? () => (template = null) : null}
+    cancelText={inline ? "Back" : undefined}
+    showCloseIcon={!inline}
     disabled={!valid}
   >
     {#if template?.fromFile}

packages/builder/src/components/start/TemplateList.svelte

@@ -7,7 +7,6 @@
 
   async function fetchTemplates() {
     const response = await api.get("/api/templates?type=app")
-    console.log("Responded")
     return await response.json()
   }
 
@@ -48,6 +47,19 @@
         <Heading size="XS">Start from scratch</Heading>
         <p class="detail">BLANK</p>
       </div>
+      <div
+        class="template import"
+        on:click={() => onSelect(null, { useImport: true })}
+      >
+        <div
+          class="background-icon"
+          style={`background: rgb(50, 50, 50); color: white;`}
+        >
+          <Icon name="Add" />
+        </div>
+        <Heading size="XS">Import an app</Heading>
+        <p class="detail">BLANK</p>
+      </div>
     </div>
   {:catch err}
     <h1 style="color:red">{err}</h1>
@@ -95,4 +107,8 @@
     background: var(--spectrum-global-color-gray-50);
     margin-top: 20px;
   }
+
+  .import {
+    background: var(--spectrum-global-color-gray-50);
+  }
 </style>

packages/builder/src/helpers/fetchTableData.js

@@ -48,7 +48,6 @@ export const fetchTableData = opts => {
   const fetchPage = async bookmark => {
     lastBookmark = bookmark
     const { tableId, limit, sortColumn, sortOrder, paginate } = options
-    store.update($store => ({ ...$store, loading: true }))
     const res = await API.post(`/api/${options.tableId}/search`, {
       tableId,
       query,
@@ -59,7 +58,6 @@ export const fetchTableData = opts => {
       paginate,
       bookmark,
     })
-    store.update($store => ({ ...$store, loading: false, loaded: true }))
     return await res.json()
   }
 
@@ -103,7 +101,7 @@ export const fetchTableData = opts => {
     if (!schema) {
       return
     }
-    store.update($store => ({ ...$store, schema }))
+    store.update($store => ({ ...$store, schema, loading: true }))
 
     // Work out what sort type to use
     if (!sortColumn || !schema[sortColumn]) {
@@ -135,6 +133,7 @@ export const fetchTableData = opts => {
     }
 
     // Fetch next page
+    store.update($store => ({ ...$store, loading: true }))
     const page = await fetchPage(state.bookmarks[state.pageNumber + 1])
 
     // Update state
@@ -148,6 +147,7 @@ export const fetchTableData = opts => {
         pageNumber: pageNumber + 1,
         rows: page.rows,
         bookmarks,
+        loading: false,
       }
     })
   }
@@ -160,6 +160,7 @@ export const fetchTableData = opts => {
     }
 
     // Fetch previous page
+    store.update($store => ({ ...$store, loading: true }))
     const page = await fetchPage(state.bookmarks[state.pageNumber - 1])
 
     // Update state
@@ -168,6 +169,7 @@ export const fetchTableData = opts => {
         ...$store,
         pageNumber: $store.pageNumber - 1,
         rows: page.rows,
+        loading: false,
       }
     })
   }

packages/builder/src/pages/builder/app/[application]/data/index.svelte

@@ -1,6 +1,7 @@
 <script>
   import { goto } from "@roxi/routify"
   import { onMount } from "svelte"
+  import { admin } from "stores/portal"
   import CreateDatasourceModal from "components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte"
   import { datasources } from "stores/backend"
 
@@ -10,7 +11,7 @@
     $datasources.list.length > 1
 
   onMount(() => {
-    if (!setupComplete) {
+    if (!setupComplete && !$admin.isDev) {
       modal.show()
     } else {
       $goto("./table")

packages/builder/src/pages/builder/apps/index.svelte

@@ -34,6 +34,12 @@
   const publishedAppsOnly = app => app.status === AppStatus.DEPLOYED
 
   $: publishedApps = $apps.filter(publishedAppsOnly)
+
+  $: userApps = $auth.user?.builder?.global
+    ? publishedApps
+    : publishedApps.filter(app =>
+        Object.keys($auth.user?.roles).includes(app.prodId)
+      )
 </script>
 
 {#if $auth.user && loaded}
@@ -82,11 +88,11 @@
             </Body>
           </Layout>
           <Divider />
-          {#if publishedApps.length}
+          {#if userApps.length}
             <Heading>Apps</Heading>
             <div class="group">
               <Layout gap="S" noPadding>
-                {#each publishedApps as app, idx (app.appId)}
+                {#each userApps as app, idx (app.appId)}
                   <a class="app" target="_blank" href={`/${app.prodId}`}>
                     <div class="preview" use:gradient={{ seed: app.name }} />
                     <div class="app-info">

packages/builder/src/pages/builder/portal/apps/index.svelte

@@ -112,16 +112,8 @@
 
   const exportApp = app => {
     const id = app.deployed ? app.prodId : app.devId
-    try {
+    const appName = encodeURIComponent(app.name)
-      download(
+    window.location = `/api/backups/export?appId=${id}&appname=${appName}`
-        `/api/backups/export?appId=${id}&appname=${encodeURIComponent(
-          app.name
-        )}`
-      )
-      notifications.success("App exported successfully")
-    } catch (err) {
-      notifications.error(`Error exporting app: ${err}`)
-    }
   }
 
   const unpublishApp = app => {
@@ -268,7 +260,7 @@
   {#if !enrichedApps.length && !creatingApp && loaded}
     <div class="empty-wrapper">
       <Modal inline>
-        <CreateAppModal {template} />
+        <CreateAppModal {template} inline={true} />
       </Modal>
     </div>
   {/if}

packages/builder/src/pages/builder/portal/manage/users/[userId].svelte

@@ -34,9 +34,13 @@
     role: {},
   }
 
-  $: defaultRoleId = $userFetch?.data?.builder?.global ? "ADMIN" : "BASIC"
+  const noRoleSchema = {
+    name: { displayName: "App" },
+  }
+
+  $: defaultRoleId = $userFetch?.data?.builder?.global ? "ADMIN" : ""
   // Merge the Apps list and the roles response to get something that makes sense for the table
-  $: appList = Object.keys($apps?.data).map(id => {
+  $: allAppList = Object.keys($apps?.data).map(id => {
     const roleId = $userFetch?.data?.roles?.[id] || defaultRoleId
     const role = $apps?.data?.[id].roles.find(role => role._id === roleId)
     return {
@@ -45,6 +49,15 @@
       role: [role],
     }
   })
+
+  $: appList = allAppList.filter(app => !!app.role[0])
+  $: noRoleAppList = allAppList
+    .filter(app => !app.role[0])
+    .map(app => {
+      delete app.role
+      return app
+    })
+
   let selectedApp
 
   const userFetch = fetchData(`/api/global/users/${userId}`)
@@ -173,6 +186,7 @@
   <Divider size="S" />
   <Layout gap="S" noPadding>
     <Heading size="S">Configure roles</Heading>
+    <Body>Specify a role to grant access to an app.</Body>
     <Table
       on:click={openUpdateRolesModal}
       schema={roleSchema}
@@ -183,6 +197,21 @@
       customRenderers={[{ column: "role", component: TagsRenderer }]}
     />
   </Layout>
+  <Layout gap="S" noPadding>
+    <Heading size="XS">No Access</Heading>
+    <Body
+      >Apps do not appear in the users portal. Public pages may still be viewed
+      if visited directly.</Body
+    >
+    <Table
+      on:click={openUpdateRolesModal}
+      schema={noRoleSchema}
+      data={noRoleAppList}
+      allowEditColumns={false}
+      allowEditRows={false}
+      allowSelectRows={false}
+    />
+  </Layout>
   <Divider size="S" />
   <Layout gap="XS" noPadding>
     <Heading size="S">Delete user</Heading>

packages/builder/src/pages/builder/portal/manage/users/_components/UpdateRolesModal.svelte

@@ -6,22 +6,38 @@
   export let app
   export let user
 
+  const NO_ACCESS = "NO_ACCESS"
+
   const dispatch = createEventDispatcher()
 
   const roles = app.roles
-  let options = roles
+  let options = roles.map(role => ({ value: role._id, label: role.name }))
-    .filter(role => role._id !== "PUBLIC")
+  options.push({ value: NO_ACCESS, label: "No Access" })
-    .map(role => ({ value: role._id, label: role.name }))
   let selectedRole = user?.roles?.[app?._id]
 
   async function updateUserRoles() {
-    const res = await users.save({
+    let res
-      ...user,
+    if (selectedRole === NO_ACCESS) {
-      roles: {
+      // remove the user role
-        ...user.roles,
+      const filteredRoles = { ...user.roles }
-        [app._id]: selectedRole,
+      delete filteredRoles[app?._id]
-      },
+      res = await users.save({
-    })
+        ...user,
+        roles: {
+          ...filteredRoles,
+        },
+      })
+    } else {
+      // add the user role
+      res = await users.save({
+        ...user,
+        roles: {
+          ...user.roles,
+          [app._id]: selectedRole,
+        },
+      })
+    }
+
     if (res.status === 400) {
       notifications.error("Failed to update role")
     } else {

packages/builder/src/stores/portal/admin.js

@@ -7,6 +7,7 @@ export function createAdminStore() {
     loaded: false,
     multiTenancy: false,
     cloud: false,
+    isDev: false,
     disableAccountPortal: false,
     accountPortalUrl: "",
     importComplete: false,
@@ -62,6 +63,7 @@ export function createAdminStore() {
     let cloud = false
     let disableAccountPortal = false
     let accountPortalUrl = ""
+    let isDev = false
     try {
       const response = await api.get(`/api/system/environment`)
       const json = await response.json()
@@ -69,6 +71,7 @@ export function createAdminStore() {
       cloud = json.cloud
       disableAccountPortal = json.disableAccountPortal
       accountPortalUrl = json.accountPortalUrl
+      isDev = json.isDev
     } catch (err) {
       // just let it stay disabled
     }
@@ -77,6 +80,7 @@ export function createAdminStore() {
       store.cloud = cloud
       store.disableAccountPortal = disableAccountPortal
       store.accountPortalUrl = accountPortalUrl
+      store.isDev = isDev
       return store
     })
   }

packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/cli",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "description": "Budibase CLI, for developers, self hosting and migrations.",
   "main": "src/index.js",
   "bin": {

packages/client/manifest.json

@@ -2385,7 +2385,7 @@
   },
   "dataprovider": {
     "name": "Data Provider",
-    "info": "Pagination is only available for data stored in internal tables.",
+    "info": "Pagination is only available for data stored in tables.",
     "icon": "Data",
     "illegalChildren": ["section"],
     "hasChildren": true,

packages/client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/client",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "license": "MPL-2.0",
   "module": "dist/budibase-client.js",
   "main": "dist/budibase-client.js",
@@ -19,9 +19,9 @@
     "dev:builder": "rollup -cw"
   },
   "dependencies": {
-    "@budibase/bbui": "^0.9.167-alpha.2",
+    "@budibase/bbui": "^0.9.169-alpha.10",
     "@budibase/standard-components": "^0.9.139",
-    "@budibase/string-templates": "^0.9.167-alpha.2",
+    "@budibase/string-templates": "^0.9.169-alpha.10",
     "regexparam": "^1.3.0",
     "shortid": "^2.2.15",
     "svelte-spa-router": "^3.0.5"

packages/client/src/components/ClientApp.svelte

@@ -113,6 +113,13 @@
                     />
                   {/key}
 
+                  <!--
+                    Flatpickr needs to be inside the theme wrapper.
+                    It also needs its own container because otherwise it hijacks
+                    key events on the whole page. It is painful to work with.
+                  -->
+                  <div id="flatpickr-root" />
+
                   <!-- Layers on top of app -->
                   <NotificationDisplay />
                   <ConfirmationDisplay />

packages/client/src/components/CustomThemeWrapper.svelte

@@ -16,7 +16,10 @@
     /*  Buttons */
     --spectrum-semantic-cta-color-background-default: var(--primaryColor);
     --spectrum-semantic-cta-color-background-hover: var(--primaryColorHover);
+    --spectrum-button-primary-s-border-radius: var(--buttonBorderRadius);
     --spectrum-button-primary-m-border-radius: var(--buttonBorderRadius);
+    --spectrum-button-primary-l-border-radius: var(--buttonBorderRadius);
+    --spectrum-button-primary-xl-border-radius: var(--buttonBorderRadius);
 
     /* Loading spinners */
     --spectrum-progresscircle-medium-track-fill-color: var(--primaryColor);

packages/client/src/components/app/forms/DateTimeField.svelte

@@ -12,31 +12,6 @@
 
   let fieldState
   let fieldApi
-
-  const parseDate = val => {
-    if (!val) {
-      return null
-    }
-    let date
-    if (val instanceof Date) {
-      // Use real date obj if already parsed
-      date = val
-    } else if (isNaN(val)) {
-      // Treat as date string of some sort
-      date = new Date(val)
-    } else {
-      // Treat as numerical timestamp
-      date = new Date(parseInt(val))
-    }
-    const time = date.getTime()
-    if (isNaN(time)) {
-      return null
-    }
-    // By rounding to the nearest second we avoid locking up in an endless
-    // loop in the builder, caused by potentially enriching {{ now }} to every
-    // millisecond.
-    return new Date(Math.floor(time / 1000) * 1000)
-  }
 </script>
 
 <Field
@@ -44,7 +19,7 @@
   {field}
   {disabled}
   {validation}
-  defaultValue={parseDate(defaultValue)}
+  {defaultValue}
   type="datetime"
   bind:fieldState
   bind:fieldApi
@@ -56,7 +31,7 @@
       disabled={fieldState.disabled}
       error={fieldState.error}
       id={fieldState.fieldId}
-      appendTo={document.getElementById("theme-root")}
+      appendTo={document.getElementById("flatpickr-root")}
       {enableTime}
       {placeholder}
     />

packages/client/src/components/app/forms/Field.svelte

@@ -67,7 +67,7 @@
         <Placeholder
           text="Add the Field setting to start using your component"
         />
-      {:else if fieldSchema?.type && fieldSchema?.type !== type}
+      {:else if fieldSchema?.type && fieldSchema?.type !== type && type !== "options"}
         <Placeholder
           text="This Field setting is the wrong data type for this component"
         />

packages/server/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/server",
   "email": "hi@budibase.com",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "description": "Budibase Web Server",
   "main": "src/index.js",
   "repository": {
@@ -68,13 +68,13 @@
   "author": "Budibase",
   "license": "AGPL-3.0-or-later",
   "dependencies": {
-    "@budibase/auth": "^0.9.167-alpha.2",
+    "@budibase/auth": "^0.9.169-alpha.10",
-    "@budibase/client": "^0.9.167-alpha.2",
+    "@budibase/client": "^0.9.169-alpha.10",
-    "@budibase/string-templates": "^0.9.167-alpha.2",
+    "@budibase/string-templates": "^0.9.169-alpha.10",
     "@elastic/elasticsearch": "7.10.0",
     "@koa/router": "8.0.0",
     "@sendgrid/mail": "7.1.1",
-    "@sentry/node": "5.19.2",
+    "@sentry/node": "^6.0.0",
     "airtable": "0.10.1",
     "arangojs": "7.2.0",
     "aws-sdk": "^2.767.0",
@@ -119,6 +119,7 @@
     "to-json-schema": "0.2.5",
     "uuid": "3.3.2",
     "validate.js": "0.13.1",
+    "vm2": "^3.9.3",
     "yargs": "13.2.4",
     "zlib": "1.0.5"
   },

packages/server/src/api/controllers/backup.js

@@ -1,10 +1,9 @@
-const { performBackup } = require("../../utilities/fileSystem")
+const { streamBackup } = require("../../utilities/fileSystem")
 
 exports.exportAppDump = async function (ctx) {
   const { appId } = ctx.query
-  const appname = decodeURI(ctx.query.appname)
+  const appName = decodeURI(ctx.query.appname)
-  const backupIdentifier = `${appname}Backup${new Date().getTime()}.txt`
+  const backupIdentifier = `${appName}-export-${new Date().getTime()}.txt`
-
   ctx.attachment(backupIdentifier)
-  ctx.body = await performBackup(appId, backupIdentifier)
+  ctx.body = await streamBackup(appId)
 }

packages/server/src/api/controllers/permission.js

@@ -147,6 +147,7 @@ exports.getResourcePerms = async function (ctx) {
       const rolePerms = role.permissions
       if (
         rolePerms &&
+        rolePerms[resourceId] &&
         (rolePerms[resourceId] === level ||
           rolePerms[resourceId].indexOf(level) !== -1)
       ) {

packages/server/src/api/controllers/query.js

@@ -4,6 +4,7 @@ const { generateQueryID, getQueryParams } = require("../../db/utils")
 const { integrations } = require("../../integrations")
 const { BaseQueryVerbs } = require("../../constants")
 const env = require("../../environment")
+const ScriptRunner = require("../../utilities/scriptRunner")
 
 // simple function to append "readable" to all read queries
 function enrichQueries(input) {
@@ -28,12 +29,39 @@ function formatResponse(resp) {
       resp = { response: resp }
     }
   }
-  if (!Array.isArray(resp)) {
-    resp = [resp]
-  }
   return resp
 }
 
+async function runAndTransform(
+  integration,
+  queryVerb,
+  enrichedQuery,
+  transformer
+) {
+  let rows = formatResponse(await integration[queryVerb](enrichedQuery))
+
+  // transform as required
+  if (transformer) {
+    const runner = new ScriptRunner(transformer, { data: rows })
+    rows = runner.execute()
+  }
+
+  // needs to an array for next step
+  if (!Array.isArray(rows)) {
+    rows = [rows]
+  }
+
+  // map into JSON if just raw primitive here
+  if (rows.find(row => typeof row !== "object")) {
+    rows = rows.map(value => ({ value }))
+  }
+
+  // get all the potential fields in the schema
+  let keys = rows.flatMap(Object.keys)
+
+  return { rows, keys }
+}
+
 exports.fetch = async function (ctx) {
   const db = new CouchDB(ctx.appId)
 
@@ -122,15 +150,16 @@ exports.preview = async function (ctx) {
     ctx.throw(400, "Integration type does not exist.")
   }
 
-  const { fields, parameters, queryVerb } = ctx.request.body
+  const { fields, parameters, queryVerb, transformer } = ctx.request.body
-
   const enrichedQuery = await enrichQueryFields(fields, parameters)
-
   const integration = new Integration(datasource.config)
-  const rows = formatResponse(await integration[queryVerb](enrichedQuery))
 
-  // get all the potential fields in the schema
+  const { rows, keys } = await runAndTransform(
-  const keys = rows.flatMap(Object.keys)
+    integration,
+    queryVerb,
+    enrichedQuery,
+    transformer
+  )
 
   ctx.body = {
     rows,
@@ -158,10 +187,16 @@ exports.execute = async function (ctx) {
     query.fields,
     ctx.request.body.parameters
   )
-
   const integration = new Integration(datasource.config)
+
   // call the relevant CRUD method on the integration class
-  ctx.body = formatResponse(await integration[query.queryVerb](enrichedQuery))
+  const { rows } = await runAndTransform(
+    integration,
+    query.queryVerb,
+    enrichedQuery,
+    query.transformer
+  )
+  ctx.body = rows
   // cleanup
   if (integration.end) {
     integration.end()

packages/server/src/api/controllers/row/ExternalRequest.ts

@@ -15,8 +15,9 @@ import {
 import {
   breakRowIdField,
   generateRowIdField,
+  isRowId,
+  convertRowId,
 } from "../../../integrations/utils"
-import { RelationshipTypes } from "../../../constants"
 
 interface ManyRelationship {
   tableId?: string
@@ -36,7 +37,7 @@ interface RunConfig {
 
 module External {
   const { makeExternalQuery } = require("./utils")
-  const { DataSourceOperation, FieldTypes } = require("../../../constants")
+  const { DataSourceOperation, FieldTypes, RelationshipTypes } = require("../../../constants")
   const { breakExternalTableId, isSQL } = require("../../../integrations/utils")
   const { processObjectSync } = require("@budibase/string-templates")
   const { cloneDeep } = require("lodash/fp")
@@ -83,6 +84,48 @@ module External {
     }
   }
 
+  /**
+   * This function checks the incoming parameters to make sure all the inputs are
+   * valid based on on the table schema. The main thing this is looking for is when a
+   * user has made use of the _id field of a row for a foreign key or a search parameter.
+   * In these cases the key will be sent up as [1], rather than 1. In these cases we will
+   * simplify it down to the requirements. This function is quite complex as we try to be
+   * relatively restrictive over what types of columns we will perform this action for.
+   */
+  function cleanupConfig(config: RunConfig, table: Table): RunConfig {
+    const primaryOptions = [
+      FieldTypes.STRING,
+      FieldTypes.LONGFORM,
+      FieldTypes.OPTIONS,
+      FieldTypes.NUMBER,
+    ]
+    // filter out fields which cannot be keys
+    const fieldNames = Object.entries(table.schema)
+      .filter(schema => primaryOptions.find(val => val === schema[1].type))
+      .map(([fieldName]) => fieldName)
+    const iterateObject = (obj: { [key: string]: any }) => {
+      for (let [field, value] of Object.entries(obj)) {
+        if (fieldNames.find(name => name === field) && isRowId(value)) {
+          obj[field] = convertRowId(value)
+        }
+      }
+    }
+    // check the row and filters to make sure they aren't a key of some sort
+    if (config.filters) {
+      for (let filter of Object.values(config.filters)) {
+        if (typeof filter !== "object" || Object.keys(filter).length === 0) {
+          continue
+        }
+        iterateObject(filter)
+      }
+    }
+    if (config.row) {
+      iterateObject(config.row)
+    }
+
+    return config
+  }
+
   function generateIdForRow(row: Row | undefined, table: Table): string {
     const primary = table.primary
     if (!row || !primary) {
@@ -509,7 +552,7 @@ module External {
       return fields
     }
 
-    async run({ id, row, filters, sort, paginate }: RunConfig) {
+    async run(config: RunConfig) {
       const { appId, operation, tableId } = this
       let { datasourceId, tableName } = breakExternalTableId(tableId)
       if (!this.datasource) {
@@ -525,9 +568,11 @@ module External {
       if (!table) {
         throw `Unable to process query, table "${tableName}" not defined.`
       }
-      // clean up row on ingress using schema
+      // look for specific components of config which may not be considered acceptable
+      let { id, row, filters, sort, paginate } = cleanupConfig(config, table)
       filters = buildFilters(id, filters || {}, table)
       const relationships = this.buildRelationships(table)
+      // clean up row on ingress using schema
       const processed = this.inputProcessing(row, table)
       row = processed.row
       if (

packages/server/src/api/controllers/script.js

@@ -1,24 +1,9 @@
-const fetch = require("node-fetch")
+const ScriptRunner = require("../../utilities/scriptRunner")
-const vm = require("vm")
-
-class ScriptExecutor {
-  constructor(body) {
-    const code = `let fn = () => {\n${body.script}\n}; out = fn();`
-    this.script = new vm.Script(code)
-    this.context = vm.createContext(body.context)
-    this.context.fetch = fetch
-  }
-
-  execute() {
-    this.script.runInContext(this.context)
-    return this.context.out
-  }
-}
 
 exports.execute = async function (ctx) {
-  const executor = new ScriptExecutor(ctx.request.body)
+  const { script, context } = ctx.request.body
-
+  const runner = new ScriptRunner(script, context)
-  ctx.body = executor.execute()
+  ctx.body = runner.execute()
 }
 
 exports.save = async function (ctx) {

packages/server/src/api/controllers/table/index.js

@@ -9,11 +9,7 @@ const {
   BudibaseInternalDB,
 } = require("../../../db/utils")
 const { FieldTypes } = require("../../../constants")
-const { TableSaveFunctions, getExternalTable } = require("./utils")
+const { TableSaveFunctions, getTable } = require("./utils")
-const {
-  isExternalTable,
-  breakExternalTableId,
-} = require("../../../integrations/utils")
 
 exports.fetch = async function (ctx) {
   const db = new CouchDB(ctx.appId)
@@ -48,14 +44,8 @@ exports.fetch = async function (ctx) {
 }
 
 exports.find = async function (ctx) {
-  const db = new CouchDB(ctx.appId)
   const tableId = ctx.params.id
-  if (isExternalTable(tableId)) {
+  ctx.body = await getTable(ctx.appId, tableId)
-    let { datasourceId, tableName } = breakExternalTableId(tableId)
-    ctx.body = await getExternalTable(ctx.appId, datasourceId, tableName)
-  } else {
-    ctx.body = await db.get(ctx.params.id)
-  }
 }
 
 exports.save = async function (ctx) {
@@ -91,6 +81,9 @@ exports.save = async function (ctx) {
     for (let propKey of Object.keys(tableToSave.schema)) {
       let column = tableToSave.schema[propKey]
       let oldColumn = oldTable.schema[propKey]
+      if (oldColumn && oldColumn.type === "internal") {
+        oldColumn.type = "auto"
+      }
       if (oldColumn && oldColumn.type !== column.type) {
         ctx.throw(400, "Cannot change the type of a column")
       }

packages/server/src/api/controllers/table/utils.js

@@ -1,5 +1,4 @@
 const CouchDB = require("../../../db")
-const linkRows = require("../../../db/linkedRows")
 const csvParser = require("../../../utilities/csvParser")
 const {
   getRowParams,
@@ -10,6 +9,12 @@ const { isEqual } = require("lodash/fp")
 const { AutoFieldSubTypes, FieldTypes } = require("../../../constants")
 const { inputProcessing } = require("../../../utilities/rowProcessor")
 const { USERS_TABLE_SCHEMA } = require("../../../constants")
+const {
+  isExternalTable,
+  breakExternalTableId,
+} = require("../../../integrations/utils")
+const { getViews, saveView } = require("../view/utils")
+const viewTemplate = require("../view/viewBuilder")
 
 exports.checkForColumnUpdates = async (db, oldTable, updatedTable) => {
   let updatedRows = []
@@ -22,6 +27,7 @@ exports.checkForColumnUpdates = async (db, oldTable, updatedTable) => {
   }
   // check for renaming of columns or deleted columns
   if (rename || deletedColumns.length !== 0) {
+    // Update all rows
     const rows = await db.allDocs(
       getRowParams(updatedTable._id, null, {
         include_docs: true,
@@ -36,6 +42,9 @@ exports.checkForColumnUpdates = async (db, oldTable, updatedTable) => {
       }
       return doc
     })
+
+    // Update views
+    await exports.checkForViewUpdates(db, updatedTable, rename, deletedColumns)
     delete updatedTable._rename
   }
   return { rows: updatedRows, table: updatedTable }
@@ -93,19 +102,10 @@ exports.handleDataImport = async (appId, user, table, dataImport) => {
         }
       }
 
-      // make sure link rows are up to date
+      finalData.push(row)
-      finalData.push(
-        linkRows.updateLinks({
-          appId,
-          eventType: linkRows.EventType.ROW_SAVE,
-          row,
-          tableId: row.tableId,
-          table,
-        })
-      )
     }
 
-    await db.bulkDocs(await Promise.all(finalData))
+    await db.bulkDocs(finalData)
     let response = await db.put(table)
     table._rev = response._rev
   }
@@ -233,4 +233,86 @@ exports.getExternalTable = async (appId, datasourceId, tableName) => {
   return entities[tableName]
 }
 
+exports.getTable = async (appId, tableId) => {
+  const db = new CouchDB(appId)
+  if (isExternalTable(tableId)) {
+    let { datasourceId, tableName } = breakExternalTableId(tableId)
+    return exports.getExternalTable(appId, datasourceId, tableName)
+  } else {
+    return db.get(tableId)
+  }
+}
+
+exports.checkForViewUpdates = async (db, table, rename, deletedColumns) => {
+  const views = await getViews(db)
+  const tableViews = views.filter(view => view.meta.tableId === table._id)
+
+  // Check each table view to see if impacted by this table action
+  for (let view of tableViews) {
+    let needsUpdated = false
+
+    // First check for renames, otherwise check for deletions
+    if (rename) {
+      // Update calculation field if required
+      if (view.meta.field === rename.old) {
+        view.meta.field = rename.updated
+        needsUpdated = true
+      }
+
+      // Update group by field if required
+      if (view.meta.groupBy === rename.old) {
+        view.meta.groupBy = rename.updated
+        needsUpdated = true
+      }
+
+      // Update filters if required
+      if (view.meta.filters) {
+        view.meta.filters.forEach(filter => {
+          if (filter.key === rename.old) {
+            filter.key = rename.updated
+            needsUpdated = true
+          }
+        })
+      }
+    } else if (deletedColumns) {
+      deletedColumns.forEach(column => {
+        // Remove calculation statement if required
+        if (view.meta.field === column) {
+          delete view.meta.field
+          delete view.meta.calculation
+          delete view.meta.groupBy
+          needsUpdated = true
+        }
+
+        // Remove group by field if required
+        if (view.meta.groupBy === column) {
+          delete view.meta.groupBy
+          needsUpdated = true
+        }
+
+        // Remove filters referencing deleted field if required
+        if (view.meta.filters && view.meta.filters.length) {
+          const initialLength = view.meta.filters.length
+          view.meta.filters = view.meta.filters.filter(filter => {
+            return filter.key !== column
+          })
+          if (initialLength !== view.meta.filters.length) {
+            needsUpdated = true
+          }
+        }
+      })
+    }
+
+    // Update view if required
+    if (needsUpdated) {
+      const newViewTemplate = viewTemplate(view.meta)
+      await saveView(db, null, view.name, newViewTemplate)
+      if (!newViewTemplate.meta.schema) {
+        newViewTemplate.meta.schema = table.schema
+      }
+      table.views[view.name] = newViewTemplate.meta
+    }
+  }
+}
+
 exports.TableSaveFunctions = TableSaveFunctions

packages/server/src/api/routes/query.js

@@ -31,7 +31,8 @@ function generateQueryValidation() {
     })),
     queryVerb: Joi.string().allow().required(),
     extra: Joi.object().optional(),
-    schema: Joi.object({}).required().unknown(true)
+    schema: Joi.object({}).required().unknown(true),
+    transformer: Joi.string().optional(),
   }))
 }
 
@@ -42,6 +43,7 @@ function generateQueryPreviewValidation() {
     queryVerb: Joi.string().allow().required(),
     extra: Joi.object().optional(),
     datasourceId: Joi.string().required(),
+    transformer: Joi.string().optional(),
     parameters: Joi.object({}).required().unknown(true)
   }))
 }

packages/server/src/automations/actions.js

@@ -3,7 +3,6 @@ const createRow = require("./steps/createRow")
 const updateRow = require("./steps/updateRow")
 const deleteRow = require("./steps/deleteRow")
 const executeScript = require("./steps/executeScript")
-const bash = require("./steps/bash")
 const executeQuery = require("./steps/executeQuery")
 const outgoingWebhook = require("./steps/outgoingWebhook")
 const serverLog = require("./steps/serverLog")
@@ -14,6 +13,7 @@ const integromat = require("./steps/integromat")
 let filter = require("./steps/filter")
 let delay = require("./steps/delay")
 let queryRow = require("./steps/queryRows")
+const env = require("../environment")
 
 const ACTION_IMPLS = {
   SEND_EMAIL_SMTP: sendSmtpEmail.run,
@@ -22,7 +22,6 @@ const ACTION_IMPLS = {
   DELETE_ROW: deleteRow.run,
   OUTGOING_WEBHOOK: outgoingWebhook.run,
   EXECUTE_SCRIPT: executeScript.run,
-  EXECUTE_BASH: bash.run,
   EXECUTE_QUERY: executeQuery.run,
   SERVER_LOG: serverLog.run,
   DELAY: delay.run,
@@ -42,7 +41,6 @@ const ACTION_DEFINITIONS = {
   OUTGOING_WEBHOOK: outgoingWebhook.definition,
   EXECUTE_SCRIPT: executeScript.definition,
   EXECUTE_QUERY: executeQuery.definition,
-  EXECUTE_BASH: bash.definition,
   SERVER_LOG: serverLog.definition,
   DELAY: delay.definition,
   FILTER: filter.definition,
@@ -54,6 +52,15 @@ const ACTION_DEFINITIONS = {
   integromat: integromat.definition,
 }
 
+// don't add the bash script/definitions unless in self host
+// the fact this isn't included in any definitions means it cannot be
+// ran at all
+if (env.SELF_HOSTED) {
+  const bash = require("./steps/bash")
+  ACTION_IMPLS["EXECUTE_BASH"] = bash.run
+  ACTION_DEFINITIONS["EXECUTE_BASH"] = bash.definition
+}
+
 /* istanbul ignore next */
 exports.getAction = async function (actionName) {
   if (ACTION_IMPLS[actionName] != null) {

packages/server/src/automations/automationUtils.js

@@ -1,9 +1,4 @@
-const CouchDB = require("../db")
+const { getTable } = require("../api/controllers/table/utils")
-const {
-  isExternalTable,
-  breakExternalTableId,
-} = require("../integrations/utils")
-const { getExternalTable } = require("../api/controllers/table/utils")
 
 /**
  * When values are input to the system generally they will be of type string as this is required for template strings.
@@ -21,7 +16,7 @@ const { getExternalTable } = require("../api/controllers/table/utils")
  * @returns {object} The inputs object which has had all the various types supported by this function converted to their
  * primitive types.
  */
-module.exports.cleanInputValues = (inputs, schema) => {
+exports.cleanInputValues = (inputs, schema) => {
   if (schema == null) {
     return inputs
   }
@@ -63,30 +58,11 @@ module.exports.cleanInputValues = (inputs, schema) => {
  * @param {object} row The input row structure which requires clean-up after having been through template statements.
  * @returns {Promise<Object>} The cleaned up rows object, will should now have all the required primitive types.
  */
-module.exports.cleanUpRow = async (appId, tableId, row) => {
+exports.cleanUpRow = async (appId, tableId, row) => {
-  const db = new CouchDB(appId)
+  let table = await getTable(appId, tableId)
-  let table
+  return exports.cleanInputValues(row, { properties: table.schema })
-  if (isExternalTable(tableId)) {
-    const { datasourceId, tableName } = breakExternalTableId(tableId)
-    table = await getExternalTable(appId, datasourceId, tableName)
-  } else {
-    table = await db.get(tableId)
-  }
-
-  return module.exports.cleanInputValues(row, { properties: table.schema })
 }
 
-/**
+exports.getError = err => {
- * A utility function for the cleanUpRow, which can be used if only the row ID is known (not the table ID) to clean
+  return typeof err !== "string" ? err.toString() : err
- * up a row after template statements have been replaced. This is specifically useful for the update row action.
- *
- * @param {string} appId The instance which the Table/Table is contained under.
- * @param {string} rowId The ID of the row from which the tableId will be extracted, to get the Table/Table schema.
- * @param {object} row The input row structure which requires clean-up after having been through template statements.
- * @returns {Promise<Object>} The cleaned up rows object, which will now have all the required primitive types.
- */
-module.exports.cleanUpRowById = async (appId, rowId, row) => {
-  const db = new CouchDB(appId)
-  const foundRow = await db.get(rowId)
-  return module.exports.cleanUpRow(appId, foundRow.tableId, row)
 }

packages/server/src/automations/steps/bash.js

@@ -1,5 +1,6 @@
 const { execSync } = require("child_process")
 const { processStringSync } = require("@budibase/string-templates")
+const automationUtils = require("../automationUtils")
 
 exports.definition = {
   name: "Bash Scripting",
@@ -63,7 +64,7 @@ exports.run = async function ({ inputs, context }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/createRow.js

@@ -97,7 +97,7 @@ exports.run = async function ({ inputs, appId, emitter }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/deleteRow.js

@@ -2,6 +2,7 @@ const rowController = require("../../api/controllers/row")
 const env = require("../../environment")
 const usage = require("../../utilities/usageQuota")
 const { buildCtx } = require("./utils")
+const automationUtils = require("../automationUtils")
 
 exports.definition = {
   description: "Delete a row from your database",
@@ -85,7 +86,7 @@ exports.run = async function ({ inputs, appId, emitter }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/executeQuery.js

@@ -1,5 +1,6 @@
 const queryController = require("../../api/controllers/query")
 const { buildCtx } = require("./utils")
+const automationUtils = require("../automationUtils")
 
 exports.definition = {
   name: "External Data Connector",
@@ -74,7 +75,7 @@ exports.run = async function ({ inputs, appId, emitter }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/executeScript.js

@@ -1,5 +1,6 @@
 const scriptController = require("../../api/controllers/script")
 const { buildCtx } = require("./utils")
+const automationUtils = require("../automationUtils")
 
 exports.definition = {
   name: "JS Scripting",
@@ -63,7 +64,7 @@ exports.run = async function ({ inputs, appId, context, emitter }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/outgoingWebhook.js

@@ -1,5 +1,6 @@
 const fetch = require("node-fetch")
 const { getFetchResponse } = require("./utils")
+const automationUtils = require("../automationUtils")
 
 const RequestType = {
   POST: "POST",
@@ -127,7 +128,7 @@ exports.run = async function ({ inputs }) {
     /* istanbul ignore next */
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/queryRows.js

@@ -2,6 +2,7 @@ const rowController = require("../../api/controllers/row")
 const tableController = require("../../api/controllers/table")
 const { FieldTypes } = require("../../constants")
 const { buildCtx } = require("./utils")
+const automationUtils = require("../automationUtils")
 
 const SortOrders = {
   ASCENDING: "ascending",
@@ -110,7 +111,7 @@ exports.run = async function ({ inputs, appId }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/sendSmtpEmail.js

@@ -1,4 +1,5 @@
 const { sendSmtpEmail } = require("../../utilities/workerRequests")
+const automationUtils = require("../automationUtils")
 
 exports.definition = {
   description: "Send an email using SMTP",
@@ -61,7 +62,7 @@ exports.run = async function ({ inputs }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/steps/updateRow.js

@@ -64,6 +64,7 @@ exports.run = async function ({ inputs, appId, emitter }) {
       },
     }
   }
+  const tableId = inputs.row.tableId
 
   // clear any falsy properties so that they aren't updated
   for (let propKey of Object.keys(inputs.row)) {
@@ -80,15 +81,14 @@ exports.run = async function ({ inputs, appId, emitter }) {
     },
     params: {
       rowId: inputs.rowId,
+      tableId: tableId,
     },
   })
 
   try {
-    inputs.row = await automationUtils.cleanUpRowById(
+    if (tableId) {
-      appId,
+      inputs.row = await automationUtils.cleanUpRow(appId, tableId, inputs.row)
-      inputs.rowId,
+    }
-      inputs.row
-    )
     await rowController.patch(ctx)
     return {
       row: ctx.body,
@@ -100,7 +100,7 @@ exports.run = async function ({ inputs, appId, emitter }) {
   } catch (err) {
     return {
       success: false,
-      response: err,
+      response: automationUtils.getError(err),
     }
   }
 }

packages/server/src/automations/thread.js

@@ -10,6 +10,7 @@ const env = require("../environment")
 const usage = require("../utilities/usageQuota")
 
 const FILTER_STEP_ID = actions.ACTION_DEFINITIONS.FILTER.stepId
+const STOPPED_STATUS = { success: false, status: "STOPPED" }
 
 /**
  * The automation orchestrator is a class responsible for executing automations.
@@ -68,7 +69,13 @@ class Orchestrator {
   async execute() {
     let automation = this._automation
     const app = await this.getApp()
+    let stopped = false
     for (let step of automation.definition.steps) {
+      // execution stopped, record state for that
+      if (stopped) {
+        this.updateExecutionOutput(step.id, step.stepId, {}, STOPPED_STATUS)
+        continue
+      }
       let stepFn = await this.getStepFunctionality(step.stepId)
       step.inputs = await processObject(step.inputs, this._context)
       step.inputs = automationUtils.cleanInputValues(
@@ -86,10 +93,17 @@ class Orchestrator {
             context: this._context,
           })
         })
-        if (step.stepId === FILTER_STEP_ID && !outputs.success) {
-          break
-        }
         this._context.steps.push(outputs)
+        // if filter causes us to stop execution don't break the loop, set a var
+        // so that we can finish iterating through the steps and record that it stopped
+        if (step.stepId === FILTER_STEP_ID && !outputs.success) {
+          stopped = true
+          this.updateExecutionOutput(step.id, step.stepId, step.inputs, {
+            ...outputs,
+            ...STOPPED_STATUS,
+          })
+          continue
+        }
         this.updateExecutionOutput(step.id, step.stepId, step.inputs, outputs)
       } catch (err) {
         console.error(`Automation error - ${step.stepId} - ${err}`)
@@ -99,7 +113,7 @@ class Orchestrator {
 
     // Increment quota for automation runs
     if (!env.SELF_HOSTED && !isDevAppID(this._appId)) {
-      usage.update(usage.Properties.AUTOMATION, 1)
+      await usage.update(usage.Properties.AUTOMATION, 1)
     }
     return this.executionOutput
   }

packages/server/src/constants/index.js

@@ -42,6 +42,7 @@ exports.FieldTypes = {
   FORMULA: "formula",
   AUTO: "auto",
   JSON: "json",
+  INTERNAL: "internal",
 }
 
 exports.RelationshipTypes = {

packages/server/src/integrations/base/sql.ts

@@ -98,7 +98,9 @@ function addFilters(
 }
 
 function addRelationships(
+  knex: Knex,
   query: KnexQuery,
+  fields: string | string[],
   fromTable: string,
   relationships: RelationshipsJson[] | undefined
 ): KnexQuery {
@@ -114,7 +116,7 @@ function addRelationships(
       query = query.leftJoin(
         toTable,
         `${fromTable}.${from}`,
-        `${relationship.tableName}.${to}`
+        `${toTable}.${to}`
       )
     } else {
       const throughTable = relationship.through
@@ -130,7 +132,7 @@ function addRelationships(
         .leftJoin(toTable, `${toTable}.${toPrimary}`, `${throughTable}.${to}`)
     }
   }
-  return query
+  return query.limit(BASE_LIMIT)
 }
 
 function buildCreate(
@@ -199,7 +201,7 @@ function buildRead(knex: Knex, json: QueryJson, limit: number): KnexQuery {
     [tableName]: query,
   }).select(selectStatement)
   // handle joins
-  return addRelationships(preQuery, tableName, relationships)
+  return addRelationships(knex, preQuery, selectStatement, tableName, relationships)
 }
 
 function buildUpdate(

packages/server/src/integrations/tests/mongo.spec.js

@@ -27,7 +27,7 @@ describe("MongoDB Integration", () => {
     const body = {
       name: "Hello"
     }
-    const response = await config.integration.create({ 
+    await config.integration.create({
       index: indexName,
       json: body,
       extra: { collection: 'testCollection', actionTypes: 'insertOne'}
@@ -54,7 +54,7 @@ describe("MongoDB Integration", () => {
       },
       extra: { collection: 'testCollection', actionTypes: 'deleteOne'}
     }
-    const response = await config.integration.delete(query)
+    await config.integration.delete(query)
     expect(config.integration.client.deleteOne).toHaveBeenCalledWith(query.json)
   })
 
@@ -65,7 +65,7 @@ describe("MongoDB Integration", () => {
       },
       extra: { collection: 'testCollection', actionTypes: 'updateOne'}
     }
-    const response = await config.integration.update(query)
+    await config.integration.update(query)
     expect(config.integration.client.updateOne).toHaveBeenCalledWith(query.json)
   })
 
@@ -75,10 +75,14 @@ describe("MongoDB Integration", () => {
     const query = {
       extra: { collection: 'testCollection', actionTypes: 'deleteOne'}
     }
-    // Weird, need to do an IIFE for jest to recognize that it throws
-    expect(() => config.integration.read(query)()).toThrow(expect.any(Object))
 
+    let error = null
+    try {
+      await config.integration.read(query)
+    } catch (err) {
+      error = err
+    }
+    expect(error).toBeDefined()
     restore()
   })
-
 })

packages/server/src/integrations/utils.ts

@@ -5,6 +5,7 @@ const { DocumentTypes, SEPARATOR } = require("../db/utils")
 const { FieldTypes } = require("../constants")
 
 const DOUBLE_SEPARATOR = `${SEPARATOR}${SEPARATOR}`
+const ROW_ID_REGEX = /^\[.*]$/g
 
 export function isExternalTable(tableId: string) {
   return tableId.includes(DocumentTypes.DATASOURCE)
@@ -32,6 +33,20 @@ export function generateRowIdField(keyProps: any[] = []) {
   return encodeURIComponent(JSON.stringify(keyProps).replace(/"/g, "'"))
 }
 
+export function isRowId(field: any) {
+  return Array.isArray(field) || (typeof field === "string" && field.match(ROW_ID_REGEX) != null)
+}
+
+export function convertRowId(field: any) {
+  if (Array.isArray(field)) {
+    return field[0]
+  }
+  if (typeof field === "string" && field.match(ROW_ID_REGEX) != null) {
+    return field.substring(1, field.length - 1)
+  }
+  return field
+}
+
 // should always return an array
 export function breakRowIdField(_id: string | { _id: string }): any[] {
   if (!_id) {

packages/server/src/middleware/currentapp.js

@@ -45,7 +45,7 @@ module.exports = async (ctx, next) => {
     const globalUser = await getCachedSelf(ctx, requestAppId)
     appId = requestAppId
     // retrieving global user gets the right role
-    roleId = globalUser.roleId || BUILTIN_ROLE_IDS.BASIC
+    roleId = globalUser.roleId || roleId
   }
 
   // nothing more to do

packages/server/src/middleware/tests/currentapp.spec.js

@@ -127,8 +127,8 @@ describe("Current app middleware", () => {
       } else {
         expect(cookieFn).not.toHaveBeenCalled()
       }
-      expect(config.ctx.roleId).toEqual("BASIC")
+      expect(config.ctx.roleId).toEqual("PUBLIC")
-      expect(config.ctx.user.role._id).toEqual("BASIC")
+      expect(config.ctx.user.role._id).toEqual("PUBLIC")
       expect(config.ctx.appId).toEqual("app_test")
       expect(config.next).toHaveBeenCalled()
     }
@@ -163,7 +163,7 @@ describe("Current app middleware", () => {
             return "app_test"
           },
           setCookie: jest.fn(),
-          getCookie: () => ({appId: "app_test", roleId: "BASIC"}),
+          getCookie: () => ({appId: "app_test", roleId: "PUBLIC"}),
         },
         constants: { Cookies: {} },
       }))

packages/server/src/utilities/fileSystem/index.js

@@ -106,64 +106,86 @@ exports.apiFileReturn = contents => {
 }
 
 /**
- * Takes a copy of the database state for an app to the object store.
+ * Local utility to back up the database state for an app, excluding global user
- * @param {string} appId The ID of the app which is to be backed up.
+ * data or user relationships.
- * @param {string} backupName The name of the backup located in the object store.
+ * @param {string} appId The app to backup
- * @return The backup has been completed when this promise completes and returns a file stream
+ * @param {object} config Config to send to export DB
- * to the temporary backup file (to return via API if required).
+ * @returns {*} either a string or a stream of the backup
  */
-exports.performBackup = async (appId, backupName) => {
+const backupAppData = async (appId, config) => {
-  return exports.exportDB(appId, {
+  return await exports.exportDB(appId, {
-    exportName: backupName,
+    ...config,
     filter: doc =>
       !(
         doc._id.includes(USER_METDATA_PREFIX) ||
-        doc.includes(LINK_USER_METADATA_PREFIX)
+        doc._id.includes(LINK_USER_METADATA_PREFIX)
       ),
   })
 }
 
 /**
- * exports a DB to either file or a variable (memory).
+ * Takes a copy of the database state for an app to the object store.
- * @param {string} dbName the DB which is to be exported.
+ * @param {string} appId The ID of the app which is to be backed up.
- * @param {string} exportName optional - the file name to export to, if not in memory.
+ * @param {string} backupName The name of the backup located in the object store.
- * @param {function} filter optional - a filter function to clear out any un-wanted docs.
+ * @return {*} a readable stream to the completed backup file
- * @return Either the file stream or the variable (if no export name provided).
  */
-exports.exportDB = async (
+exports.performBackup = async (appId, backupName) => {
-  dbName,
+  return await backupAppData(appId, { exportName: backupName })
-  { exportName, filter } = { exportName: undefined, filter: undefined }
+}
-) => {
+
-  let stream,
+/**
-    appString = "",
+ * Streams a backup of the database state for an app
-    path = null
+ * @param {string} appId The ID of the app which is to be backed up.
-  if (exportName) {
+ * @returns {*} a readable stream of the backup which is written in real time
-    path = join(budibaseTempDir(), exportName)
+ */
-    stream = fs.createWriteStream(path)
+exports.streamBackup = async appId => {
-  } else {
+  return await backupAppData(appId, { stream: true })
-    stream = new MemoryStream()
+}
-    stream.on("data", chunk => {
+
-      appString += chunk.toString()
+/**
-    })
+ * Exports a DB to either file or a variable (memory).
-  }
+ * @param {string} dbName the DB which is to be exported.
-  // perform couch dump
+ * @param {string} exportName optional - provide a filename to write the backup to a file
+ * @param {boolean} stream optional - whether to perform a full backup
+ * @param {function} filter optional - a filter function to clear out any un-wanted docs.
+ * @return {*} either a readable stream or a string
+ */
+exports.exportDB = async (dbName, { stream, filter, exportName } = {}) => {
   const instanceDb = new CouchDB(dbName)
-  await instanceDb.dump(stream, {
+
-    filter,
+  // Stream the dump if required
+  if (stream) {
+    const memStream = new MemoryStream()
+    instanceDb.dump(memStream, { filter })
+    return memStream
+  }
+
+  // Write the dump to file if required
+  if (exportName) {
+    const path = join(budibaseTempDir(), exportName)
+    const writeStream = fs.createWriteStream(path)
+    await instanceDb.dump(writeStream, { filter })
+
+    // Upload the dump to the object store if self hosted
+    if (env.SELF_HOSTED) {
+      await streamUpload(
+        ObjectStoreBuckets.BACKUPS,
+        join(dbName, exportName),
+        fs.createReadStream(path)
+      )
+    }
+
+    return fs.createReadStream(path)
+  }
+
+  // Stringify the dump in memory if required
+  const memStream = new MemoryStream()
+  let appString = ""
+  memStream.on("data", chunk => {
+    appString += chunk.toString()
   })
-  // just in memory, return the final string
+  await instanceDb.dump(memStream, { filter })
-  if (!exportName) {
+  return appString
-    return appString
-  }
-  // write the file to the object store
-  if (env.SELF_HOSTED) {
-    await streamUpload(
-      ObjectStoreBuckets.BACKUPS,
-      join(dbName, exportName),
-      fs.createReadStream(path)
-    )
-  }
-  return fs.createReadStream(path)
 }
 
 /**

packages/server/src/utilities/global.js

@@ -26,7 +26,7 @@ exports.updateAppRole = (appId, user) => {
   if (!user.roleId && user.builder && user.builder.global) {
     user.roleId = BUILTIN_ROLE_IDS.ADMIN
   } else if (!user.roleId) {
-    user.roleId = BUILTIN_ROLE_IDS.BASIC
+    user.roleId = BUILTIN_ROLE_IDS.PUBLIC
   }
   delete user.roles
   return user

packages/server/src/utilities/rowProcessor/index.js

@@ -200,6 +200,12 @@ exports.inputProcessing = (
       clonedRow[key] = exports.coerce(value, field.type)
     }
   }
+
+  if (!clonedRow._id || !clonedRow._rev) {
+    clonedRow._id = row._id
+    clonedRow._rev = row._rev
+  }
+
   // handle auto columns - this returns an object like {table, row}
   return processAutoColumn(user, copiedTable, clonedRow, opts)
 }

packages/server/src/utilities/scriptRunner.js

@@ -0,0 +1,21 @@
+const fetch = require("node-fetch")
+const { VM, VMScript } = require("vm2")
+
+class ScriptRunner {
+  constructor(script, context) {
+    const code = `let fn = () => {\n${script}\n}; results.out = fn();`
+    this.vm = new VM()
+    this.results = { out: "" }
+    this.vm.setGlobals(context)
+    this.vm.setGlobal("fetch", fetch)
+    this.vm.setGlobal("results", this.results)
+    this.script = new VMScript(code)
+  }
+
+  execute() {
+    this.vm.run(this.script)
+    return this.results.out
+  }
+}
+
+module.exports = ScriptRunner

packages/server/yarn.lock

@@ -1826,83 +1826,72 @@
     "@sendgrid/client" "^7.1.1"
     "@sendgrid/helpers" "^7.0.1"
 
-"@sentry/apm@5.19.2":
+"@sentry/core@6.13.3":
-  version "5.19.2"
+  version "6.13.3"
-  resolved "https://registry.yarnpkg.com/@sentry/apm/-/apm-5.19.2.tgz#369fdcbc9fa5db992f707b24f3165e106a277cf7"
+  resolved "https://registry.yarnpkg.com/@sentry/core/-/core-6.13.3.tgz#5cbbb995128e793ebebcbf1d3b7514e0e5e8b221"
-  integrity sha512-V7p5niqG/Nn1OSMAyreChiIrQFYzFHKADKNaDEvIXqC4hxFnMG8lPRqEFJH49fNjsFBFfIG9iY1rO1ZFg3S42Q==
+  integrity sha512-obm3SjgCk8A7nB37b2AU1eq1q7gMoJRrGMv9VRIyfcG0Wlz/5lJ9O3ohUk+YZaaVfZMxXn6hFtsBiOWmlv7IIA==
   dependencies:
-    "@sentry/browser" "5.19.2"
+    "@sentry/hub" "6.13.3"
-    "@sentry/hub" "5.19.2"
+    "@sentry/minimal" "6.13.3"
-    "@sentry/minimal" "5.19.2"
+    "@sentry/types" "6.13.3"
-    "@sentry/types" "5.19.2"
+    "@sentry/utils" "6.13.3"
-    "@sentry/utils" "5.19.2"
     tslib "^1.9.3"
 
-"@sentry/browser@5.19.2":
+"@sentry/hub@6.13.3":
-  version "5.19.2"
+  version "6.13.3"
-  resolved "https://registry.yarnpkg.com/@sentry/browser/-/browser-5.19.2.tgz#8bad445b8d1efd50e6510bb43b3018b941f6e5cb"
+  resolved "https://registry.yarnpkg.com/@sentry/hub/-/hub-6.13.3.tgz#cc09623a69b5343315fdb61c7fdd0be74b72299f"
-  integrity sha512-o6Z532n+0N5ANDzgR9GN+Q6CU7zVlIJvBEW234rBiB+ZZj6XwTLS1dD+JexGr8lCo8PeXI2rypKcj1jUGLVW8w==
+  integrity sha512-eYppBVqvhs5cvm33snW2sxfcw6G20/74RbBn+E4WDo15hozis89kU7ZCJDOPkXuag3v1h9igns/kM6PNBb41dw==
   dependencies:
-    "@sentry/core" "5.19.2"
+    "@sentry/types" "6.13.3"
-    "@sentry/types" "5.19.2"
+    "@sentry/utils" "6.13.3"
-    "@sentry/utils" "5.19.2"
     tslib "^1.9.3"
 
-"@sentry/core@5.19.2":
+"@sentry/minimal@6.13.3":
-  version "5.19.2"
+  version "6.13.3"
-  resolved "https://registry.yarnpkg.com/@sentry/core/-/core-5.19.2.tgz#99a64ef0e55230fc02a083c48fa07ada85de4929"
+  resolved "https://registry.yarnpkg.com/@sentry/minimal/-/minimal-6.13.3.tgz#a675a79bcc830142e4f95e6198a2efde2cd3901e"
-  integrity sha512-sfbBsVXpA0WYJUichz5IhvqKD8xJUfQvsszrTsUKa7PQAMAboOmuh6bo8KquaVQnAZyZWZU08UduvlSV3tA7tw==
+  integrity sha512-63MlYYRni3fs5Bh8XBAfVZ+ctDdWg0fapSTP1ydIC37fKvbE+5zhyUqwrEKBIiclEApg1VKX7bkKxVdu/vsFdw==
   dependencies:
-    "@sentry/hub" "5.19.2"
+    "@sentry/hub" "6.13.3"
-    "@sentry/minimal" "5.19.2"
+    "@sentry/types" "6.13.3"
-    "@sentry/types" "5.19.2"
-    "@sentry/utils" "5.19.2"
     tslib "^1.9.3"
 
-"@sentry/hub@5.19.2":
+"@sentry/node@^6.0.0":
-  version "5.19.2"
+  version "6.13.3"
-  resolved "https://registry.yarnpkg.com/@sentry/hub/-/hub-5.19.2.tgz#ab7f3d2d253c3441b2833a530b17c6de2418b2c7"
+  resolved "https://registry.yarnpkg.com/@sentry/node/-/node-6.13.3.tgz#94c646c31fd240ab68ee8b85aa663e65eb499d06"
-  integrity sha512-2KkEYX4q9TDCOiaVEo2kQ1W0IXyZxJxZtIjDdFQyes9T4ubYlKHAbvCjTxHSQv37lDO4t7sOIApWG9rlkHzlEA==
+  integrity sha512-ZeZSw+TcPcf4e0j7iEqNMtoVmz+WFW/TEoGokXIwysZqSgchKdAXDHqn+CqUqFan7d76JcJmzztAUK2JruQ2Kg==
   dependencies:
-    "@sentry/types" "5.19.2"
+    "@sentry/core" "6.13.3"
-    "@sentry/utils" "5.19.2"
+    "@sentry/hub" "6.13.3"
-    tslib "^1.9.3"
+    "@sentry/tracing" "6.13.3"
-
+    "@sentry/types" "6.13.3"
-"@sentry/minimal@5.19.2":
+    "@sentry/utils" "6.13.3"
-  version "5.19.2"
+    cookie "^0.4.1"
-  resolved "https://registry.yarnpkg.com/@sentry/minimal/-/minimal-5.19.2.tgz#0fc2fdf9911a0cb31b52f7ccad061b74785724a3"
-  integrity sha512-rApEOkjy+ZmkeqEItgFvUFxe5l+dht9AumuUzq74pWp+HJqxxv9IVTusKppBsE1adjtmyhwK4O3Wr8qyc75xlw==
-  dependencies:
-    "@sentry/hub" "5.19.2"
-    "@sentry/types" "5.19.2"
-    tslib "^1.9.3"
-
-"@sentry/node@5.19.2":
-  version "5.19.2"
-  resolved "https://registry.yarnpkg.com/@sentry/node/-/node-5.19.2.tgz#8c1c2f6c983c3d8b25143e5b99c4b6cc745125ec"
-  integrity sha512-gbww3iTWkdvYIAhOmULbv8znKwkIpklGJ0SPtAh0orUMuaa0lVht+6HQIhRgeXp50lMzNaYC3fuzkbFfYgpS7A==
-  dependencies:
-    "@sentry/apm" "5.19.2"
-    "@sentry/core" "5.19.2"
-    "@sentry/hub" "5.19.2"
-    "@sentry/types" "5.19.2"
-    "@sentry/utils" "5.19.2"
-    cookie "^0.3.1"
     https-proxy-agent "^5.0.0"
     lru_map "^0.3.3"
     tslib "^1.9.3"
 
-"@sentry/types@5.19.2":
+"@sentry/tracing@6.13.3":
-  version "5.19.2"
+  version "6.13.3"
-  resolved "https://registry.yarnpkg.com/@sentry/types/-/types-5.19.2.tgz#ead586f0b64b91c396d3521b938ca25f7b59d655"
+  resolved "https://registry.yarnpkg.com/@sentry/tracing/-/tracing-6.13.3.tgz#ca657d4afa99c50f15e638fe38405bac33e780ee"
-  integrity sha512-O6zkW8oM1qK5Uma9+B/UMlmlm9/gkw9MooqycWuEhIaKfDBj/yVbwb/UTiJmNkGc5VJQo0v1uXUZZQt6/Xq1GA==
+  integrity sha512-yyOFIhqlprPM0g4f35Icear3eZk2mwyYcGEzljJfY2iU6pJwj1lzia5PfSwiCW7jFGMmlBJNhOAIpfhlliZi8Q==
-
-"@sentry/utils@5.19.2":
-  version "5.19.2"
-  resolved "https://registry.yarnpkg.com/@sentry/utils/-/utils-5.19.2.tgz#f2819d9de5abc33173019e81955904247e4a8246"
-  integrity sha512-gEPkC0CJwvIWqcTcPSdIzqJkJa9N5vZzUZyBvdu1oiyJu7MfazpJEvj3whfJMysSfXJQxoJ+a1IPrA73VY23VA==
   dependencies:
-    "@sentry/types" "5.19.2"
+    "@sentry/hub" "6.13.3"
+    "@sentry/minimal" "6.13.3"
+    "@sentry/types" "6.13.3"
+    "@sentry/utils" "6.13.3"
+    tslib "^1.9.3"
+
+"@sentry/types@6.13.3":
+  version "6.13.3"
+  resolved "https://registry.yarnpkg.com/@sentry/types/-/types-6.13.3.tgz#63ad5b6735b0dfd90b3a256a9f8e77b93f0f66b2"
+  integrity sha512-Vrz5CdhaTRSvCQjSyIFIaV9PodjAVFkzJkTRxyY7P77RcegMsRSsG1yzlvCtA99zG9+e6MfoJOgbOCwuZids5A==
+
+"@sentry/utils@6.13.3":
+  version "6.13.3"
+  resolved "https://registry.yarnpkg.com/@sentry/utils/-/utils-6.13.3.tgz#188754d40afe693c3fcae410f9322531588a9926"
+  integrity sha512-zYFuFH3MaYtBZTeJ4Yajg7pDf0pM3MWs3+9k5my9Fd+eqNcl7dYQYJbT9gyC0HXK1QI4CAMNNlHNl4YXhF91ag==
+  dependencies:
+    "@sentry/types" "6.13.3"
     tslib "^1.9.3"
 
 "@sindresorhus/is@^0.14.0":
@@ -3860,10 +3849,10 @@ cookie@0.4.0:
   resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.4.0.tgz#beb437e7022b3b6d49019d088665303ebe9c14ba"
   integrity sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==
 
-cookie@^0.3.1:
+cookie@^0.4.1:
-  version "0.3.1"
+  version "0.4.1"
-  resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.3.1.tgz#e7e0a1f9ef43b4c8ba925c5c5a96e806d16873bb"
+  resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.4.1.tgz#afd713fe26ebd21ba95ceb61f9a8116e50a537d1"
-  integrity sha1-5+Ch+e9DtMi6klxcWpboBtFoc7s=
+  integrity sha512-ZwrFkGJxUR3EIoXtO+yVE69Eb7KlixbaeAWfBQB9vVsNn/o+Yw69gBWSSDK825hQNdN+wF8zELf3dFNl/kxkUA==
 
 cookiejar@^2.1.0:
   version "2.1.2"
@@ -11811,6 +11800,11 @@ verror@1.10.0:
     core-util-is "1.0.2"
     extsprintf "^1.2.0"
 
+vm2@^3.9.3:
+  version "3.9.3"
+  resolved "https://registry.yarnpkg.com/vm2/-/vm2-3.9.3.tgz#29917f6cc081cc43a3f580c26c5b553fd3c91f40"
+  integrity sha512-smLS+18RjXYMl9joyJxMNI9l4w7biW8ilSDaVRvFBDwOH8P0BK1ognFQTpg0wyQ6wIKLTblHJvROW692L/E53Q==
+
 vuvuzela@1.0.3:
   version "1.0.3"
   resolved "https://registry.yarnpkg.com/vuvuzela/-/vuvuzela-1.0.3.tgz#3be145e58271c73ca55279dd851f12a682114b0b"

packages/string-templates/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/string-templates",
-  "version": "0.9.167-alpha.2",
+  "version": "0.9.169-alpha.10",
   "description": "Handlebars wrapper for Budibase templating.",
   "main": "src/index.cjs",
   "module": "dist/bundle.mjs",
@@ -24,7 +24,8 @@
     "dayjs": "^1.10.4",
     "handlebars": "^4.7.6",
     "handlebars-utils": "^1.0.6",
-    "lodash": "^4.17.20"
+    "lodash": "^4.17.20",
+    "vm2": "^3.9.4"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^17.1.0",

packages/string-templates/rollup.config.js

@@ -7,18 +7,6 @@ import globals from "rollup-plugin-node-globals"
 
 const production = !process.env.ROLLUP_WATCH
 
-const plugins = [
-  resolve({
-    preferBuiltins: true,
-    browser: true,
-  }),
-  commonjs(),
-  globals(),
-  builtins(),
-  json(),
-  production && terser(),
-]
-
 export default [
   {
     input: "src/index.mjs",
@@ -27,18 +15,16 @@ export default [
       format: "esm",
       file: "./dist/bundle.mjs",
     },
-    plugins,
+    plugins: [
+      resolve({
+        preferBuiltins: true,
+        browser: true,
+      }),
+      commonjs(),
+      globals(),
+      builtins(),
+      json(),
+      production && terser(),
+    ],
   },
-  // This is the valid configuration for a CommonJS bundle, but since we have
-  // no use for this, it's better to leave it out.
-  // {
-  //   input: "src/index.cjs",
-  //   output: {
-  //     sourcemap: !production,
-  //     format: "cjs",
-  //     file: "./dist/bundle.cjs",
-  //     exports: "named",
-  //   },
-  //   plugins,
-  // },
 ]

packages/string-templates/src/helpers/Helper.js

@@ -1,13 +1,23 @@
 class Helper {
-  constructor(name, fn) {
+  constructor(name, fn, useValueFallback = true) {
     this.name = name
     this.fn = fn
+    this.useValueFallback = useValueFallback
   }
 
   register(handlebars) {
     // wrap the function so that no helper can cause handlebars to break
-    handlebars.registerHelper(this.name, value => {
+    handlebars.registerHelper(this.name, (value, info) => {
-      return this.fn(value) || value
+      let context = {}
+      if (info && info.data && info.data.root) {
+        context = info.data.root
+      }
+      const result = this.fn(value, context)
+      if (result == null) {
+        return this.useValueFallback ? value : null
+      } else {
+        return result
+      }
     })
   }
 

packages/string-templates/src/helpers/constants.js

@@ -19,6 +19,7 @@ module.exports.HelperFunctionNames = {
   OBJECT: "object",
   ALL: "all",
   LITERAL: "literal",
+  JS: "js",
 }
 
 module.exports.LITERAL_MARKER = "%LITERAL%"

packages/string-templates/src/helpers/index.js

@@ -1,6 +1,7 @@
 const Helper = require("./Helper")
 const { SafeString } = require("handlebars")
 const externalHandlebars = require("./external")
+const { processJS } = require("./javascript")
 const {
   HelperFunctionNames,
   HelperFunctionBuiltin,
@@ -17,6 +18,8 @@ const HELPERS = [
   new Helper(HelperFunctionNames.OBJECT, value => {
     return new SafeString(JSON.stringify(value))
   }),
+  // javascript helper
+  new Helper(HelperFunctionNames.JS, processJS, false),
   // this help is applied to all statements
   new Helper(HelperFunctionNames.ALL, value => {
     if (

packages/string-templates/src/helpers/javascript.js

@@ -0,0 +1,49 @@
+const { atob } = require("../utilities")
+
+// The method of executing JS scripts depends on the bundle being built.
+// This setter is used in the entrypoint (either index.cjs or index.mjs).
+let runJS
+module.exports.setJSRunner = runner => (runJS = runner)
+
+// Helper utility to strip square brackets from a value
+const removeSquareBrackets = value => {
+  if (!value || typeof value !== "string") {
+    return value
+  }
+  const regex = /\[+(.+)]+/
+  const matches = value.match(regex)
+  if (matches && matches[1]) {
+    return matches[1]
+  }
+  return value
+}
+
+// Our context getter function provided to JS code as $.
+// Extracts a value from context.
+const getContextValue = (path, context) => {
+  let data = context
+  path.split(".").forEach(key => {
+    if (data == null || typeof data !== "object") {
+      return null
+    }
+    data = data[removeSquareBrackets(key)]
+  })
+  return data
+}
+
+// Evaluates JS code against a certain context
+module.exports.processJS = (handlebars, context) => {
+  try {
+    // Wrap JS in a function and immediately invoke it.
+    // This is required to allow the final `return` statement to be valid.
+    const js = `function run(){${atob(handlebars)}};run();`
+
+    // Our $ context function gets a value from context
+    const sandboxContext = { $: path => getContextValue(path, context) }
+
+    // Create a sandbox with out context and run the JS
+    return runJS(js, sandboxContext)
+  } catch (error) {
+    return "Error while executing JS"
+  }
+}