Merge branch 'develop' of github.com:Budibase/budibase into s3-upload

.github/CONTRIBUTING.md

@@ -79,6 +79,8 @@ Component libraries are collections of components as well as the definition of t
 ### Getting Started For Contributors
 #### 1.  Prerequisites
 
+NodeJS Version `14.x.x`
+
 *yarn -* `npm install -g yarn`
 
 *jest* - `npm install -g jest`
@@ -177,36 +179,7 @@ To enable this mode, use:
 yarn mode:account
 ```
 ### CI
-
+ An overview of the CI pipelines can be found [here](./workflows/README.md)
-#### PR Job
-
-After your pr is submitted a github action (can be found at `.github/workflows/budibase_ci.yml`) will run to perform some checks against the changes such as linting, build and test. 
-
-The job will run when changes are pushed to or targetted at `master` and `develop`
-#### Release Develop
-
-To test changes before a release, a prerelease action (can be found at `.github/workflows/release-develop.yml`) will run to build and release develop versions of npm packages and docker images. On each subsequent commit to develop a new alpha version of npm packages will be created and released. 
-
-For example:
-
-- `feature1` -> `develop` = `v0.9.160-alpha.1`
-- `feature2` -> `develop` = `v0.9.160-alpha.0`
-
-The job will run when changes are pushed to `develop`
-#### Release Job
-
-To release changes a release job (can be found at `.github/workflows/release.yml`) will run to create final versions of npm packages and docker images. 
-
-Following the example above:
-
-- `develop` -> `master` = `v0.9.160`
-
-The job will run when changes are pushed to `master`
-
-#### Release Self Host Job
-
-To release the self hosted version of docker images, an additional job (can be found at `.github/workflows/release-selfhost.yml`) must be ran manually. This will releaae docker images to docker hub under the tag `latest` to be picked up by self hosted installations. 
-
 ### Troubleshooting
 
 Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above. You should have a fresh Budibase installation.

.github/workflows/README.md

@@ -0,0 +1,93 @@
+
+# Budibase CI Pipelines
+
+Welcome to the budibase CI pipelines directory. This document details what each of the CI pipelines are for, and come common combinations.
+
+## All CI Pipelines
+
+### Note
+- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs.
+
+### Standard CI Build Job (budibase_ci.yml)
+Triggers:
+- PR or push to develop
+- PR or push to master
+
+The standard CI Build job is what runs when you raise a PR to develop or master. 
+- Installs all dependencies,
+- builds the project 
+- run the unit tests
+- Generate test coverage metrics with codecov
+- Run the cypress tests
+
+### Release Develop Job (release-develop.yml)
+Triggers:
+- Push to develop
+
+The job responsible for building, tagging and pushing docker images out to the test and staging environments. 
+- Installs all dependencies
+- builds the project 
+- run the unit tests
+- publish the budibase JS packages under a prerelease tag to NPM
+- build, tag and push docker images under the `develop` tag to docker hub
+
+These images will then be pulled by the test and staging environments, updating the latest automatically. Discord notifications are sent to the #infra channel when this occurs.
+
+### Release Job (release.yml)
+Triggers:
+- Push to master
+
+This job is responsible for building and pushing the latest code to NPM and docker hub, so that it can be deployed.
+- Installs all dependencies
+- builds the project 
+- run the unit tests
+- publish the budibase JS packages under a release tag to NPM (always incremented by patch versions)
+- build, tag and push docker images under the `v.x.x.x` (the tag of the NPM release) tag to docker hub
+
+### Release Selfhost Job (release-selfhost.yml)
+Triggers:
+- Manual Workflow Dispatch Trigger
+
+This job is responsible for delivering the latest version of budibase to those that are self-hosting. 
+
+This job relies on the release job to have run first, so the latest image is pushed to dockerhub. This job then will pull the latest version from `lerna.json` and try to find an image in dockerhub corresponding to that version. For example, if the version in `lerna.json` is `1.0.0`:
+- Pull the images for all budibase services tagged `v1.0.0` from dockerhub
+- Tag these images as `latest`
+- Push them back to dockerhub. This now means anyone who pulls `latest` (self hosters using docker-compose) will get the latest version.
+- Build and release the budibase helm chart for kubernetes users
+- Perform a github release with the latest version. You can see previous releases here (https://github.com/Budibase/budibase/releases)
+
+
+### Cloud Deploy (deploy-cloud.yml)
+Triggers:
+- Manual Workflow Dispatch Trigger
+
+This job is responsible for deploying to our production, cloud kubernetes environment. You must run the release job first, to ensure that the latest images have been built and pushed to docker hub. You can also manually enter a version number for this job, so you can perform rollbacks or upgrade to a specific version. After kicking off this job, the following will occur:
+
+- Checks out the master branch 
+- Pulls the latest `values.yaml` from budibase infra, a private repo containing budibases infrastructure configuration
+- Gets the latest budibase version from `lerna.json`, if it hasn't been specified in the workflow when you kicked it off
+- Configures AWS Credentials  
+- Deploys the helm chart in the budibase repo to our production EKS cluster, injecting the `values.yaml` we pulled from budibase-infra
+- Fires off a discord webhook in the #infra channel to show that the deployment completely successfully.
+
+## Common Workflows
+
+### Deploy Changes to Production (Release)
+- Merge `develop` into `master`
+- Wait for budibase CI job and release job to run
+- Run cloud deploy job
+- Run release selfhost job
+
+### Deploy Changes to Production (Hotfix)
+- Branch off `master`
+- Perform your hotfix
+- Merge back into `master`
+- Wait for budibase CI job and release job to run
+- Run cloud deploy job
+- Run release selfhost job
+
+### Rollback A Bad Cloud Deployment
+- Kick off cloud deploy job
+- Ensure you are running off master
+- Enter the version number of the last known good version of budibase. For example `1.0.0`

.github/workflows/budibase_ci.yml

@@ -41,4 +41,6 @@ jobs:
         files: ./packages/server/coverage/clover.xml
         name: codecov-umbrella
         verbose: true 
+
+    # TODO: parallelise this
     - run: yarn test:e2e:ci

.github/workflows/deploy-cloud.yaml

@@ -12,6 +12,12 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
+      - name: Fail if branch is not master
+        if: github.ref != 'refs/heads/master' 
+        run: | 
+          echo "Ref is not master, you must run this job from master."
+          exit 1
+
       - uses: actions/checkout@v2
 
       - name: Pull values.yaml from budibase-infra

.github/workflows/deploy-preprod.yml

@@ -23,16 +23,19 @@ jobs:
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: eu-west-1
 
-      - name: 'Get Previous tag'
+      - name: Get the latest budibase release version
-        id: previoustag
+        id: version
-        uses: "WyriHaximus/github-action-get-previous-tag@v1"
+        run: | 
+          release_version=$(cat lerna.json | jq -r '.version')
+          echo "RELEASE_VERSION=$release_version" >> $GITHUB_ENV
 
-      # - name: Pull values.yaml from budibase-infra
+      - name: Pull values.yaml from budibase-infra
-      #   run: | 
+        run: | 
-      #     curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \                                                                                [c3a7a9d12]
+          curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \
-      #     -H 'Accept: application/vnd.github.v3.raw' \
+          -H 'Accept: application/vnd.github.v3.raw' \
-      #     -o values.preprod.yaml \
+          -o values.preprod.yaml \
-      #     -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml
+          -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/budibase-preprod/values.yaml
+          wc -l values.preprod.yaml
 
       - name: Deploy to Preprod Environment
         uses: deliverybot/helm@v1
@@ -41,13 +44,17 @@ jobs:
           namespace: budibase
           chart: charts/budibase
           token: ${{ github.token }}
+          helm: helm3
           values: |
             globals: 
-              appVersion: ${{ steps.previoustag.outputs.tag }}
+              appVersion: v${{ env.RELEASE_VERSION }}
-          # value-files: >-
+            ingress:
-          #   [
+              enabled: true
-          #     "charts/budibase/values.yaml"
+              nginx: true
-          #   ]
+          value-files: >-
+            [
+              "values.preprod.yaml"
+            ]
         env:
           KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}'
 

.github/workflows/release-develop.yml

@@ -4,6 +4,16 @@ on:
  push:
     branches:
       - develop
+    paths:
+      - '.aws/**'
+      - '.github/**'
+      - 'charts/**'
+      - 'packages/**'
+      - 'scripts/**'
+      - 'package.json'
+      - 'yarn.lock'
+      - 'package.json'
+      - 'yarn.lock'
 
 env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}

.github/workflows/release-selfhost.yml

@@ -9,9 +9,9 @@ jobs:
 
     steps:
       - uses: actions/checkout@v2
-      - uses: actions/setup-node@v1
         with: 
           node-version: 14.x
+          fetch_depth: 0
 
       - name: Tag and release Docker images (Self Host)
         run: | 
@@ -38,10 +38,6 @@ jobs:
           DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }}
           SELFHOST_TAG: latest
       
-      - name: Setup Helm
-        uses: azure/setup-helm@v1
-        id: helm-install
-
       - name: Build CLI executables
         run: |
           pushd packages/cli
@@ -49,25 +45,24 @@ jobs:
           yarn build
           popd
       
-      # - name: Build and release helm chart
+      - name: Setup Helm 
-      #   run: |
+        uses: azure/setup-helm@v1
-      #     git config user.name "Budibase Helm Bot"
+        id: helm-install
-      #     git config user.email "<>"
-      #     mv budibase-${{ env.RELEASE_VERSION }}.tgz docs
-      #     helm repo index docs
-      #     git checkout gh-pages
-      #     git add -A
-      #     git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}"
-      #     git push
-      #   env:
-      #     GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
-      # - name: Deploy
+      - name: Build and release helm chart
-      #   uses: peaceiris/actions-gh-pages@v3
+        run: | 
-      #   with:
+          git config user.name "Budibase Helm Bot"
-      #     github_token: ${{ secrets.GITHUB_TOKEN }}
+          git config user.email "<>"
-      #     publish_dir: ./public
+          git pull
-      #     full_commit_message: "Helm Release: ${{ env.RELEASE_VERSION }}"
+          helm package charts/budibase
+          git checkout gh-pages
+          mv *.tgz docs
+          helm repo index docs
+          git add -A
+          git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}"
+          git push
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Perform Github Release
         uses: softprops/action-gh-release@v1

.github/workflows/release.yml

@@ -3,7 +3,17 @@ name: Budibase Release
 on:
  push:
     branches:
-      - test
+      - master
+    paths:
+      - '.aws/**'
+      - '.github/**'
+      - 'charts/**'
+      - 'packages/**'
+      - 'scripts/**'
+      - 'package.json'
+      - 'yarn.lock'
+      - 'package.json'
+      - 'yarn.lock'
 
 env:
   POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }}
@@ -56,34 +66,3 @@ jobs:
           DOCKER_USER: ${{ secrets.DOCKER_USERNAME }}
           DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }}
           BUDIBASE_RELEASE_VERSION: ${{ steps.previoustag.outputs.tag }}
-      
-      # - name: Pull values.yaml from budibase-infra
-      #   run: | 
-      #     curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \                                                                                [c3a7a9d12]
-      #     -H 'Accept: application/vnd.github.v3.raw' \
-      #     -o values.preprod.yaml \
-      #     -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml
-
-      # - name: Deploy to Preprod Environment
-      #   uses: deliverybot/helm@v1
-      #   with:
-      #     release: budibase-preprod
-      #     namespace: budibase
-      #     chart: charts/budibase
-      #     token: ${{ github.token }}
-      #     values: |
-      #       globals: 
-      #         appVersion: ${{ steps.previoustag.outputs.tag }}
-      #     # value-files: >-
-      #     #   [
-      #     #     "charts/budibase/values.yaml"
-      #     #   ]
-      #   env:
-      #     KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}'
-
-      # - name: Discord Webhook Action
-      #   uses: tsickert/discord-webhook@v4.0.0
-      #   with:
-      #     webhook-url: ${{ secrets.PROD_DEPLOY_WEBHOOK_URL }}
-      #     content: "Preprod Deployment Complete: ${{ env.RELEASE_VERSION }} deployed to Budibase Pre-prod."
-      #     embed-title: ${{ env.RELEASE_VERSION }}

charts/budibase/Chart.lock

@@ -0,0 +1,9 @@
+dependencies:
+- name: couchdb
+  repository: https://apache.github.io/couchdb-helm
+  version: 3.3.4
+- name: ingress-nginx
+  repository: https://kubernetes.github.io/ingress-nginx
+  version: 4.0.13
+digest: sha256:20892705c2d8e64c98257d181063a514ac55013e2b43399a6e54868a97f97845
+generated: "2021-12-30T18:55:30.878411Z"

charts/budibase/Chart.yaml

@@ -11,14 +11,14 @@ sources:
   - https://github.com/Budibase/budibase
   - https://budibase.com
 type: application
-version: 1.0.0
+version: 0.2.5
-appVersion: 1.0.20
+appVersion: 1.0.25
 dependencies:
   - name: couchdb
     version: 3.3.4
     repository: https://apache.github.io/couchdb-helm
     condition: services.couchdb.enabled
   - name: ingress-nginx
-    version: 3.35.0
+    version: 4.0.13
-    repository: https://github.com/kubernetes/ingress-nginx
+    repository: https://kubernetes.github.io/ingress-nginx
     condition: ingress.nginx

charts/budibase/templates/proxy-service-service.yaml

@@ -9,11 +9,10 @@ metadata:
     app.kubernetes.io/name: budibase-proxy
   name: proxy-service
 spec:
-  type: NodePort
   ports:
-  - port: {{ .Values.services.proxy.port }}
+  - name: {{ .Values.services.proxy.port | quote }}
+    port: {{ .Values.services.proxy.port }}
     targetPort: {{ .Values.services.proxy.port }}
-    protocol: TCP
   selector:
     app.kubernetes.io/name: budibase-proxy
 status:

charts/budibase/values.yaml

@@ -40,7 +40,7 @@ service:
   port: 10000
 
 ingress:
-  enabled: false
+  enabled: true
   aws: false
   nginx: true
   certificateArn: ""

lerna.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.0.27-alpha.6",
+  "version": "1.0.27-alpha.17",
   "npmClient": "yarn",
   "packages": [
     "packages/*"

packages/backend-core/README.md

@@ -4,7 +4,7 @@ This library contains core functionality, like auth and security features
 which are shared between backend services.
 
 #### Note about top level JS files
-For the purposes of being able to do say `require("@budibase/auth/permissions")` we need to
+For the purposes of being able to do say `require("@budibase/backend-core/permissions")` we need to
 specify the exports at the top-level of the module.
 
 For these files they should be limited to a single `require` of the file that should

packages/backend-core/auth.js

@@ -0,0 +1 @@
+module.exports = require("./src/auth")

packages/backend-core/middleware.js

@@ -0,0 +1 @@
+module.exports = require("./src/middleware")

packages/backend-core/objectStore.js

@@ -0,0 +1,4 @@
+module.exports = {
+  ...require("./src/objectStore"),
+  ...require("./src/objectStore/utils"),
+}

packages/backend-core/package.json

@@ -1,7 +1,7 @@
 {
-  "name": "@budibase/auth",
+  "name": "@budibase/backend-core",
-  "version": "1.0.27-alpha.6",
+  "version": "1.0.27-alpha.17",
-  "description": "Authentication middlewares for budibase builder and apps",
+  "description": "Budibase backend core libraries used in server and worker",
   "main": "src/index.js",
   "author": "Budibase",
   "license": "GPL-3.0",

packages/backend-core/src/auth.js

@@ -1,7 +1,6 @@
 const passport = require("koa-passport")
 const LocalStrategy = require("passport-local").Strategy
 const JwtStrategy = require("passport-jwt").Strategy
-const { StaticDatabases } = require("./db/utils")
 const { getGlobalDB } = require("./tenancy")
 const {
   jwt,
@@ -14,8 +13,6 @@ const {
   appTenancy,
   authError,
 } = require("./middleware")
-const { setDB } = require("./db")
-const userCache = require("./cache/user")
 
 // Strategies
 passport.use(new LocalStrategy(local.options, local.authenticate))
@@ -36,36 +33,13 @@ passport.deserializeUser(async (user, done) => {
 })
 
 module.exports = {
-  init(pouch) {
+  buildAuthMiddleware: authenticated,
-    setDB(pouch)
+  passport,
-  },
+  google,
-  db: require("./db/utils"),
+  oidc,
-  redis: {
+  jwt: require("jsonwebtoken"),
-    Client: require("./redis"),
+  buildTenancyMiddleware: tenancy,
-    utils: require("./redis/utils"),
+  buildAppTenancyMiddleware: appTenancy,
-  },
+  auditLog,
-  objectStore: {
+  authError,
-    ...require("./objectStore"),
-    ...require("./objectStore/utils"),
-  },
-  utils: {
-    ...require("./utils"),
-    ...require("./hashing"),
-  },
-  auth: {
-    buildAuthMiddleware: authenticated,
-    passport,
-    google,
-    oidc,
-    jwt: require("jsonwebtoken"),
-    buildTenancyMiddleware: tenancy,
-    buildAppTenancyMiddleware: appTenancy,
-    auditLog,
-    authError,
-  },
-  cache: {
-    user: userCache,
-  },
-  StaticDatabases,
-  constants: require("./constants"),
 }

packages/backend-core/src/db/utils.js

@@ -224,8 +224,15 @@ exports.getAllDbs = async () => {
     }
   }
   let couchUrl = `${exports.getCouchUrl()}/_all_dbs`
-  if (env.MULTI_TENANCY) {
+  let tenantId = getTenantId()
-    let tenantId = getTenantId()
+  if (!env.MULTI_TENANCY || tenantId == DEFAULT_TENANT_ID) {
+    // just get all DBs when:
+    // - single tenancy
+    // - default tenant
+    //    - apps dbs don't contain tenant id
+    //    - non-default tenant dbs are filtered out application side in getAllApps
+    await addDbs(couchUrl)
+  } else {
     // get prod apps
     await addDbs(
       exports.getStartEndKeyURL(couchUrl, DocumentTypes.APP, tenantId)
@@ -236,9 +243,6 @@ exports.getAllDbs = async () => {
     )
     // add global db name
     dbs.push(getGlobalDBName(tenantId))
-  } else {
-    // just get all DBs in self host
-    await addDbs(couchUrl)
   }
   return dbs
 }

packages/backend-core/src/index.js

@@ -0,0 +1,17 @@
+const { setDB } = require("./db")
+
+module.exports = {
+  init(pouch) {
+    setDB(pouch)
+  },
+  // some default exports from the library, however these ideally shouldn't
+  // be used, instead the syntax require("@budibase/backend-core/db") should be used
+  StaticDatabases: require("./db/utils").StaticDatabases,
+  db: require("../db"),
+  redis: require("../redis"),
+  objectStore: require("../objectStore"),
+  utils: require("../utils"),
+  cache: require("../cache"),
+  auth: require("../auth"),
+  constants: require("../constants"),
+}

packages/backend-core/src/objectStore/index.js

@@ -206,6 +206,34 @@ exports.retrieveToTmp = async (bucketName, filepath) => {
   return outputPath
 }
 
+/**
+ * Delete a single file.
+ */
+exports.deleteFile = async (bucketName, filepath) => {
+  const objectStore = exports.ObjectStore(bucketName)
+  await exports.makeSureBucketExists(objectStore, bucketName)
+  const params = {
+    Bucket: bucketName,
+    Key: filepath,
+  }
+  return objectStore.deleteObject(params)
+}
+
+exports.deleteFiles = async (bucketName, filepaths) => {
+  const objectStore = exports.ObjectStore(bucketName)
+  await exports.makeSureBucketExists(objectStore, bucketName)
+  const params = {
+    Bucket: bucketName,
+    Delete: {
+      Objects: filepaths.map(path => ({ Key: path })),
+    },
+  }
+  return objectStore.deleteObjects(params).promise()
+}
+
+/**
+ * Delete a path, including everything within.
+ */
 exports.deleteFolder = async (bucketName, folder) => {
   bucketName = sanitizeBucket(bucketName)
   folder = sanitizeKey(folder)

packages/backend-core/utils.js

@@ -0,0 +1,4 @@
+module.exports = {
+  ...require("./src/utils"),
+  ...require("./src/hashing"),
+}

packages/bbui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/bbui",
   "description": "A UI solution used in the different Budibase projects.",
-  "version": "1.0.27-alpha.6",
+  "version": "1.0.27-alpha.17",
   "license": "MPL-2.0",
   "svelte": "src/index.js",
   "module": "dist/bbui.es.js",

packages/bbui/src/Tabs/Tabs.svelte

@@ -10,6 +10,8 @@
   export let noHorizPadding = false
   export let quiet = false
   export let emphasized = false
+  // overlay content from the tab bar onto tabs e.g. for a dropdown
+  export let onTop = false
 
   let thisSelected = undefined
 
@@ -78,6 +80,7 @@
     'spectrum-Tabs--quiet'} spectrum-Tabs--{vertical
     ? 'vertical'
     : 'horizontal'}"
+  class:onTop
 >
   <slot />
   {#if $tab.info}
@@ -98,7 +101,9 @@
   .quiet {
     border-bottom: none !important;
   }
-
+  .onTop {
+    z-index: 20;
+  }
   .spectrum-Tabs {
     padding-left: var(--spacing-xl);
     padding-right: var(--spacing-xl);

packages/bbui/yarn.lock

@@ -2076,9 +2076,9 @@ postcss@^7.0.0, postcss@^7.0.1, postcss@^7.0.27:
     supports-color "^6.1.0"
 
 postcss@^8.2.9:
-  version "8.2.10"
+  version "8.2.13"
-  resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.2.10.tgz#ca7a042aa8aff494b334d0ff3e9e77079f6f702b"
+  resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.2.13.tgz#dbe043e26e3c068e45113b1ed6375d2d37e2129f"
-  integrity sha512-b/h7CPV7QEdrqIxtAf2j31U5ef05uBDuvoXv6L51Q4rcS1jdlXAVKJv+atCFdUXYl9dyTHGyoMzIepwowRJjFw==
+  integrity sha512-FCE5xLH+hjbzRdpbRb1IMCvPv9yZx2QnDarBEYSN0N0HYk+TcXsEhwdFcFb+SRWOKzKGErhIEbBK2ogyLdTtfQ==
   dependencies:
     colorette "^1.2.2"
     nanoid "^3.1.22"

packages/builder/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/builder",
-  "version": "1.0.27-alpha.6",
+  "version": "1.0.27-alpha.17",
   "license": "GPL-3.0",
   "private": true,
   "scripts": {
@@ -65,10 +65,10 @@
     }
   },
   "dependencies": {
-    "@budibase/bbui": "^1.0.27-alpha.6",
+    "@budibase/bbui": "^1.0.27-alpha.17",
-    "@budibase/client": "^1.0.27-alpha.6",
+    "@budibase/client": "^1.0.27-alpha.17",
     "@budibase/colorpicker": "1.1.2",
-    "@budibase/string-templates": "^1.0.27-alpha.6",
+    "@budibase/string-templates": "^1.0.27-alpha.17",
     "@sentry/browser": "5.19.1",
     "@spectrum-css/page": "^3.0.1",
     "@spectrum-css/vars": "^3.0.1",

packages/builder/src/builderStore/datasource.js

@@ -23,10 +23,10 @@ function prepareData(config) {
   return datasource
 }
 
-export async function saveDatasource(config) {
+export async function saveDatasource(config, skipFetch = false) {
   const datasource = prepareData(config)
   // Create datasource
-  const resp = await datasources.save(datasource, datasource.plus)
+  const resp = await datasources.save(datasource, !skipFetch && datasource.plus)
 
   // update the tables incase data source plus
   await tables.fetch()

packages/builder/src/components/automation/AutomationBuilder/FlowChart/FlowItem.svelte

@@ -9,6 +9,7 @@
     Modal,
     Button,
     StatusLight,
+    ActionButton,
   } from "@budibase/bbui"
   import AutomationBlockSetup from "../../SetupPanel/AutomationBlockSetup.svelte"
   import CreateWebhookModal from "components/automation/Shared/CreateWebhookModal.svelte"
@@ -27,7 +28,7 @@
   let blockComplete
 
   $: testResult = $automationStore.selectedAutomation.testResults?.steps.filter(
-    step => step.stepId === block.stepId
+    step => (block.id ? step.id === block.id : step.stepId === block.stepId)
   )
   $: isTrigger = block.type === "TRIGGER"
 
@@ -119,19 +120,13 @@
     <div class="blockSection">
       <Layout noPadding gap="S">
         <div class="splitHeader">
-          <div
+          <ActionButton
-            on:click|stopPropagation={() => {
+            on:click={() => (setupToggled = !setupToggled)}
-              setupToggled = !setupToggled
+            quiet
-            }}
+            icon={setupToggled ? "ChevronDown" : "ChevronRight"}
-            class="center-items"
           >
-            {#if setupToggled}
-              <Icon size="M" name="ChevronDown" />
-            {:else}
-              <Icon size="M" name="ChevronRight" />
-            {/if}
             <Detail size="S">Setup</Detail>
-          </div>
+          </ActionButton>
           {#if !isTrigger}
             <div on:click={() => deleteStep()}>
               <Icon name="DeleteOutline" />
@@ -187,6 +182,7 @@
   .splitHeader {
     display: flex;
     justify-content: space-between;
+    align-items: center;
   }
   .iconAlign {
     padding: 0 0 0 var(--spacing-m);

packages/builder/src/components/automation/SetupPanel/AutomationBlockSetup.svelte

@@ -10,6 +10,7 @@
     ActionButton,
     Drawer,
     Modal,
+    Detail,
   } from "@budibase/bbui"
   import CreateWebhookModal from "components/automation/Shared/CreateWebhookModal.svelte"
 
@@ -37,6 +38,7 @@
   let drawer
   let tempFilters = lookForFilters(schemaProperties) || []
   let fillWidth = true
+  let codeBindingOpen = false
 
   $: stepId = block.stepId
   $: bindings = getAvailableBindings(
@@ -233,7 +235,16 @@
         <SchemaSetup on:change={e => onChange(e, key)} value={inputData[key]} />
       {:else if value.customType === "code"}
         <CodeEditorModal>
-          <pre>{JSON.stringify(bindings, null, 2)}</pre>
+          <ActionButton
+            on:click={() => (codeBindingOpen = !codeBindingOpen)}
+            quiet
+            icon={codeBindingOpen ? "ChevronDown" : "ChevronRight"}
+          >
+            <Detail size="S">Bindings</Detail>
+          </ActionButton>
+          {#if codeBindingOpen}
+            <pre>{JSON.stringify(bindings, null, 2)}</pre>
+          {/if}
           <Editor
             mode="javascript"
             on:change={e => {

packages/builder/src/components/backend/DatasourceNavigator/DatasourceNavigator.svelte

@@ -137,7 +137,7 @@
             selected={$queries.selected === query._id}
             on:click={() => onClickQuery(query)}
           >
-            <EditQueryPopover {query} />
+            <EditQueryPopover {query} {onClickQuery} />
           </NavItem>
         {/each}
       {/if}

packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/PlusConfigForm.svelte

@@ -199,18 +199,18 @@
   <Body>
     Tell budibase how your tables are related to get even more smart features.
   </Body>
-{/if}
+  {#if relationshipInfo && relationshipInfo.length > 0}
-{#if relationshipInfo && relationshipInfo.length > 0}
+    <Table
-  <Table
+      on:click={({ detail }) => openRelationshipModal(detail.from, detail.to)}
-    on:click={({ detail }) => openRelationshipModal(detail.from, detail.to)}
+      schema={relationshipSchema}
-    schema={relationshipSchema}
+      data={relationshipInfo}
-    data={relationshipInfo}
+      allowEditColumns={false}
-    allowEditColumns={false}
+      allowEditRows={false}
-    allowEditRows={false}
+      allowSelectRows={false}
-    allowSelectRows={false}
+    />
-  />
+  {:else}
-{:else}
+    <Body size="S"><i>No relationships configured.</i></Body>
-  <Body size="S"><i>No relationships configured.</i></Body>
+  {/if}
 {/if}
 
 <style>

packages/builder/src/components/backend/DatasourceNavigator/TableIntegrationMenu/rest/auth/RestAuthenticationBuilder.svelte

@@ -58,7 +58,7 @@
     />
   {/if}
   <div>
-    <ActionButton on:click={() => openConfigModal()} con="Add"
+    <ActionButton on:click={() => openConfigModal()} icon="Add"
       >Add authentication</ActionButton
     >
   </div>

packages/builder/src/components/backend/DatasourceNavigator/modals/DatasourceConfigModal.svelte

@@ -5,22 +5,28 @@
   import { IntegrationNames } from "constants/backend"
   import cloneDeep from "lodash/cloneDeepWith"
   import { saveDatasource as save } from "builderStore/datasource"
+  import { onMount } from "svelte"
 
   export let integration
   export let modal
 
   // kill the reference so the input isn't saved
   let datasource = cloneDeep(integration)
+  let skipFetch = false
 
   async function saveDatasource() {
     try {
-      const resp = await save(datasource)
+      const resp = await save(datasource, skipFetch)
       $goto(`./datasource/${resp._id}`)
       notifications.success(`Datasource updated successfully.`)
     } catch (err) {
       notifications.error(`Error saving datasource: ${err}`)
     }
   }
+
+  onMount(() => {
+    skipFetch = false
+  })
 </script>
 
 <ModalContent
@@ -28,9 +34,16 @@
   onConfirm={() => saveDatasource()}
   onCancel={() => modal.show()}
   confirmText={datasource.plus
-    ? "Fetch tables from database"
+    ? "Save and fetch tables"
     : "Save and continue to query"}
   cancelText="Back"
+  showSecondaryButton={datasource.plus}
+  secondaryButtonText={datasource.plus ? "Skip table fetch" : undefined}
+  secondaryAction={() => {
+    skipFetch = true
+    saveDatasource()
+    return true
+  }}
   size="L"
 >
   <Layout noPadding>

packages/builder/src/components/backend/DatasourceNavigator/popovers/EditQueryPopover.svelte

@@ -5,22 +5,29 @@
   import { datasources, queries } from "stores/backend"
 
   export let query
+  export let onClickQuery
 
   let confirmDeleteDialog
 
   async function deleteQuery() {
     const wasSelectedQuery = $queries.selected
-    const selectedDatasource = $datasources.selected
+    // need to calculate this before the query is deleted
+    const navigateToDatasource = wasSelectedQuery === query._id
+
     await queries.delete(query)
-    if (wasSelectedQuery === query._id) {
+    await datasources.fetch()
-      $goto(`./datasource/${selectedDatasource}`)
+
+    if (navigateToDatasource) {
+      await datasources.select(query.datasourceId)
+      $goto(`./datasource/${query.datasourceId}`)
     }
     notifications.success("Query deleted")
   }
 
   async function duplicateQuery() {
     try {
-      await queries.duplicate(query)
+      const newQuery = await queries.duplicate(query)
+      onClickQuery(newQuery)
     } catch (e) {
       notifications.error(e.message)
     }

packages/builder/src/components/common/NavItem.svelte

@@ -133,5 +133,6 @@
   .iconText {
     margin-top: 1px;
     font-size: var(--spectrum-global-dimension-font-size-50);
+    flex: 0 0 34px;
   }
 </style>

packages/builder/src/components/integration/KeyValueBuilder.svelte

@@ -31,7 +31,10 @@
   export let menuItems
   export let showMenu = false
 
-  let fields = Object.entries(object).map(([name, value]) => ({ name, value }))
+  let fields = Object.entries(object || {}).map(([name, value]) => ({
+    name,
+    value,
+  }))
   let fieldActivity = buildFieldActivity(activity)
 
   $: object = fields.reduce(

packages/builder/src/constants/backend/index.js

@@ -219,3 +219,13 @@ export const RestBodyTypes = [
   { name: "raw (XML)", value: "xml" },
   { name: "raw (Text)", value: "text" },
 ]
+
+export const PaginationTypes = [
+  { label: "Page number based", value: "page" },
+  { label: "Cursor based", value: "cursor" },
+]
+
+export const PaginationLocations = [
+  { label: "Query parameters", value: "query" },
+  { label: "Request body", value: "body" },
+]