Merge branch 'develop' into api-create-app

docs/DEV-SETUP-DEBIAN.md

@@ -1,12 +1,15 @@
 ## Dev Environment on Debian 11
 
-### Install Node
+### Install NVM & Node 14
+NVM documentation: https://github.com/nvm-sh/nvm#installing-and-updating
 
-Budibase requires a recent version of node (14+):
+Install NVM
 ```
-curl -sL https://deb.nodesource.com/setup_16.x | sudo bash -
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
-apt -y install nodejs
+```
-node -v
+Install Node 14
+```
+nvm install 14
 ```
 
 ### Install npm requirements
@@ -31,7 +34,7 @@ This setup process was tested on Debian 11 (bullseye) with version numbers show
 
 - Docker: 20.10.5
 - Docker-Compose: 1.29.2
-- Node: v16.15.1
+- Node: v14.20.1
 - Yarn: 1.22.19
 - Lerna: 5.1.4
 

docs/DEV-SETUP-MACOSX.md

@@ -11,7 +11,7 @@ through brew.
 
 ### Install Node
 
-Budibase requires a recent version of node (14+):
+Budibase requires a recent version of node 14:
 ```
 brew install node npm
 node -v
@@ -38,7 +38,7 @@ This setup process was tested on Mac OSX 12 (Monterey) with version numbers show
 
 - Docker: 20.10.14
 - Docker-Compose: 2.6.0
-- Node: 18.3.0
+- Node: 14.20.1
 - Yarn: 1.22.19
 - Lerna: 5.1.4
 
@@ -60,3 +60,6 @@ http://127.0.0.1:10000/builder/admin
 
 | **NOTE**: If you are working on a M1 Apple Silicon, you will need to uncomment `# platform: linux/amd64` line in
 [hosting/docker-compose-dev.yaml](../hosting/docker-compose.dev.yaml)
+
+### Troubleshooting
+If there are errors with the `yarn setup` command, you can try installing nvm and node 14. This is the same as the instructions for Debian 11. 

docs/DEV-SETUP-WINDOWS.md

@@ -0,0 +1,81 @@
+## Dev Environment on Windows 10/11 (WSL2)
+
+
+### Install WSL with Ubuntu LTS
+
+Enable WSL 2 on Windows 10/11 for docker support.
+```
+wsl --set-default-version 2
+```
+Install Ubuntu LTS.
+```
+wsl --install Ubuntu
+```
+
+Or follow the instruction here:
+https://learn.microsoft.com/en-us/windows/wsl/install
+
+### Install Docker in windows
+Download the installer from docker and install it.
+
+Check this url for more detailed instructions:
+https://docs.docker.com/desktop/install/windows-install/
+
+You should follow the next steps from within the Ubuntu terminal.
+
+### Install NVM & Node 14
+NVM documentation: https://github.com/nvm-sh/nvm#installing-and-updating
+
+Install NVM
+```
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
+```
+Install Node 14
+```
+nvm install 14
+```
+
+
+### Install npm requirements
+
+```
+npm install -g yarn jest lerna
+```
+
+### Clone the repo
+```
+git clone https://github.com/Budibase/budibase.git
+```
+
+### Check Versions
+
+This setup process was tested on Windows 11 with version numbers show below. Your mileage may vary using anything else.
+
+- Docker: 20.10.7
+- Docker-Compose: 2.10.2
+- Node: v14.20.1
+- Yarn: 1.22.19
+- Lerna: 5.5.4
+
+### Build
+
+```
+cd budibase
+yarn setup
+```
+The yarn setup command runs several build steps i.e.
+```
+node ./hosting/scripts/setup.js && yarn && yarn bootstrap && yarn build && yarn dev
+```
+So this command will actually run the application in dev mode. It creates .env files under `./packages/server` and `./packages/worker` and runs docker containers for each service via docker-compose.
+
+The dev version will be available on port 10000 i.e.
+
+http://127.0.0.1:10000/builder/admin
+
+### Working with the code
+Here are the instructions to work on the application from within Visual Studio Code (in Windows) through the WSL. All the commands and files are within the Ubuntu system and it should run as if you were working on a Linux machine.
+
+https://code.visualstudio.com/docs/remote/wsl
+
+Note you will be able to run the application from within the WSL terminal and you will be able to access the application from the a browser in Windows.

hosting/single/Dockerfile

@@ -19,8 +19,8 @@ ADD packages/worker .
 RUN node /pinVersions.js && yarn && yarn build && /cleanup.sh
 
 FROM couchdb:3.2.1
-# TARGETARCH can be amd64 or arm e.g. docker build --build-arg TARGETARCH=amd64
+ARG TARGETARCH
-ARG TARGETARCH=amd64
+ENV TARGETARCH $TARGETARCH
 #TARGETBUILD can be set to single (for single docker image) or aas (for azure app service)
 # e.g. docker build --build-arg TARGETBUILD=aas ....
 ARG TARGETBUILD=single

lerna.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "npmClient": "yarn",
   "packages": [
     "packages/*"

packages/backend-core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/backend-core",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase backend core libraries used in server and worker",
   "main": "dist/src/index.js",
   "types": "dist/src/index.d.ts",
@@ -20,7 +20,7 @@
     "test:watch": "jest --watchAll"
   },
   "dependencies": {
-    "@budibase/types": "2.0.14-alpha.4",
+    "@budibase/types": "2.0.24-alpha.0",
     "@shopify/jest-koa-mocks": "5.0.1",
     "@techpass/passport-openidconnect": "0.3.2",
     "aws-sdk": "2.1030.0",

packages/bbui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/bbui",
   "description": "A UI solution used in the different Budibase projects.",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "license": "MPL-2.0",
   "svelte": "src/index.js",
   "module": "dist/bbui.es.js",
@@ -38,7 +38,7 @@
   ],
   "dependencies": {
     "@adobe/spectrum-css-workflow-icons": "^1.2.1",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
     "@spectrum-css/actionbutton": "^1.0.1",
     "@spectrum-css/actiongroup": "^1.0.1",
     "@spectrum-css/avatar": "^3.0.2",

packages/builder/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/builder",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "license": "GPL-3.0",
   "private": true,
   "scripts": {
@@ -71,10 +71,10 @@
     }
   },
   "dependencies": {
-    "@budibase/bbui": "2.0.14-alpha.4",
+    "@budibase/bbui": "2.0.24-alpha.0",
-    "@budibase/client": "2.0.14-alpha.4",
+    "@budibase/client": "2.0.24-alpha.0",
-    "@budibase/frontend-core": "2.0.14-alpha.4",
+    "@budibase/frontend-core": "2.0.24-alpha.0",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
     "@sentry/browser": "5.19.1",
     "@spectrum-css/page": "^3.0.1",
     "@spectrum-css/vars": "^3.0.1",

packages/builder/src/builderStore/dataBinding.js

@@ -396,7 +396,6 @@ export const getUserBindings = () => {
 
   bindings = keys.reduce((acc, key) => {
     const fieldSchema = schema[key]
-    if (fieldSchema.type !== "link") {
     acc.push({
       type: "context",
       runtimeBinding: `${safeUser}.${makePropSafe(key)}`,
@@ -408,7 +407,6 @@ export const getUserBindings = () => {
       category: "Current User",
       icon: "User",
     })
-    }
     return acc
   }, [])
 

packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/cli",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase CLI, for developers, self hosting and migrations.",
   "main": "src/index.js",
   "bin": {
@@ -26,9 +26,9 @@
     "outputPath": "build"
   },
   "dependencies": {
-    "@budibase/backend-core": "2.0.14-alpha.4",
+    "@budibase/backend-core": "2.0.24-alpha.0",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
-    "@budibase/types": "2.0.14-alpha.4",
+    "@budibase/types": "2.0.24-alpha.0",
     "axios": "0.21.2",
     "chalk": "4.1.0",
     "cli-progress": "3.11.2",

packages/client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/client",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "license": "MPL-2.0",
   "module": "dist/budibase-client.js",
   "main": "dist/budibase-client.js",
@@ -19,9 +19,9 @@
     "dev:builder": "rollup -cw"
   },
   "dependencies": {
-    "@budibase/bbui": "2.0.14-alpha.4",
+    "@budibase/bbui": "2.0.24-alpha.0",
-    "@budibase/frontend-core": "2.0.14-alpha.4",
+    "@budibase/frontend-core": "2.0.24-alpha.0",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
     "@spectrum-css/button": "^3.0.3",
     "@spectrum-css/card": "^3.0.3",
     "@spectrum-css/divider": "^1.0.3",

packages/client/src/components/ClientApp.svelte

@@ -16,7 +16,6 @@
     themeStore,
     appStore,
     devToolsStore,
-    environmentStore,
   } from "stores"
   import NotificationDisplay from "components/overlay/NotificationDisplay.svelte"
   import ConfirmationDisplay from "components/overlay/ConfirmationDisplay.svelte"
@@ -48,8 +47,6 @@
     !$builderStore.inBuilder &&
     $devToolsStore.enabled &&
     !$routeStore.queryParams?.peek
-  $: objectStoreUrl = $environmentStore.cloud ? "https://cdn.budi.live" : ""
-  $: pluginsUrl = `${objectStoreUrl}/plugins`
 
   // Handle no matching route
   $: {
@@ -95,8 +92,7 @@
 <svelte:head>
   {#if $builderStore.usedPlugins?.length}
     {#each $builderStore.usedPlugins as plugin (plugin.hash)}
-      <script
+      <script src={`${plugin.jsUrl}?r=${plugin.hash || ""}`}></script>
-        src={`${pluginsUrl}/${plugin.jsUrl}?r=${plugin.hash || ""}`}></script>
     {/each}
   {/if}
 </svelte:head>

packages/client/src/components/app/charts/ApexOptionsBuilder.js

@@ -1,10 +1,11 @@
 export class ApexOptionsBuilder {
-  formatters = {
+  constructor() {
+    this.formatters = {
       ["Default"]: val => (isNaN(val) ? val : Math.round(val * 100) / 100),
       ["Thousands"]: val => `${Math.round(val / 1000)}K`,
       ["Millions"]: val => `${Math.round(val / 1000000)}M`,
     }
-  options = {
+    this.options = {
       series: [],
       legend: {
         show: false,
@@ -33,6 +34,7 @@ export class ApexOptionsBuilder {
         },
       },
     }
+  }
 
   setOption(path, value) {
     if (value == null || value === "") {

packages/frontend-core/package.json

@@ -1,12 +1,12 @@
 {
   "name": "@budibase/frontend-core",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase frontend core libraries used in builder and client",
   "author": "Budibase",
   "license": "MPL-2.0",
   "svelte": "src/index.js",
   "dependencies": {
-    "@budibase/bbui": "2.0.14-alpha.4",
+    "@budibase/bbui": "2.0.24-alpha.0",
     "lodash": "^4.17.21",
     "svelte": "^3.46.2"
   }

packages/frontend-core/src/fetch/DataFetch.js

@@ -14,18 +14,23 @@ import { convertJSONSchemaToTableSchema } from "../utils/json"
  * For other types of datasource, this class is overridden and extended.
  */
 export default class DataFetch {
+  /**
+   * Constructs a new DataFetch instance.
+   * @param opts the fetch options
+   */
+  constructor(opts) {
     // API client
-  API = null
+    this.API = null
 
     // Feature flags
-  featureStore = writable({
+    this.featureStore = writable({
       supportsSearch: false,
       supportsSort: false,
       supportsPagination: false,
     })
 
     // Config
-  options = {
+    this.options = {
       datasource: null,
       limit: 10,
 
@@ -43,7 +48,7 @@ export default class DataFetch {
     }
 
     // State of the fetch
-  store = writable({
+    this.store = writable({
       rows: [],
       info: null,
       schema: null,
@@ -55,11 +60,6 @@ export default class DataFetch {
       cursors: [],
     })
 
-  /**
-   * Constructs a new DataFetch instance.
-   * @param opts the fetch options
-   */
-  constructor(opts) {
     // Merge options with their default values
     this.API = opts?.API
     this.options = {

packages/frontend-core/src/utils/lucene.js

@@ -121,7 +121,12 @@ export const buildLuceneQuery = filter => {
         query.allOr = true
         return
       }
-      if (type === "datetime" && !isHbs) {
+      if (
+        type === "datetime" &&
+        !isHbs &&
+        operator !== "empty" &&
+        operator !== "notEmpty"
+      ) {
         // Ensure date value is a valid date and parse into correct format
         if (!value) {
           return

packages/sdk/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/sdk",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase Public API SDK",
   "author": "Budibase",
   "license": "MPL-2.0",

packages/server/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/server",
   "email": "hi@budibase.com",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase Web Server",
   "main": "src/index.ts",
   "repository": {
@@ -77,11 +77,11 @@
   "license": "GPL-3.0",
   "dependencies": {
     "@apidevtools/swagger-parser": "10.0.3",
-    "@budibase/backend-core": "2.0.14-alpha.4",
+    "@budibase/backend-core": "2.0.24-alpha.0",
-    "@budibase/client": "2.0.14-alpha.4",
+    "@budibase/client": "2.0.24-alpha.0",
-    "@budibase/pro": "2.0.14-alpha.4",
+    "@budibase/pro": "2.0.24-alpha.0",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
-    "@budibase/types": "2.0.14-alpha.4",
+    "@budibase/types": "2.0.24-alpha.0",
     "@bull-board/api": "3.7.0",
     "@bull-board/koa": "3.9.4",
     "@elastic/elasticsearch": "7.10.0",

packages/server/src/api/controllers/application.ts

@@ -50,6 +50,7 @@ import { errors, events, migrations } from "@budibase/backend-core"
 import { App, Layout, Screen, MigrationType } from "@budibase/types"
 import { BASE_LAYOUT_PROP_IDS } from "../../constants/layouts"
 import { groups } from "@budibase/pro"
+import { enrichPluginURLs } from "../../utilities/plugins"
 
 const URL_REGEX_SLASH = /\/|\\/g
 
@@ -208,10 +209,13 @@ export const fetchAppDefinition = async (ctx: any) => {
 
 export const fetchAppPackage = async (ctx: any) => {
   const db = context.getAppDB()
-  const application = await db.get(DocumentType.APP_METADATA)
+  let application = await db.get(DocumentType.APP_METADATA)
   const layouts = await getLayouts()
   let screens = await getScreens()
 
+  // Enrich plugin URLs
+  application.usedPlugins = enrichPluginURLs(application.usedPlugins)
+
   // Only filter screens if the user is not a builder
   if (!(ctx.user.builder && ctx.user.builder.global)) {
     const userRoleId = getUserRoleId(ctx)

packages/server/src/api/controllers/datasource.js

@@ -68,6 +68,7 @@ exports.buildSchemaFromDb = async function (ctx) {
     datasource.entities = tables
   }
 
+  setDefaultDisplayColumns(datasource)
   const dbResp = await db.put(datasource)
   datasource._rev = dbResp.rev
 
@@ -78,6 +79,24 @@ exports.buildSchemaFromDb = async function (ctx) {
   ctx.body = response
 }
 
+/**
+ * Make sure all datasource entities have a display name selected
+ */
+const setDefaultDisplayColumns = datasource => {
+  //
+  for (let entity of Object.values(datasource.entities)) {
+    if (entity.primaryDisplay) {
+      continue
+    }
+    const notAutoColumn = Object.values(entity.schema).find(
+      schema => !schema.autocolumn
+    )
+    if (notAutoColumn) {
+      entity.primaryDisplay = notAutoColumn.name
+    }
+  }
+}
+
 /**
  * Check for variables that have been updated or removed and invalidate them.
  */
@@ -155,6 +174,7 @@ exports.save = async function (ctx) {
     const { tables, error } = await buildSchemaHelper(datasource)
     schemaError = error
     datasource.entities = tables
+    setDefaultDisplayColumns(datasource)
   }
 
   const dbResp = await db.put(datasource)
@@ -238,19 +258,6 @@ const buildSchemaHelper = async datasource => {
   const connector = new Connector(datasource.config)
   await connector.buildSchema(datasource._id, datasource.entities)
 
-  // make sure they all have a display name selected
-  for (let entity of Object.values(datasource.entities ?? {})) {
-    if (entity.primaryDisplay) {
-      continue
-    }
-    const notAutoColumn = Object.values(entity.schema).find(
-      schema => !schema.autocolumn
-    )
-    if (notAutoColumn) {
-      entity.primaryDisplay = notAutoColumn.name
-    }
-  }
-
   const errors = connector.schemaErrors
   let error = null
   if (errors && Object.keys(errors).length > 0) {

packages/server/src/api/controllers/public/rows.ts

@@ -52,14 +52,19 @@ export async function read(ctx: any, next: any) {
 }
 
 export async function update(ctx: any, next: any) {
-  ctx.request.body = await addRev(fixRow(ctx.request.body, ctx.params))
+  const { tableId } = ctx.params
+  ctx.request.body = await addRev(fixRow(ctx.request.body, ctx.params), tableId)
   await rowController.save(ctx)
   await next()
 }
 
 export async function destroy(ctx: any, next: any) {
+  const { tableId } = ctx.params
   // set the body as expected, with the _id and _rev fields
-  ctx.request.body = await addRev(fixRow({ _id: ctx.params.rowId }, ctx.params))
+  ctx.request.body = await addRev(
+    fixRow({ _id: ctx.params.rowId }, ctx.params),
+    tableId
+  )
   await rowController.destroy(ctx)
   // destroy controller doesn't currently return the row as the body, need to adjust this
   // in the public API to be correct

packages/server/src/api/controllers/public/utils.ts

@@ -22,7 +22,7 @@ export async function addRev(
 }
 
 /**
- * Performs a case insensitive search on the provided documents, using the
+ * Performs a case in-sensitive search on the provided documents, using the
  * provided key and value. This will be a string based search, using the
  * startsWith function.
  */

packages/server/src/api/controllers/query/index.ts

@@ -240,6 +240,10 @@ async function execute(
     const { rows, pagination, extra } = await quotas.addQuery(runFn, {
       datasourceId: datasource._id,
     })
+    // remove the raw from execution incase transformer being used to hide data
+    if (extra?.raw) {
+      delete extra.raw
+    }
     if (opts && opts.rowsOnly) {
       ctx.body = rows
     } else {

packages/server/src/api/controllers/row/internalSearch.js

@@ -145,7 +145,7 @@ class QueryBuilder {
    * @param options The preprocess options
    * @returns {string|*}
    */
-  preprocess(value, { escape, lowercase, wrap } = {}) {
+  preprocess(value, { escape, lowercase, wrap, type } = {}) {
     const hasVersion = !!this.version
     // Determine if type needs wrapped
     const originalType = typeof value
@@ -157,8 +157,11 @@ class QueryBuilder {
     if (escape && originalType === "string") {
       value = `${value}`.replace(/[ #+\-&|!(){}\]^"~*?:\\]/g, "\\$&")
     }
+
     // Wrap in quotes
-    if (hasVersion && wrap) {
+    if (originalType === "string" && !isNaN(value) && !type) {
+      value = `"${value}"`
+    } else if (hasVersion && wrap) {
       value = originalType === "number" ? value : `"${value}"`
     }
     return value
@@ -253,6 +256,7 @@ class QueryBuilder {
         value = builder.preprocess(value, {
           escape: true,
           lowercase: true,
+          type: "string",
         })
         return `${key}:${value}*`
       })
@@ -281,6 +285,7 @@ class QueryBuilder {
         value = builder.preprocess(value, {
           escape: true,
           lowercase: true,
+          type: "fuzzy",
         })
         return `${key}:${value}~`
       })

packages/server/src/api/controllers/static/index.ts

@@ -1,3 +1,5 @@
+import { enrichPluginURLs } from "../../../utilities/plugins"
+
 require("svelte/register")
 
 const send = require("koa-send")
@@ -107,12 +109,13 @@ export const serveApp = async function (ctx: any) {
 
   if (!env.isJest()) {
     const App = require("./templates/BudibaseApp.svelte").default
+    const plugins = enrichPluginURLs(appInfo.usedPlugins)
     const { head, html, css } = App.render({
       title: appInfo.name,
       production: env.isProd(),
       appId,
       clientLibPath: clientLibraryPath(appId, appInfo.version, ctx),
-      usedPlugins: appInfo.usedPlugins,
+      usedPlugins: plugins,
     })
 
     const appHbs = loadHandlebarsFile(`${__dirname}/templates/app.hbs`)

packages/server/src/api/controllers/static/templates/BudibaseApp.svelte

@@ -88,9 +88,7 @@
   <!-- But before loadBudibase is called -->
   {#if usedPlugins?.length}
     {#each usedPlugins as plugin}
-      <script
+      <script type="application/javascript" src={plugin.jsUrl}></script>
-        type="application/javascript"
-        src={`/plugins/${plugin.jsUrl}`}></script>
     {/each}
   {/if}
   <script type="application/javascript">

packages/server/src/api/controllers/table/internal.ts

@@ -14,8 +14,10 @@ import {
   fixAutoColumnSubType,
 } from "../../../utilities/rowProcessor"
 import { runStaticFormulaChecks } from "./bulkFormula"
-import { Table } from "../../../definitions/common"
+import { Table } from "@budibase/types"
 import { quotas } from "@budibase/pro"
+import { isEqual } from "lodash"
+import { cloneDeep } from "lodash/fp"
 
 function checkAutoColumns(table: Table, oldTable: Table) {
   if (!table.schema) {
@@ -123,10 +125,16 @@ export async function save(ctx: any) {
   if (updatedRows && updatedRows.length !== 0) {
     await db.bulkDocs(updatedRows)
   }
-  const result = await db.put(tableToSave)
+  let result = await db.put(tableToSave)
   tableToSave._rev = result.rev
+  const savedTable = cloneDeep(tableToSave)
 
   tableToSave = await tableSaveFunctions.after(tableToSave)
+  // the table may be updated as part of the table save after functionality - need to write it
+  if (!isEqual(savedTable, tableToSave)) {
+    result = await db.put(tableToSave)
+    tableToSave._rev = result.rev
+  }
   // has to run after, make sure it has _id
   await runStaticFormulaChecks(tableToSave, { oldTable, deletion: null })
   return tableToSave

packages/server/src/api/controllers/table/utils.ts

@@ -247,7 +247,7 @@ class TableSaveFunctions {
   // after saving
   async after(table: any) {
     table = await handleSearchIndexes(table)
-    await handleDataImport(this.user, table, this.dataImport)
+    table = await handleDataImport(this.user, table, this.dataImport)
     return table
   }
 

packages/server/src/api/routes/tests/internalSearch.spec.js

@@ -173,4 +173,24 @@ describe("internal search", () => {
     }, PARAMS)
     checkLucene(response, `*:* AND NOT column:(a AND b AND c)`, PARAMS)
   })
+
+  it("test equal without version query", async () => {
+    PARAMS.version = null
+    const response = await search.paginatedSearch({
+      equal: {
+        "column": "1",
+      }
+    }, PARAMS)
+    
+    const query = response.rows[0].query
+    const json = JSON.parse(query)
+    if (PARAMS.sort) {
+      expect(json.sort).toBe(`${PARAMS.sort}<${PARAMS.sortType}>`)
+    }
+    if (PARAMS.bookmark) {
+      expect(json.bookmark).toBe(PARAMS.bookmark)
+    }
+    expect(json.include_docs).toBe(true)
+    expect(json.q).toBe(`(*:* AND column:"1") AND tableId:${PARAMS.tableId}`)
+  })
 })

packages/server/src/threads/automation.ts

@@ -252,7 +252,7 @@ class Orchestrator {
     let loopStepNumber: any = undefined
     let loopSteps: LoopStep[] | undefined = []
     let metadata
-
+    let wasLoopStep = false
     // check if this is a recurring automation,
     if (isProdAppID(this._appId) && isRecurring(automation)) {
       metadata = await this.getMetadata()
@@ -267,6 +267,7 @@ class Orchestrator {
       let input,
         iterations = 1,
         iterationCount = 0
+
       if (step.stepId === LOOP_STEP_ID) {
         loopStep = step
         loopStepNumber = stepCount
@@ -277,10 +278,8 @@ class Orchestrator {
         input = await processObject(loopStep.inputs, this._context)
         iterations = getLoopIterations(loopStep as LoopStep, input)
       }
-
       for (let index = 0; index < iterations; index++) {
         let originalStepInput = cloneDeep(step.inputs)
-
         // Handle if the user has set a max iteration count or if it reaches the max limit set by us
         if (loopStep && input.binding) {
           let newInput = await processObject(
@@ -313,7 +312,6 @@ class Orchestrator {
           } else {
             item = loopStep.inputs.binding
           }
-
           this._context.steps[loopStepNumber] = {
             currentItem: item[index],
           }
@@ -331,6 +329,16 @@ class Orchestrator {
                       innerValue,
                       `steps.${loopStepNumber}`
                     )
+                } else if (typeof value === "object") {
+                  for (let [innerObject, innerValue] of Object.entries(
+                    originalStepInput[key][innerKey]
+                  )) {
+                    originalStepInput[key][innerKey][innerObject] =
+                      automationUtils.substituteLoopStep(
+                        innerValue,
+                        `steps.${loopStepNumber}`
+                      )
+                  }
                 }
               }
             } else {
@@ -386,6 +394,7 @@ class Orchestrator {
         let stepFn = await this.getStepFunctionality(step.stepId)
         let inputs = await processObject(originalStepInput, this._context)
         inputs = automationUtils.cleanInputValues(inputs, step.schema.inputs)
+
         try {
           // appId is always passed
           const outputs = await stepFn({
@@ -394,6 +403,7 @@ class Orchestrator {
             emitter: this._emitter,
             context: this._context,
           })
+
           this._context.steps[stepCount] = outputs
           // if filter causes us to stop execution don't break the loop, set a var
           // so that we can finish iterating through the steps and record that it stopped
@@ -419,6 +429,7 @@ class Orchestrator {
           console.error(`Automation error - ${step.stepId} - ${err}`)
           return err
         }
+
         if (loopStep) {
           iterationCount++
           if (index === iterations - 1) {
@@ -429,6 +440,13 @@ class Orchestrator {
         }
       }
 
+      // Delete the step after the loop step as it's irrelevant, since information is included
+      // in the loop step
+      if (wasLoopStep) {
+        this._context.steps.splice(loopStepNumber + 1, 1)
+        wasLoopStep = false
+      }
+
       if (loopSteps && loopSteps.length) {
         let tempOutput = {
           success: true,
@@ -441,9 +459,10 @@ class Orchestrator {
           outputs: tempOutput,
           inputs: step.inputs,
         })
+        this._context.steps[loopStepNumber] = tempOutput
 
-        this._context.steps.splice(loopStepNumber, 0, tempOutput)
         loopSteps = undefined
+        wasLoopStep = true
       }
     }
 

packages/server/src/utilities/plugins.js

@@ -0,0 +1,21 @@
+const env = require("../environment")
+const { plugins: ProPlugins } = require("@budibase/pro")
+const { objectStore } = require("@budibase/backend-core")
+
+exports.enrichPluginURLs = plugins => {
+  if (!plugins || !plugins.length) {
+    return []
+  }
+  return plugins.map(plugin => {
+    const cloud = !env.SELF_HOSTED
+    const bucket = objectStore.ObjectStoreBuckets.PLUGINS
+    const jsFileName = "plugin.min.js"
+
+    // In self host we need to prefix the path, as the bucket name is not part
+    // of the bucket path. In cloud, it's already part of the bucket path.
+    let jsUrl = cloud ? "https://cdn.budi.live/" : `/${bucket}/`
+    jsUrl += ProPlugins.getBucketPath(plugin.name)
+    jsUrl += jsFileName
+    return { ...plugin, jsUrl }
+  })
+}

packages/server/yarn.lock

@@ -1094,12 +1094,12 @@
   resolved "https://registry.yarnpkg.com/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz#75a2e8b51cb758a7553d6804a5932d7aace75c39"
   integrity sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==
 
-"@budibase/backend-core@2.0.14-alpha.4":
+"@budibase/backend-core@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-2.0.14-alpha.4.tgz#213446a9e04385fb38ab8785742cdb8b4eaf2cac"
+  resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-2.0.24-alpha.0.tgz#db53e562cd40aa21783f6f4d6eaf880ec8225a8f"
-  integrity sha512-ma1Ipst4AQVi4sx+ULs3bX9xT4eHqavCvM/BXtZzV23SOTdjJTi9wX2KWbIvAUT5xo2NgN0uRf5zf7B6CBRuXw==
+  integrity sha512-o8Ooa5Ga3hcbovoO+Fjyp++TcJmt8v83Y7GfurLznjAc5lakdEe8RWKR9JV7xqenXpZ49ZwWYTSOLwxFBiJuvw==
   dependencies:
-    "@budibase/types" "2.0.14-alpha.4"
+    "@budibase/types" "2.0.24-alpha.0"
     "@shopify/jest-koa-mocks" "5.0.1"
     "@techpass/passport-openidconnect" "0.3.2"
     aws-sdk "2.1030.0"
@@ -1180,13 +1180,13 @@
     svelte-flatpickr "^3.2.3"
     svelte-portal "^1.0.0"
 
-"@budibase/pro@2.0.14-alpha.4":
+"@budibase/pro@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-2.0.14-alpha.4.tgz#9796271365f4ffb875655f86df84a2b4111b793a"
+  resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-2.0.24-alpha.0.tgz#c7c70f431d9d17603d965b353dbb7c4956b6f7a0"
-  integrity sha512-I5QIv04hfoUPEYHPkCD5L/5WU7cJurmfhjy62ZMTNbJzFzhIGDzQlOjM0MnQO5kJvGv1NWl2PQvSty1rYzXGsw==
+  integrity sha512-bp+m9mXEonvxiLo4s9Gt4vhHuVrderFB4vfCISMRcIpZTo/CPVkMTOEM7C4liZyXOiLB9hh4Y+vQY1T6ilIr3Q==
   dependencies:
-    "@budibase/backend-core" "2.0.14-alpha.4"
+    "@budibase/backend-core" "2.0.24-alpha.0"
-    "@budibase/types" "2.0.14-alpha.4"
+    "@budibase/types" "2.0.24-alpha.0"
     "@koa/router" "8.0.8"
     joi "17.6.0"
     node-fetch "^2.6.1"
@@ -1209,10 +1209,10 @@
     svelte-apexcharts "^1.0.2"
     svelte-flatpickr "^3.1.0"
 
-"@budibase/types@2.0.14-alpha.4":
+"@budibase/types@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/types/-/types-2.0.14-alpha.4.tgz#37b1ff362d2551fd978e8c8153a4cec357eda0b0"
+  resolved "https://registry.yarnpkg.com/@budibase/types/-/types-2.0.24-alpha.0.tgz#d5ff01dc4a6f1d584077b1808c0a6c8cffbd1eea"
-  integrity sha512-l4dfMh5it1N56nc+jCtdXklrZA1cMt6WSVBoKCXBedZGpguJr18Wo/cfn1YWryuUMuSlpFVDTrlQQXCZ+ku92g==
+  integrity sha512-eiTumTQp6d9+bizhImqK+OJ4g4b9IuUTCOdNp7K2rwu0v94xxLuQEtJUA5MKiqClx7Uu27IlybeQmP+oM99gTg==
 
 "@bull-board/api@3.7.0":
   version "3.7.0"

packages/string-templates/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/string-templates",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Handlebars wrapper for Budibase templating.",
   "main": "src/index.cjs",
   "module": "dist/bundle.mjs",

packages/types/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/types",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase types",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

packages/worker/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/worker",
   "email": "hi@budibase.com",
-  "version": "2.0.14-alpha.4",
+  "version": "2.0.24-alpha.0",
   "description": "Budibase background service",
   "main": "src/index.ts",
   "repository": {
@@ -36,10 +36,10 @@
   "author": "Budibase",
   "license": "GPL-3.0",
   "dependencies": {
-    "@budibase/backend-core": "2.0.14-alpha.4",
+    "@budibase/backend-core": "2.0.24-alpha.0",
-    "@budibase/pro": "2.0.14-alpha.4",
+    "@budibase/pro": "2.0.24-alpha.0",
-    "@budibase/string-templates": "2.0.14-alpha.4",
+    "@budibase/string-templates": "2.0.24-alpha.0",
-    "@budibase/types": "2.0.14-alpha.4",
+    "@budibase/types": "2.0.24-alpha.0",
     "@koa/router": "8.0.8",
     "@sentry/node": "6.17.7",
     "@techpass/passport-openidconnect": "0.3.2",

packages/worker/yarn.lock

@@ -291,12 +291,12 @@
   resolved "https://registry.yarnpkg.com/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz#75a2e8b51cb758a7553d6804a5932d7aace75c39"
   integrity sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==
 
-"@budibase/backend-core@2.0.14-alpha.4":
+"@budibase/backend-core@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-2.0.14-alpha.4.tgz#213446a9e04385fb38ab8785742cdb8b4eaf2cac"
+  resolved "https://registry.yarnpkg.com/@budibase/backend-core/-/backend-core-2.0.24-alpha.0.tgz#db53e562cd40aa21783f6f4d6eaf880ec8225a8f"
-  integrity sha512-ma1Ipst4AQVi4sx+ULs3bX9xT4eHqavCvM/BXtZzV23SOTdjJTi9wX2KWbIvAUT5xo2NgN0uRf5zf7B6CBRuXw==
+  integrity sha512-o8Ooa5Ga3hcbovoO+Fjyp++TcJmt8v83Y7GfurLznjAc5lakdEe8RWKR9JV7xqenXpZ49ZwWYTSOLwxFBiJuvw==
   dependencies:
-    "@budibase/types" "2.0.14-alpha.4"
+    "@budibase/types" "2.0.24-alpha.0"
     "@shopify/jest-koa-mocks" "5.0.1"
     "@techpass/passport-openidconnect" "0.3.2"
     aws-sdk "2.1030.0"
@@ -327,21 +327,21 @@
     uuid "8.3.2"
     zlib "1.0.5"
 
-"@budibase/pro@2.0.14-alpha.4":
+"@budibase/pro@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-2.0.14-alpha.4.tgz#9796271365f4ffb875655f86df84a2b4111b793a"
+  resolved "https://registry.yarnpkg.com/@budibase/pro/-/pro-2.0.24-alpha.0.tgz#c7c70f431d9d17603d965b353dbb7c4956b6f7a0"
-  integrity sha512-I5QIv04hfoUPEYHPkCD5L/5WU7cJurmfhjy62ZMTNbJzFzhIGDzQlOjM0MnQO5kJvGv1NWl2PQvSty1rYzXGsw==
+  integrity sha512-bp+m9mXEonvxiLo4s9Gt4vhHuVrderFB4vfCISMRcIpZTo/CPVkMTOEM7C4liZyXOiLB9hh4Y+vQY1T6ilIr3Q==
   dependencies:
-    "@budibase/backend-core" "2.0.14-alpha.4"
+    "@budibase/backend-core" "2.0.24-alpha.0"
-    "@budibase/types" "2.0.14-alpha.4"
+    "@budibase/types" "2.0.24-alpha.0"
     "@koa/router" "8.0.8"
     joi "17.6.0"
     node-fetch "^2.6.1"
 
-"@budibase/types@2.0.14-alpha.4":
+"@budibase/types@2.0.24-alpha.0":
-  version "2.0.14-alpha.4"
+  version "2.0.24-alpha.0"
-  resolved "https://registry.yarnpkg.com/@budibase/types/-/types-2.0.14-alpha.4.tgz#37b1ff362d2551fd978e8c8153a4cec357eda0b0"
+  resolved "https://registry.yarnpkg.com/@budibase/types/-/types-2.0.24-alpha.0.tgz#d5ff01dc4a6f1d584077b1808c0a6c8cffbd1eea"
-  integrity sha512-l4dfMh5it1N56nc+jCtdXklrZA1cMt6WSVBoKCXBedZGpguJr18Wo/cfn1YWryuUMuSlpFVDTrlQQXCZ+ku92g==
+  integrity sha512-eiTumTQp6d9+bizhImqK+OJ4g4b9IuUTCOdNp7K2rwu0v94xxLuQEtJUA5MKiqClx7Uu27IlybeQmP+oM99gTg==
 
 "@cspotcode/source-map-consumer@0.8.0":
   version "0.8.0"

scripts/install-minio.sh

@@ -1,8 +1,10 @@
 #!/bin/bash
 if [[ $TARGETARCH == arm* ]] ;
 then
+  echo "INSTALLING ARM64 MINIO"
   wget https://dl.min.io/server/minio/release/linux-arm64/minio
 else
+  echo "INSTALLING AMD64 MINIO"
   wget https://dl.min.io/server/minio/release/linux-amd64/minio
 fi
 chmod +x minio

scripts/pro/install.sh

@@ -18,6 +18,11 @@ git clone https://$PERSONAL_ACCESS_TOKEN@github.com/Budibase/budibase-pro.git
 if [[ -d "budibase-pro" ]]; then
   cd budibase-pro
 
+  if [[ -z "${BRANCH}" ]]; then
+    echo Using GITHUB_REF_NAME: $GITHUB_REF_NAME
+    export BRANCH=$GITHUB_REF_NAME
+  fi
+
   # Try to checkout the matching pro branch
   git checkout $BRANCH