Merge branch 'develop' of github.com:Budibase/budibase into new-design-ui

.github/workflows/README.md

@@ -6,7 +6,7 @@ Welcome to the budibase CI pipelines directory. This document details what each
 ## All CI Pipelines
 
 ### Note
-- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs.
+- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. The exception to this case is the `deploy-release` job which requires the develop branch. 
 
 ### Standard CI Build Job (budibase_ci.yml)
 Triggers:
@@ -116,3 +116,74 @@ This job is responsible for deploying to our production, cloud kubernetes enviro
 - Kick off cloud deploy job
 - Ensure you are running off master
 - Enter the version number of the last known good version of budibase. For example `1.0.0`
+
+## Pro
+
+### Installing Pro
+
+The pro package is always installed from source in our CI jobs. 
+
+This is done to prevent pro needing to be published prior to CI runs in budiabse. This is required for two reasons:
+- To reduce developer need to manually bump versions, i.e: 
+  - release pro, bump pro dep in budibase, now ci can run successfully
+- The cyclic dependency on backend-core, i.e:
+  - pro depends on backend-core
+  - server depends on pro
+  - backend-core lives in the monorepo, so it can't be released independently to be used in pro
+  - therefore the only option is to pull pro from source and release it as a part of the monorepo release, as if it were a mono package
+
+The install is performed using the same steps as local development, via the `yarn bootstrap` command, see the [Contributing Guide#Pro](../CONTRIBUTING.md#pro)
+
+The branch to install pro from can vary depending on ref of the commit that triggered the budibase CI job. This is done to enable branches which have changes in both the monorepo and the pro repo to have their CI pass successfully. 
+
+This is done using the [pro/install.sh](../../scripts/pro/install.sh) script. The script will:
+- Clone pro to it's default branch (`develop`)
+- Check if the clone worked, on forked versions of budibase this will fail due to no access
+  - This is fine as the `yarn` command will install the version from NPM
+  - Community PRs should never touch pro so this will always work
+- Checkout the `BRANCH` argument, if this fails fallback to `BASE_BRANCH`
+  - This enables the more complex case of a feature branch being merged to another feature branch, e.g. 
+    - I am working on a branch `epic/stonks` which exists on budibase and pro. 
+    - I want to merge a change to this branch in budibase from `feature/stonks-ui`, which only exists in budibase
+    - The base branch ensures that `epic/stonks` in pro will still be checked out for the CI run, rather than falling back to `develop`
+- Run `yarn setup` to build and install dependencies
+  - `yarn` 
+  - `yarn bootstrap`
+  - `yarn build`
+    - The will build .ts files, and also update the `main` and `types` of `package.json` to point to `dist` rather than src
+    - The build command will only ever work in CI, it is prevented in local dev
+
+#### `BRANCH` and `BASE_BRANCH` arguments
+These arguments are supplied by the various budibase build and release pipelines
+- `budibase_ci` 
+  - `BRANCH: ${{ github.event.pull_request.head.ref }}` -> The branch being merged
+  - `BASE_BRANCH: ${{ github.event.pull_request.base.ref}}` -> The base branch
+- `release-develop`
+  - `BRANCH: develop` -> always use the `develop` branch in pro
+- `release`
+  - `BRANCH: master` -> always use the `master` branch in pro 
+
+
+### Releasing Pro
+After budibase dependencies have been released we will release the new version of pro to match the release version of budibase dependencies. This is to ensure that we are always keeping the version of `backend-core` in sync in the pro package and in budibase packages. Without this we could run into scenarios where different versions are being used when installed via `yarn` inside the docker images, creating very difficult to debug cases.
+
+Pro is released using the [pro/release.sh](../../scripts/pro/release.sh) script. The script will:
+- Inspect the `VERSION` from the `lerna.json` file in budibase
+- Determine whether to use the `latest` or `develop` tag based on the command argument
+- Go to pro directory
+  - install npm creds
+  - update the version of `backend-core` to be `VERSION`, the version just released by lerna
+  - publish to npm. Uses a `lerna publish` command, pro itself is a mono repo. 
+    - force the version to be the same as `VERSION` to keep pro and budibase in sync
+  - reverts the changes to `main` and `types` in `package.json` that were made by the build step, to point back to source
+  - commit & push: `Prep next development iteration`
+- Go to budibase
+  - Update to the new version of pro in `server` and `worker` so the latest pro version is used in the docker builds
+  - commit & push: `Update pro version to $VERSION`
+
+
+#### `COMMAND` argument
+This argument is supplied by the existing `release` and `release:develop` budibase commands, which invoke the pro release
+- `release` will supply no command and default to use `latest`
+- `release:develop` will supply `develop` 
+

docs/CONTRIBUTING.md

@@ -190,6 +190,21 @@ yarn mode:account
 ```
 ### CI
  An overview of the CI pipelines can be found [here](./workflows/README.md)
+
+### Pro
+
+@budibase/pro is the closed source package that supports licensed features in budibase. By default the package will be pulled from NPM and will not normally need to be touched in local development. If you require to update code inside the pro package it can be cloned to the same root level as budibase, e.g. 
+
+```
+.
+|_ budibase
+|_ budibase-pro
+```
+
+Note that only budibase maintainers will be able to access the pro repo. 
+
+The `yarn bootstrap` command can be used to replace the NPM supplied dependency with the local source aware version. This is achieved using the `yarn link` command. To see specifically how dependencies are linked see [scripts/link-dependencies.sh](../scripts/link-dependencies.sh). The same link script is used to link dependencies to account-portal in local dev. 
+
 ### Troubleshooting
 
 Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above to create a fresh Budibase installation.

lerna.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "npmClient": "yarn",
   "packages": [
     "packages/*"

packages/backend-core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/backend-core",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase backend core libraries used in server and worker",
   "main": "dist/src/index.js",
   "types": "dist/src/index.d.ts",
@@ -20,7 +20,7 @@
     "test:watch": "jest --watchAll"
   },
   "dependencies": {
-    "@budibase/types": "^1.0.207-alpha.6",
+    "@budibase/types": "^1.0.207-alpha.8",
     "@techpass/passport-openidconnect": "0.3.2",
     "aws-sdk": "2.1030.0",
     "bcrypt": "5.0.1",

packages/backend-core/src/db/pouch.js

@@ -1,21 +1,42 @@
 const PouchDB = require("pouchdb")
 const env = require("../environment")
 
-function getUrlInfo() {
-  let url = env.COUCH_DB_URL
-  let username, password, host
-  const [protocol, rest] = url.split("://")
+exports.getUrlInfo = (url = env.COUCH_DB_URL) => {
+  let cleanUrl, username, password, host
+  if (url) {
+    // Ensure the URL starts with a protocol
+    const protoRegex = /^https?:\/\//i
+    if (!protoRegex.test(url)) {
+      url = `http://${url}`
+    }
+
+    // Split into protocol and remainder
+    const split = url.split("://")
+    const protocol = split[0]
+    const rest = split.slice(1).join("://")
+
+    // Extract auth if specified
     if (url.includes("@")) {
-    const hostParts = rest.split("@")
-    host = hostParts[1]
-    const authParts = hostParts[0].split(":")
+      // Split into host and remainder
+      let parts = rest.split("@")
+      host = parts[parts.length - 1]
+      let auth = parts.slice(0, -1).join("@")
+
+      // Split auth into username and password
+      if (auth.includes(":")) {
+        const authParts = auth.split(":")
         username = authParts[0]
-    password = authParts[1]
+        password = authParts.slice(1).join(":")
+      } else {
+        username = auth
+      }
     } else {
       host = rest
     }
+    cleanUrl = `${protocol}://${host}`
+  }
   return {
-    url: `${protocol}://${host}`,
+    url: cleanUrl,
     auth: {
       username,
       password,
@@ -24,7 +45,7 @@ function getUrlInfo() {
 }
 
 exports.getCouchInfo = () => {
-  const urlInfo = getUrlInfo()
+  const urlInfo = exports.getUrlInfo()
   let username
   let password
   if (env.COUCH_DB_USERNAME) {

packages/backend-core/src/db/tests/pouch.spec.js

@@ -0,0 +1,62 @@
+require("../../../tests/utilities/TestConfiguration")
+const getUrlInfo = require("../pouch").getUrlInfo
+
+describe("pouch", () => {
+  describe("Couch DB URL parsing", () => {
+    it("should handle a null Couch DB URL", () => {
+      const info = getUrlInfo(null)
+      expect(info.url).toBeUndefined()
+      expect(info.auth.username).toBeUndefined()
+    })
+    it("should be able to parse a basic Couch DB URL", () => {
+      const info = getUrlInfo("http://host.com")
+      expect(info.url).toBe("http://host.com")
+      expect(info.auth.username).toBeUndefined()
+    })
+    it("should be able to parse a Couch DB basic URL with HTTPS", () => {
+      const info = getUrlInfo("https://host.com")
+      expect(info.url).toBe("https://host.com")
+      expect(info.auth.username).toBeUndefined()
+    })
+    it("should be able to parse a basic Couch DB URL with a custom port", () => {
+      const info = getUrlInfo("https://host.com:1234")
+      expect(info.url).toBe("https://host.com:1234")
+      expect(info.auth.username).toBeUndefined()
+    })
+    it("should be able to parse a Couch DB URL with auth", () => {
+      const info = getUrlInfo("https://user:pass@host.com:1234")
+      expect(info.url).toBe("https://host.com:1234")
+      expect(info.auth.username).toBe("user")
+      expect(info.auth.password).toBe("pass")
+    })
+    it("should be able to parse a Couch DB URL with auth and special chars", () => {
+      const info = getUrlInfo("https://user:s:p@s://@://:d@;][~s@host.com:1234")
+      expect(info.url).toBe("https://host.com:1234")
+      expect(info.auth.username).toBe("user")
+      expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
+    })
+    it("should be able to parse a Couch DB URL without a protocol", () => {
+      const info = getUrlInfo("host.com:1234")
+      expect(info.url).toBe("http://host.com:1234")
+      expect(info.auth.username).toBeUndefined()
+    })
+    it("should be able to parse a Couch DB URL with auth and without a protocol", () => {
+      const info = getUrlInfo("user:s:p@s://@://:d@;][~s@host.com:1234")
+      expect(info.url).toBe("http://host.com:1234")
+      expect(info.auth.username).toBe("user")
+      expect(info.auth.password).toBe("s:p@s://@://:d@;][~s")
+    })
+    it("should be able to parse a Couch DB URL with only username auth", () => {
+      const info = getUrlInfo("https://user@host.com:1234")
+      expect(info.url).toBe("https://host.com:1234")
+      expect(info.auth.username).toBe("user")
+      expect(info.auth.password).toBeUndefined()
+    })
+    it("should be able to parse a Couch DB URL with only username auth and without a protocol", () => {
+      const info = getUrlInfo("user@host.com:1234")
+      expect(info.url).toBe("http://host.com:1234")
+      expect(info.auth.username).toBe("user")
+      expect(info.auth.password).toBeUndefined()
+    })
+  })
+})

packages/bbui/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/bbui",
   "description": "A UI solution used in the different Budibase projects.",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "license": "MPL-2.0",
   "svelte": "src/index.js",
   "module": "dist/bbui.es.js",
@@ -38,7 +38,7 @@
   ],
   "dependencies": {
     "@adobe/spectrum-css-workflow-icons": "^1.2.1",
-    "@budibase/string-templates": "^1.0.207-alpha.6",
+    "@budibase/string-templates": "^1.0.207-alpha.8",
     "@spectrum-css/actionbutton": "^1.0.1",
     "@spectrum-css/actiongroup": "^1.0.1",
     "@spectrum-css/avatar": "^3.0.2",

packages/builder/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/builder",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "license": "GPL-3.0",
   "private": true,
   "scripts": {
@@ -69,10 +69,10 @@
     }
   },
   "dependencies": {
-    "@budibase/bbui": "^1.0.207-alpha.6",
-    "@budibase/client": "^1.0.207-alpha.6",
-    "@budibase/frontend-core": "^1.0.207-alpha.6",
-    "@budibase/string-templates": "^1.0.207-alpha.6",
+    "@budibase/bbui": "^1.0.207-alpha.8",
+    "@budibase/client": "^1.0.207-alpha.8",
+    "@budibase/frontend-core": "^1.0.207-alpha.8",
+    "@budibase/string-templates": "^1.0.207-alpha.8",
     "@sentry/browser": "5.19.1",
     "@spectrum-css/page": "^3.0.1",
     "@spectrum-css/vars": "^3.0.1",

packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/cli",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase CLI, for developers, self hosting and migrations.",
   "main": "src/index.js",
   "bin": {

packages/client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/client",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "license": "MPL-2.0",
   "module": "dist/budibase-client.js",
   "main": "dist/budibase-client.js",
@@ -19,9 +19,9 @@
     "dev:builder": "rollup -cw"
   },
   "dependencies": {
-    "@budibase/bbui": "^1.0.207-alpha.6",
-    "@budibase/frontend-core": "^1.0.207-alpha.6",
-    "@budibase/string-templates": "^1.0.207-alpha.6",
+    "@budibase/bbui": "^1.0.207-alpha.8",
+    "@budibase/frontend-core": "^1.0.207-alpha.8",
+    "@budibase/string-templates": "^1.0.207-alpha.8",
     "@spectrum-css/button": "^3.0.3",
     "@spectrum-css/card": "^3.0.3",
     "@spectrum-css/divider": "^1.0.3",

packages/frontend-core/package.json

@@ -1,12 +1,12 @@
 {
   "name": "@budibase/frontend-core",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase frontend core libraries used in builder and client",
   "author": "Budibase",
   "license": "MPL-2.0",
   "svelte": "src/index.js",
   "dependencies": {
-    "@budibase/bbui": "^1.0.207-alpha.6",
+    "@budibase/bbui": "^1.0.207-alpha.8",
     "lodash": "^4.17.21",
     "svelte": "^3.46.2"
   }

packages/server/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/server",
   "email": "hi@budibase.com",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase Web Server",
   "main": "src/index.ts",
   "repository": {
@@ -77,11 +77,11 @@
   "license": "GPL-3.0",
   "dependencies": {
     "@apidevtools/swagger-parser": "10.0.3",
-    "@budibase/backend-core": "^1.0.207-alpha.6",
-    "@budibase/client": "^1.0.207-alpha.6",
-    "@budibase/pro": "1.0.207-alpha.6",
-    "@budibase/string-templates": "^1.0.207-alpha.6",
-    "@budibase/types": "^1.0.207-alpha.6",
+    "@budibase/backend-core": "^1.0.207-alpha.8",
+    "@budibase/client": "^1.0.207-alpha.8",
+    "@budibase/pro": "1.0.207-alpha.8",
+    "@budibase/string-templates": "^1.0.207-alpha.8",
+    "@budibase/types": "^1.0.207-alpha.8",
     "@bull-board/api": "3.7.0",
     "@bull-board/koa": "3.9.4",
     "@elastic/elasticsearch": "7.10.0",

packages/server/src/definitions/common.ts

@@ -12,6 +12,7 @@ export interface Application extends Base {
 export interface FieldSchema {
   // TODO: replace with field types enum when done
   type: string
+  externalType?: string
   fieldName?: string
   name: string
   tableId?: string

packages/server/src/integrations/base/sql.ts

@@ -89,6 +89,28 @@ function parseFilters(filters: SearchFilters | undefined): SearchFilters {
   return filters
 }
 
+function generateSelectStatement(
+  json: QueryJson,
+  knex: Knex
+): (string | Knex.Raw)[] {
+  const { resource, meta } = json
+  const schema = meta?.table?.schema
+  return resource.fields.map(field => {
+    const fieldNames = field.split(/\./g)
+    const tableName = fieldNames[0]
+    const columnName = fieldNames[1]
+    if (columnName && knex.client.config.client === SqlClients.POSTGRES) {
+      const externalType = schema?.[columnName].externalType
+      if (externalType?.includes("money")) {
+        return knex.raw(
+          `"${tableName}"."${columnName}"::money::numeric as "${field}"`
+        )
+      }
+    }
+    return `${field} as ${field}`
+  })
+}
+
 class InternalBuilder {
   private readonly client: string
 
@@ -216,9 +238,7 @@ class InternalBuilder {
   }
 
   addRelationships(
-    knex: Knex,
     query: KnexQuery,
-    fields: string | string[],
     fromTable: string,
     relationships: RelationshipsJson[] | undefined
   ): KnexQuery {
@@ -323,12 +343,12 @@ class InternalBuilder {
     if (!resource) {
       resource = { fields: [] }
     }
-    let selectStatement: string | string[] = "*"
+    let selectStatement: string | (string | Knex.Raw)[] = "*"
     // handle select
     if (resource.fields && resource.fields.length > 0) {
       // select the resources as the format "table.columnName" - this is what is provided
       // by the resource builder further up
-      selectStatement = resource.fields.map(field => `${field} as ${field}`)
+      selectStatement = generateSelectStatement(json, knex)
     }
     let foundLimit = limit || BASE_LIMIT
     // handle pagination
@@ -363,13 +383,7 @@ class InternalBuilder {
       preQuery = this.addSorting(preQuery, json)
     }
     // handle joins
-    query = this.addRelationships(
-      knex,
-      preQuery,
-      selectStatement,
-      tableName,
-      relationships
-    )
+    query = this.addRelationships(preQuery, tableName, relationships)
     return this.addFilters(query, filters, { relationship: true })
   }
 

packages/string-templates/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/string-templates",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Handlebars wrapper for Budibase templating.",
   "main": "src/index.cjs",
   "module": "dist/bundle.mjs",

packages/types/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@budibase/types",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase types",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

packages/worker/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@budibase/worker",
   "email": "hi@budibase.com",
-  "version": "1.0.207-alpha.6",
+  "version": "1.0.207-alpha.8",
   "description": "Budibase background service",
   "main": "src/index.ts",
   "repository": {
@@ -34,10 +34,10 @@
   "author": "Budibase",
   "license": "GPL-3.0",
   "dependencies": {
-    "@budibase/backend-core": "^1.0.207-alpha.6",
-    "@budibase/pro": "1.0.207-alpha.6",
-    "@budibase/string-templates": "^1.0.207-alpha.6",
-    "@budibase/types": "^1.0.207-alpha.6",
+    "@budibase/backend-core": "^1.0.207-alpha.8",
+    "@budibase/pro": "1.0.207-alpha.8",
+    "@budibase/string-templates": "^1.0.207-alpha.8",
+    "@budibase/types": "^1.0.207-alpha.8",
     "@koa/router": "8.0.8",
     "@sentry/node": "6.17.7",
     "@techpass/passport-openidconnect": "0.3.2",