diff --git a/.github/workflows/README.md b/.github/workflows/README.md index f77323d85a..7e4172e602 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -115,77 +115,4 @@ This job is responsible for deploying to our production, cloud kubernetes enviro ### Rollback A Bad Cloud Deployment - Kick off cloud deploy job - Ensure you are running off master -- Enter the version number of the last known good version of budibase. For example `1.0.0` - -## Pro - -| **NOTE**: When developing for both pro / budibase repositories, your branch names need to match, or else the correct pro doesn't get run within your CI job. - -### Installing Pro - -The pro package is always installed from source in our CI jobs. - -This is done to prevent pro needing to be published prior to CI runs in budiabse. This is required for two reasons: -- To reduce developer need to manually bump versions, i.e: - - release pro, bump pro dep in budibase, now ci can run successfully -- The cyclic dependency on backend-core, i.e: - - pro depends on backend-core - - server depends on pro - - backend-core lives in the monorepo, so it can't be released independently to be used in pro - - therefore the only option is to pull pro from source and release it as a part of the monorepo release, as if it were a mono package - -The install is performed using the same steps as local development, via the `yarn bootstrap` command, see the [Contributing Guide#Pro](../../docs/CONTRIBUTING.md#pro) - -The branch to install pro from can vary depending on ref of the commit that triggered the budibase CI job. This is done to enable branches which have changes in both the monorepo and the pro repo to have their CI pass successfully. - -This is done using the [pro/install.sh](../../scripts/pro/install.sh) script. The script will: -- Clone pro to it's default branch (`develop`) -- Check if the clone worked, on forked versions of budibase this will fail due to no access - - This is fine as the `yarn` command will install the version from NPM - - Community PRs should never touch pro so this will always work -- Checkout the `BRANCH` argument, if this fails fallback to `BASE_BRANCH` - - This enables the more complex case of a feature branch being merged to another feature branch, e.g. - - I am working on a branch `epic/stonks` which exists on budibase and pro. - - I want to merge a change to this branch in budibase from `feature/stonks-ui`, which only exists in budibase - - The base branch ensures that `epic/stonks` in pro will still be checked out for the CI run, rather than falling back to `develop` -- Run `yarn setup` to build and install dependencies - - `yarn` - - `yarn bootstrap` - - `yarn build` - - The will build .ts files, and also update the `main` and `types` of `package.json` to point to `dist` rather than src - - The build command will only ever work in CI, it is prevented in local dev - -#### `BRANCH` and `BASE_BRANCH` arguments -These arguments are supplied by the various budibase build and release pipelines -- `budibase_ci` - - `BRANCH: ${{ github.event.pull_request.head.ref }}` -> The branch being merged - - `BASE_BRANCH: ${{ github.event.pull_request.base.ref}}` -> The base branch -- `release-develop` - - `BRANCH: develop` -> always use the `develop` branch in pro -- `release` - - `BRANCH: master` -> always use the `master` branch in pro - - -### Releasing Pro -After budibase dependencies have been released we will release the new version of pro to match the release version of budibase dependencies. This is to ensure that we are always keeping the version of `backend-core` in sync in the pro package and in budibase packages. Without this we could run into scenarios where different versions are being used when installed via `yarn` inside the docker images, creating very difficult to debug cases. - -Pro is released using the [pro/release.sh](../../scripts/pro/release.sh) script. The script will: -- Inspect the `VERSION` from the `lerna.json` file in budibase -- Determine whether to use the `latest` or `develop` tag based on the command argument -- Go to pro directory - - install npm creds - - update the version of `backend-core` to be `VERSION`, the version just released by lerna - - publish to npm. Uses a `lerna publish` command, pro itself is a mono repo. - - force the version to be the same as `VERSION` to keep pro and budibase in sync - - reverts the changes to `main` and `types` in `package.json` that were made by the build step, to point back to source - - commit & push: `Prep next development iteration` -- Go to budibase - - Update to the new version of pro in `server` and `worker` so the latest pro version is used in the docker builds - - commit & push: `Update pro version to $VERSION` - - -#### `COMMAND` argument -This argument is supplied by the existing `release` and `release:develop` budibase commands, which invoke the pro release -- `release` will supply no command and default to use `latest` -- `release:develop` will supply `develop` - +- Enter the version number of the last known good version of budibase. For example `1.0.0` \ No newline at end of file diff --git a/.github/workflows/release-selfhost.yml b/.github/workflows/release-selfhost.yml index 39ee812726..9e479c84d9 100644 --- a/.github/workflows/release-selfhost.yml +++ b/.github/workflows/release-selfhost.yml @@ -67,7 +67,6 @@ jobs: - name: Bootstrap and build (CLI) run: | yarn - yarn bootstrap yarn build - name: Build OpenAPI spec diff --git a/charts/budibase/templates/secrets.yaml b/charts/budibase/templates/secrets.yaml index 1c0a914ed3..263934187e 100644 --- a/charts/budibase/templates/secrets.yaml +++ b/charts/budibase/templates/secrets.yaml @@ -1,4 +1,5 @@ -{{- if .Values.globals.createSecrets -}} +{{- $existingSecret := lookup "v1" "Secret" .Release.Namespace (include "budibase.fullname" .) }} +{{- if .Values.globals.createSecrets }} apiVersion: v1 kind: Secret metadata: @@ -10,8 +11,15 @@ metadata: heritage: "{{ .Release.Service }}" type: Opaque data: + {{- if $existingSecret }} + internalApiKey: {{ index $existingSecret.data "internalApiKey" }} + jwtSecret: {{ index $existingSecret.data "jwtSecret" }} + objectStoreAccess: {{ index $existingSecret.data "objectStoreAccess" }} + objectStoreSecret: {{ index $existingSecret.data "objectStoreSecret" }} + {{- else }} internalApiKey: {{ template "budibase.defaultsecret" .Values.globals.internalApiKey }} jwtSecret: {{ template "budibase.defaultsecret" .Values.globals.jwtSecret }} objectStoreAccess: {{ template "budibase.defaultsecret" .Values.services.objectStore.accessKey }} objectStoreSecret: {{ template "budibase.defaultsecret" .Values.services.objectStore.secretKey }} -{{- end -}} + {{- end }} +{{- end }} diff --git a/docs/DEV-SETUP-DEBIAN.md b/docs/DEV-SETUP-DEBIAN.md index a8b1e3dce4..e098862c64 100644 --- a/docs/DEV-SETUP-DEBIAN.md +++ b/docs/DEV-SETUP-DEBIAN.md @@ -55,7 +55,7 @@ yarn setup The yarn setup command runs several build steps i.e. ``` -node ./hosting/scripts/setup.js && yarn && yarn bootstrap && yarn build && yarn dev +node ./hosting/scripts/setup.js && yarn && yarn build && yarn dev ``` So this command will actually run the application in dev mode. It creates .env files under `./packages/server` and `./packages/worker` and runs docker containers for each service via docker-compose. diff --git a/docs/DEV-SETUP-MACOSX.md b/docs/DEV-SETUP-MACOSX.md index 94ed3fc1ee..0e13d540b3 100644 --- a/docs/DEV-SETUP-MACOSX.md +++ b/docs/DEV-SETUP-MACOSX.md @@ -55,7 +55,7 @@ yarn setup The yarn setup command runs several build steps i.e. ``` -node ./hosting/scripts/setup.js && yarn && yarn bootstrap && yarn build && yarn dev +node ./hosting/scripts/setup.js && yarn && yarn build && yarn dev ``` So this command will actually run the application in dev mode. It creates .env files under `./packages/server` and `./packages/worker` and runs docker containers for each service via docker-compose. diff --git a/docs/DEV-SETUP-WINDOWS.md b/docs/DEV-SETUP-WINDOWS.md index 176e0700d7..f26a5a0882 100644 --- a/docs/DEV-SETUP-WINDOWS.md +++ b/docs/DEV-SETUP-WINDOWS.md @@ -74,7 +74,7 @@ yarn setup The yarn setup command runs several build steps i.e. ``` -node ./hosting/scripts/setup.js && yarn && yarn bootstrap && yarn build && yarn dev +node ./hosting/scripts/setup.js && yarn && yarn build && yarn dev ``` So this command will actually run the application in dev mode. It creates .env files under `./packages/server` and `./packages/worker` and runs docker containers for each service via docker-compose. diff --git a/hosting/single/README.md b/hosting/single/README.md index 1147d55c89..09010f5075 100644 --- a/hosting/single/README.md +++ b/hosting/single/README.md @@ -58,7 +58,6 @@ Node setup: ``` node ./hosting/scripts/setup.js yarn -yarn bootstrap yarn build ``` #### Build Image diff --git a/hosting/tests/README.md b/hosting/tests/README.md index 8586b31948..19b9ed5037 100644 --- a/hosting/tests/README.md +++ b/hosting/tests/README.md @@ -47,7 +47,6 @@ Node setup: ``` node ./hosting/scripts/setup.js yarn -yarn bootstrap yarn build ``` #### Build Image diff --git a/lerna.json b/lerna.json index 5653cfedf1..d0910235b4 100644 --- a/lerna.json +++ b/lerna.json @@ -1,5 +1,5 @@ { - "version": "2.9.33-alpha.15", + "version": "2.9.39-alpha.4", "npmClient": "yarn", "packages": [ "packages/*" diff --git a/package.json b/package.json index 4e4befb5f2..b0c428fe70 100644 --- a/package.json +++ b/package.json @@ -33,21 +33,18 @@ "scripts": { "preinstall": "node scripts/syncProPackage.js", "setup": "git config submodule.recurse true && git submodule update && node ./hosting/scripts/setup.js && yarn && yarn build && yarn dev", - "bootstrap": "./scripts/link-dependencies.sh && echo '***BOOTSTRAP ONLY REQUIRED FOR USE WITH ACCOUNT PORTAL***'", "build": "lerna run build --stream", "build:dev": "lerna run --stream prebuild && yarn nx run-many --target=build --output-style=dynamic --watch --preserveWatchOutput", "check:types": "lerna run check:types", - "backend:bootstrap": "./scripts/scopeBackend.sh && yarn run bootstrap", - "backend:build": "./scripts/scopeBackend.sh 'lerna run --stream build'", "build:sdk": "lerna run --stream build:sdk", "deps:circular": "madge packages/server/dist/index.js packages/worker/src/index.ts packages/backend-core/dist/src/index.js packages/cli/src/index.js --circular", "release": "lerna publish from-package --yes --force-publish --no-git-tag-version --no-push --no-git-reset", "release:develop": "yarn release --dist-tag develop", - "restore": "yarn run clean && yarn run bootstrap && yarn run build", + "restore": "yarn run clean && yarn && yarn run build", "nuke": "yarn run nuke:packages && yarn run nuke:docker", "nuke:packages": "yarn run restore", "nuke:docker": "lerna run --stream dev:stack:nuke", - "clean": "lerna clean", + "clean": "lerna clean -y", "kill-builder": "kill-port 3000", "kill-server": "kill-port 4001 4002", "kill-all": "yarn run kill-builder && yarn run kill-server", @@ -93,9 +90,8 @@ "mode:account": "yarn mode:cloud && yarn env:account:enable", "security:audit": "node scripts/audit.js", "postinstall": "husky install", - "dep:clean": "yarn clean -y && yarn bootstrap", - "submodules:load": "git submodule init && git submodule update && yarn && yarn bootstrap", - "submodules:unload": "git submodule deinit --all && yarn && yarn bootstrap" + "submodules:load": "git submodule init && git submodule update && yarn", + "submodules:unload": "git submodule deinit --all && yarn" }, "workspaces": { "packages": [ diff --git a/packages/backend-core/package.json b/packages/backend-core/package.json index 1d6aa5b9fd..4267425b84 100644 --- a/packages/backend-core/package.json +++ b/packages/backend-core/package.json @@ -4,9 +4,16 @@ "description": "Budibase backend core libraries used in server and worker", "main": "dist/index.js", "types": "dist/src/index.d.ts", + "typesVersions": { + "*": { + "tests": [ + "./dist/tests/index.d.ts" + ] + } + }, "exports": { ".": "./dist/index.js", - "./tests": "./dist/tests.js", + "./tests": "./dist/tests/index.js", "./*": "./dist/*.js" }, "author": "Budibase", diff --git a/packages/backend-core/scripts/build.js b/packages/backend-core/scripts/build.js index bd00cbc7ff..b40404aade 100644 --- a/packages/backend-core/scripts/build.js +++ b/packages/backend-core/scripts/build.js @@ -1,6 +1,22 @@ #!/usr/bin/node + const coreBuild = require("../../../scripts/build") coreBuild("./src/plugin/index.ts", "./dist/plugins.js") coreBuild("./src/index.ts", "./dist/index.js") -coreBuild("./tests/index.ts", "./dist/tests.js") + +const glob = require("glob") +const inputFiles = [ + ...glob.sync("./src/**/*.[tj]s", { nodir: true }), + ...glob.sync("./tests/**/*.[tj]s", { nodir: true }), +] + +const path = require("path") +for (const file of inputFiles) { + coreBuild(file, `./${path.join("dist", file.replace(/\.ts$/, ".js"))}`, { + skipMeta: true, + bundle: false, + forcedFormat: "cjs", + silent: true, + }) +} diff --git a/packages/backend-core/src/db/couch/DatabaseImpl.ts b/packages/backend-core/src/db/couch/DatabaseImpl.ts index 89f76769b3..29ca4123f5 100644 --- a/packages/backend-core/src/db/couch/DatabaseImpl.ts +++ b/packages/backend-core/src/db/couch/DatabaseImpl.ts @@ -8,7 +8,6 @@ import { DatabasePutOpts, DatabaseCreateIndexOpts, DatabaseDeleteIndexOpts, - DocExistsResponse, Document, isDocument, } from "@budibase/types" @@ -121,19 +120,6 @@ export class DatabaseImpl implements Database { return this.updateOutput(() => db.get(id)) } - async docExists(docId: string): Promise { - const db = await this.checkSetup() - let _rev, exists - try { - const { etag } = await db.head(docId) - _rev = etag - exists = true - } catch (err) { - exists = false - } - return { _rev, exists } - } - async remove(idOrDoc: string | Document, rev?: string) { const db = await this.checkSetup() let _id: string diff --git a/packages/backend-core/src/security/roles.ts b/packages/backend-core/src/security/roles.ts index 05d536562b..e87df2e9c9 100644 --- a/packages/backend-core/src/security/roles.ts +++ b/packages/backend-core/src/security/roles.ts @@ -380,8 +380,8 @@ export function getDBRoleID(roleName: string) { export function getExternalRoleID(roleId: string, version?: string) { // for built-in roles we want to remove the DB role ID element (role_) if ( - (roleId.startsWith(DocumentType.ROLE) && isBuiltin(roleId)) || - version === RoleIDVersion.NAME + roleId.startsWith(DocumentType.ROLE) && + (isBuiltin(roleId) || version === RoleIDVersion.NAME) ) { return roleId.split(`${DocumentType.ROLE}${SEPARATOR}`)[1] } diff --git a/packages/builder/src/components/backend/DataTable/ViewDataTable.svelte b/packages/builder/src/components/backend/DataTable/ViewDataTable.svelte index d239cabd59..f6160e3caa 100644 --- a/packages/builder/src/components/backend/DataTable/ViewDataTable.svelte +++ b/packages/builder/src/components/backend/DataTable/ViewDataTable.svelte @@ -10,6 +10,7 @@ import ManageAccessButton from "./buttons/ManageAccessButton.svelte" import HideAutocolumnButton from "./buttons/HideAutocolumnButton.svelte" import { notifications } from "@budibase/bbui" + import { ROW_EXPORT_FORMATS } from "constants/backend" export let view = {} @@ -19,6 +20,14 @@ let type = "internal" $: name = view.name + $: calculation = view.calculation + + $: supportedFormats = Object.values(ROW_EXPORT_FORMATS).filter(key => { + if (calculation && key === ROW_EXPORT_FORMATS.JSON_WITH_SCHEMA) { + return false + } + return true + }) // Fetch rows for specified view $: fetchViewData(name, view.field, view.groupBy, view.calculation) @@ -68,5 +77,5 @@ {/if} - + diff --git a/packages/builder/src/components/backend/DataTable/buttons/ExportButton.svelte b/packages/builder/src/components/backend/DataTable/buttons/ExportButton.svelte index fd0d64f6cd..4fa1d07abd 100644 --- a/packages/builder/src/components/backend/DataTable/buttons/ExportButton.svelte +++ b/packages/builder/src/components/backend/DataTable/buttons/ExportButton.svelte @@ -7,6 +7,7 @@ export let sorting export let disabled = false export let selectedRows + export let formats let modal @@ -15,5 +16,5 @@ Export - + diff --git a/packages/builder/src/components/backend/DataTable/modals/ExportModal.svelte b/packages/builder/src/components/backend/DataTable/modals/ExportModal.svelte index ac168698fc..09f76d3522 100644 --- a/packages/builder/src/components/backend/DataTable/modals/ExportModal.svelte +++ b/packages/builder/src/components/backend/DataTable/modals/ExportModal.svelte @@ -9,30 +9,43 @@ import download from "downloadjs" import { API } from "api" import { Constants, LuceneUtils } from "@budibase/frontend-core" - - const FORMATS = [ - { - name: "CSV", - key: "csv", - }, - { - name: "JSON", - key: "json", - }, - { - name: "JSON with Schema", - key: "jsonWithSchema", - }, - ] + import { ROW_EXPORT_FORMATS } from "constants/backend" export let view export let filters export let sorting export let selectedRows = [] + export let formats - let exportFormat = FORMATS[0].key + const FORMATS = [ + { + name: "CSV", + key: ROW_EXPORT_FORMATS.CSV, + }, + { + name: "JSON", + key: ROW_EXPORT_FORMATS.JSON, + }, + { + name: "JSON with Schema", + key: ROW_EXPORT_FORMATS.JSON_WITH_SCHEMA, + }, + ] + + $: options = FORMATS.filter(format => { + if (formats && !formats.includes(format.key)) { + return false + } + return true + }) + + let exportFormat let filterLookup + $: if (options && !exportFormat) { + exportFormat = Array.isArray(options) ? options[0]?.key : [] + } + $: luceneFilter = LuceneUtils.buildLuceneQuery(filters) $: exportOpDisplay = buildExportOpDisplay(sorting, filterDisplay, filters) @@ -190,7 +203,7 @@