diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 7abfe537e9..d1e373003a 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -79,6 +79,8 @@ Component libraries are collections of components as well as the definition of t ### Getting Started For Contributors #### 1. Prerequisites +NodeJS Version `14.x.x` + *yarn -* `npm install -g yarn` *jest* - `npm install -g jest` @@ -177,36 +179,7 @@ To enable this mode, use: yarn mode:account ``` ### CI - -#### PR Job - -After your pr is submitted a github action (can be found at `.github/workflows/budibase_ci.yml`) will run to perform some checks against the changes such as linting, build and test. - -The job will run when changes are pushed to or targetted at `master` and `develop` -#### Release Develop - -To test changes before a release, a prerelease action (can be found at `.github/workflows/release-develop.yml`) will run to build and release develop versions of npm packages and docker images. On each subsequent commit to develop a new alpha version of npm packages will be created and released. - -For example: - -- `feature1` -> `develop` = `v0.9.160-alpha.1` -- `feature2` -> `develop` = `v0.9.160-alpha.0` - -The job will run when changes are pushed to `develop` -#### Release Job - -To release changes a release job (can be found at `.github/workflows/release.yml`) will run to create final versions of npm packages and docker images. - -Following the example above: - -- `develop` -> `master` = `v0.9.160` - -The job will run when changes are pushed to `master` - -#### Release Self Host Job - -To release the self hosted version of docker images, an additional job (can be found at `.github/workflows/release-selfhost.yml`) must be ran manually. This will releaae docker images to docker hub under the tag `latest` to be picked up by self hosted installations. - + An overview of the CI pipelines can be found [here](./workflows/README.md) ### Troubleshooting Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above. You should have a fresh Budibase installation. diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000000..d2fcd16bb0 --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,93 @@ + +# Budibase CI Pipelines + +Welcome to the budibase CI pipelines directory. This document details what each of the CI pipelines are for, and come common combinations. + +## All CI Pipelines + +### Note +- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. + +### Standard CI Build Job (budibase_ci.yml) +Triggers: +- PR or push to develop +- PR or push to master + +The standard CI Build job is what runs when you raise a PR to develop or master. +- Installs all dependencies, +- builds the project +- run the unit tests +- Generate test coverage metrics with codecov +- Run the cypress tests + +### Release Develop Job (release-develop.yml) +Triggers: +- Push to develop + +The job responsible for building, tagging and pushing docker images out to the test and staging environments. +- Installs all dependencies +- builds the project +- run the unit tests +- publish the budibase JS packages under a prerelease tag to NPM +- build, tag and push docker images under the `develop` tag to docker hub + +These images will then be pulled by the test and staging environments, updating the latest automatically. Discord notifications are sent to the #infra channel when this occurs. + +### Release Job (release.yml) +Triggers: +- Push to master + +This job is responsible for building and pushing the latest code to NPM and docker hub, so that it can be deployed. +- Installs all dependencies +- builds the project +- run the unit tests +- publish the budibase JS packages under a release tag to NPM (always incremented by patch versions) +- build, tag and push docker images under the `v.x.x.x` (the tag of the NPM release) tag to docker hub + +### Release Selfhost Job (release-selfhost.yml) +Triggers: +- Manual Workflow Dispatch Trigger + +This job is responsible for delivering the latest version of budibase to those that are self-hosting. + +This job relies on the release job to have run first, so the latest image is pushed to dockerhub. This job then will pull the latest version from `lerna.json` and try to find an image in dockerhub corresponding to that version. For example, if the version in `lerna.json` is `1.0.0`: +- Pull the images for all budibase services tagged `v1.0.0` from dockerhub +- Tag these images as `latest` +- Push them back to dockerhub. This now means anyone who pulls `latest` (self hosters using docker-compose) will get the latest version. +- Build and release the budibase helm chart for kubernetes users +- Perform a github release with the latest version. You can see previous releases here (https://github.com/Budibase/budibase/releases) + + +### Cloud Deploy (deploy-cloud.yml) +Triggers: +- Manual Workflow Dispatch Trigger + +This job is responsible for deploying to our production, cloud kubernetes environment. You must run the release job first, to ensure that the latest images have been built and pushed to docker hub. You can also manually enter a version number for this job, so you can perform rollbacks or upgrade to a specific version. After kicking off this job, the following will occur: + +- Checks out the master branch +- Pulls the latest `values.yaml` from budibase infra, a private repo containing budibases infrastructure configuration +- Gets the latest budibase version from `lerna.json`, if it hasn't been specified in the workflow when you kicked it off +- Configures AWS Credentials +- Deploys the helm chart in the budibase repo to our production EKS cluster, injecting the `values.yaml` we pulled from budibase-infra +- Fires off a discord webhook in the #infra channel to show that the deployment completely successfully. + +## Common Workflows + +### Deploy Changes to Production (Release) +- Merge `develop` into `master` +- Wait for budibase CI job and release job to run +- Run cloud deploy job +- Run release selfhost job + +### Deploy Changes to Production (Hotfix) +- Branch off `master` +- Perform your hotfix +- Merge back into `master` +- Wait for budibase CI job and release job to run +- Run cloud deploy job +- Run release selfhost job + +### Rollback A Bad Cloud Deployment +- Kick off cloud deploy job +- Ensure you are running off master +- Enter the version number of the last known good version of budibase. For example `1.0.0` \ No newline at end of file diff --git a/.github/workflows/budibase_ci.yml b/.github/workflows/budibase_ci.yml index 5c4a111e23..7e95115415 100644 --- a/.github/workflows/budibase_ci.yml +++ b/.github/workflows/budibase_ci.yml @@ -41,4 +41,6 @@ jobs: files: ./packages/server/coverage/clover.xml name: codecov-umbrella verbose: true + + # TODO: parallelise this - run: yarn test:e2e:ci diff --git a/.github/workflows/deploy-cloud.yaml b/.github/workflows/deploy-cloud.yaml index 26422a2e7b..d54e6c9c68 100644 --- a/.github/workflows/deploy-cloud.yaml +++ b/.github/workflows/deploy-cloud.yaml @@ -12,6 +12,12 @@ jobs: runs-on: ubuntu-latest steps: + - name: Fail if branch is not master + if: github.ref != 'refs/heads/master' + run: | + echo "Ref is not master, you must run this job from master." + exit 1 + - uses: actions/checkout@v2 - name: Pull values.yaml from budibase-infra diff --git a/.github/workflows/deploy-preprod.yml b/.github/workflows/deploy-preprod.yml index ce789e71ec..5b3282313c 100644 --- a/.github/workflows/deploy-preprod.yml +++ b/.github/workflows/deploy-preprod.yml @@ -23,16 +23,19 @@ jobs: aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: eu-west-1 - - name: 'Get Previous tag' - id: previoustag - uses: "WyriHaximus/github-action-get-previous-tag@v1" + - name: Get the latest budibase release version + id: version + run: | + release_version=$(cat lerna.json | jq -r '.version') + echo "RELEASE_VERSION=$release_version" >> $GITHUB_ENV - # - name: Pull values.yaml from budibase-infra - # run: | - # curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ [c3a7a9d12] - # -H 'Accept: application/vnd.github.v3.raw' \ - # -o values.preprod.yaml \ - # -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml + - name: Pull values.yaml from budibase-infra + run: | + curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ + -H 'Accept: application/vnd.github.v3.raw' \ + -o values.preprod.yaml \ + -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/budibase-preprod/values.yaml + wc -l values.preprod.yaml - name: Deploy to Preprod Environment uses: deliverybot/helm@v1 @@ -41,13 +44,17 @@ jobs: namespace: budibase chart: charts/budibase token: ${{ github.token }} + helm: helm3 values: | globals: - appVersion: ${{ steps.previoustag.outputs.tag }} - # value-files: >- - # [ - # "charts/budibase/values.yaml" - # ] + appVersion: v${{ env.RELEASE_VERSION }} + ingress: + enabled: true + nginx: true + value-files: >- + [ + "values.preprod.yaml" + ] env: KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}' diff --git a/.github/workflows/release-develop.yml b/.github/workflows/release-develop.yml index b0f052f68f..4467cd6c81 100644 --- a/.github/workflows/release-develop.yml +++ b/.github/workflows/release-develop.yml @@ -4,6 +4,16 @@ on: push: branches: - develop + paths: + - '.aws/**' + - '.github/**' + - 'charts/**' + - 'packages/**' + - 'scripts/**' + - 'package.json' + - 'yarn.lock' + - 'package.json' + - 'yarn.lock' env: POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }} diff --git a/.github/workflows/release-selfhost.yml b/.github/workflows/release-selfhost.yml index 31af7fe571..e842da9d7c 100644 --- a/.github/workflows/release-selfhost.yml +++ b/.github/workflows/release-selfhost.yml @@ -9,12 +9,12 @@ jobs: steps: - uses: actions/checkout@v2 - - uses: actions/setup-node@v1 - with: + with: node-version: 14.x + fetch_depth: 0 - name: Tag and release Docker images (Self Host) - run: | + run: | docker login -u $DOCKER_USER -p $DOCKER_PASSWORD # Get latest release version @@ -29,7 +29,7 @@ jobs: # Tag apps and worker images docker tag budibase/apps:$release_tag budibase/apps:$SELFHOST_TAG docker tag budibase/worker:$release_tag budibase/worker:$SELFHOST_TAG - + # Push images docker push budibase/apps:$SELFHOST_TAG docker push budibase/worker:$SELFHOST_TAG @@ -37,37 +37,32 @@ jobs: DOCKER_USER: ${{ secrets.DOCKER_USERNAME }} DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }} SELFHOST_TAG: latest - - - name: Setup Helm - uses: azure/setup-helm@v1 - id: helm-install - + - name: Build CLI executables run: | pushd packages/cli yarn yarn build popd + + - name: Setup Helm + uses: azure/setup-helm@v1 + id: helm-install - # - name: Build and release helm chart - # run: | - # git config user.name "Budibase Helm Bot" - # git config user.email "<>" - # mv budibase-${{ env.RELEASE_VERSION }}.tgz docs - # helm repo index docs - # git checkout gh-pages - # git add -A - # git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}" - # git push - # env: - # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - - # - name: Deploy - # uses: peaceiris/actions-gh-pages@v3 - # with: - # github_token: ${{ secrets.GITHUB_TOKEN }} - # publish_dir: ./public - # full_commit_message: "Helm Release: ${{ env.RELEASE_VERSION }}" + - name: Build and release helm chart + run: | + git config user.name "Budibase Helm Bot" + git config user.email "<>" + git pull + helm package charts/budibase + git checkout gh-pages + mv *.tgz docs + helm repo index docs + git add -A + git commit -m "Helm Release: ${{ env.RELEASE_VERSION }}" + git push + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Perform Github Release uses: softprops/action-gh-release@v1 @@ -78,4 +73,4 @@ jobs: files: | packages/cli/build/cli-win.exe packages/cli/build/cli-linux - packages/cli/build/cli-macos \ No newline at end of file + packages/cli/build/cli-macos \ No newline at end of file diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 859d89f18b..82848c78e4 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -3,7 +3,17 @@ name: Budibase Release on: push: branches: - - test + - master + paths: + - '.aws/**' + - '.github/**' + - 'charts/**' + - 'packages/**' + - 'scripts/**' + - 'package.json' + - 'yarn.lock' + - 'package.json' + - 'yarn.lock' env: POSTHOG_TOKEN: ${{ secrets.POSTHOG_TOKEN }} @@ -56,34 +66,3 @@ jobs: DOCKER_USER: ${{ secrets.DOCKER_USERNAME }} DOCKER_PASSWORD: ${{ secrets.DOCKER_API_KEY }} BUDIBASE_RELEASE_VERSION: ${{ steps.previoustag.outputs.tag }} - - # - name: Pull values.yaml from budibase-infra - # run: | - # curl -H "Authorization: token ${{ secrets.GH_PERSONAL_TOKEN }}" \ [c3a7a9d12] - # -H 'Accept: application/vnd.github.v3.raw' \ - # -o values.preprod.yaml \ - # -L https://api.github.com/repos/budibase/budibase-infra/contents/kubernetes/values.preprod.yaml - - # - name: Deploy to Preprod Environment - # uses: deliverybot/helm@v1 - # with: - # release: budibase-preprod - # namespace: budibase - # chart: charts/budibase - # token: ${{ github.token }} - # values: | - # globals: - # appVersion: ${{ steps.previoustag.outputs.tag }} - # # value-files: >- - # # [ - # # "charts/budibase/values.yaml" - # # ] - # env: - # KUBECONFIG_FILE: '${{ secrets.PREPROD_KUBECONFIG }}' - - # - name: Discord Webhook Action - # uses: tsickert/discord-webhook@v4.0.0 - # with: - # webhook-url: ${{ secrets.PROD_DEPLOY_WEBHOOK_URL }} - # content: "Preprod Deployment Complete: ${{ env.RELEASE_VERSION }} deployed to Budibase Pre-prod." - # embed-title: ${{ env.RELEASE_VERSION }} diff --git a/charts/budibase/Chart.lock b/charts/budibase/Chart.lock new file mode 100644 index 0000000000..75b9de07b5 --- /dev/null +++ b/charts/budibase/Chart.lock @@ -0,0 +1,9 @@ +dependencies: +- name: couchdb + repository: https://apache.github.io/couchdb-helm + version: 3.3.4 +- name: ingress-nginx + repository: https://kubernetes.github.io/ingress-nginx + version: 4.0.13 +digest: sha256:20892705c2d8e64c98257d181063a514ac55013e2b43399a6e54868a97f97845 +generated: "2021-12-30T18:55:30.878411Z" diff --git a/charts/budibase/Chart.yaml b/charts/budibase/Chart.yaml index cec5af2914..8c9d44f201 100644 --- a/charts/budibase/Chart.yaml +++ b/charts/budibase/Chart.yaml @@ -11,14 +11,14 @@ sources: - https://github.com/Budibase/budibase - https://budibase.com type: application -version: 1.0.0 -appVersion: 1.0.20 +version: 0.2.5 +appVersion: 1.0.25 dependencies: - name: couchdb version: 3.3.4 repository: https://apache.github.io/couchdb-helm condition: services.couchdb.enabled - name: ingress-nginx - version: 3.35.0 - repository: https://github.com/kubernetes/ingress-nginx + version: 4.0.13 + repository: https://kubernetes.github.io/ingress-nginx condition: ingress.nginx diff --git a/charts/budibase/charts/ingress-nginx-3.35.0.tgz b/charts/budibase/charts/ingress-nginx-3.35.0.tgz deleted file mode 100644 index ee5214c497..0000000000 Binary files a/charts/budibase/charts/ingress-nginx-3.35.0.tgz and /dev/null differ diff --git a/charts/budibase/charts/ingress-nginx-4.0.13.tgz b/charts/budibase/charts/ingress-nginx-4.0.13.tgz new file mode 100644 index 0000000000..1e34215c5f Binary files /dev/null and b/charts/budibase/charts/ingress-nginx-4.0.13.tgz differ diff --git a/charts/budibase/templates/proxy-service-service.yaml b/charts/budibase/templates/proxy-service-service.yaml index 8f14d97862..bf2b199ee5 100644 --- a/charts/budibase/templates/proxy-service-service.yaml +++ b/charts/budibase/templates/proxy-service-service.yaml @@ -9,12 +9,11 @@ metadata: app.kubernetes.io/name: budibase-proxy name: proxy-service spec: - type: NodePort ports: - - port: {{ .Values.services.proxy.port }} + - name: {{ .Values.services.proxy.port | quote }} + port: {{ .Values.services.proxy.port }} targetPort: {{ .Values.services.proxy.port }} - protocol: TCP selector: app.kubernetes.io/name: budibase-proxy status: - loadBalancer: {} + loadBalancer: {} \ No newline at end of file diff --git a/charts/budibase/values.yaml b/charts/budibase/values.yaml index b634f1581f..4666d01c70 100644 --- a/charts/budibase/values.yaml +++ b/charts/budibase/values.yaml @@ -40,7 +40,7 @@ service: port: 10000 ingress: - enabled: false + enabled: true aws: false nginx: true certificateArn: "" @@ -302,4 +302,4 @@ couchdb: initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 - timeoutSeconds: 1 \ No newline at end of file + timeoutSeconds: 1 diff --git a/lerna.json b/lerna.json index e492e58a75..4bbdf26b2f 100644 --- a/lerna.json +++ b/lerna.json @@ -1,5 +1,5 @@ { - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "npmClient": "yarn", "packages": [ "packages/*" diff --git a/packages/backend-core/package.json b/packages/backend-core/package.json index c358c7f9d4..648a493712 100644 --- a/packages/backend-core/package.json +++ b/packages/backend-core/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/backend-core", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "description": "Budibase backend core libraries used in server and worker", "main": "src/index.js", "author": "Budibase", diff --git a/packages/bbui/package.json b/packages/bbui/package.json index 941127eb92..199ccd6db2 100644 --- a/packages/bbui/package.json +++ b/packages/bbui/package.json @@ -1,7 +1,7 @@ { "name": "@budibase/bbui", "description": "A UI solution used in the different Budibase projects.", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "license": "MPL-2.0", "svelte": "src/index.js", "module": "dist/bbui.es.js", diff --git a/packages/builder/package.json b/packages/builder/package.json index 8b03c66b74..06583ffde8 100644 --- a/packages/builder/package.json +++ b/packages/builder/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/builder", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "license": "GPL-3.0", "private": true, "scripts": { @@ -65,10 +65,10 @@ } }, "dependencies": { - "@budibase/bbui": "^1.0.27-alpha.13", - "@budibase/client": "^1.0.27-alpha.13", + "@budibase/bbui": "^1.0.27-alpha.17", + "@budibase/client": "^1.0.27-alpha.17", "@budibase/colorpicker": "1.1.2", - "@budibase/string-templates": "^1.0.27-alpha.13", + "@budibase/string-templates": "^1.0.27-alpha.17", "@sentry/browser": "5.19.1", "@spectrum-css/page": "^3.0.1", "@spectrum-css/vars": "^3.0.1", diff --git a/packages/cli/package.json b/packages/cli/package.json index f485df8acb..2f012e7630 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/cli", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "description": "Budibase CLI, for developers, self hosting and migrations.", "main": "src/index.js", "bin": { diff --git a/packages/client/package.json b/packages/client/package.json index c0047da8ef..83b742d7fb 100644 --- a/packages/client/package.json +++ b/packages/client/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/client", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "license": "MPL-2.0", "module": "dist/budibase-client.js", "main": "dist/budibase-client.js", @@ -19,9 +19,9 @@ "dev:builder": "rollup -cw" }, "dependencies": { - "@budibase/bbui": "^1.0.27-alpha.13", + "@budibase/bbui": "^1.0.27-alpha.17", "@budibase/standard-components": "^0.9.139", - "@budibase/string-templates": "^1.0.27-alpha.13", + "@budibase/string-templates": "^1.0.27-alpha.17", "regexparam": "^1.3.0", "shortid": "^2.2.15", "svelte-spa-router": "^3.0.5" diff --git a/packages/server/package.json b/packages/server/package.json index a958e175a6..4fd5e89f4f 100644 --- a/packages/server/package.json +++ b/packages/server/package.json @@ -1,7 +1,7 @@ { "name": "@budibase/server", "email": "hi@budibase.com", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "description": "Budibase Web Server", "main": "src/index.ts", "repository": { @@ -70,9 +70,9 @@ "license": "GPL-3.0", "dependencies": { "@apidevtools/swagger-parser": "^10.0.3", - "@budibase/backend-core": "^1.0.27-alpha.13", - "@budibase/client": "^1.0.27-alpha.13", - "@budibase/string-templates": "^1.0.27-alpha.13", + "@budibase/backend-core": "^1.0.27-alpha.17", + "@budibase/client": "^1.0.27-alpha.17", + "@budibase/string-templates": "^1.0.27-alpha.17", "@bull-board/api": "^3.7.0", "@bull-board/koa": "^3.7.0", "@elastic/elasticsearch": "7.10.0", diff --git a/packages/server/src/environment.js b/packages/server/src/environment.js index 925fede086..a92e113851 100644 --- a/packages/server/src/environment.js +++ b/packages/server/src/environment.js @@ -65,6 +65,7 @@ module.exports = { DEPLOYMENT_CREDENTIALS_URL: process.env.DEPLOYMENT_CREDENTIALS_URL, ALLOW_DEV_AUTOMATIONS: process.env.ALLOW_DEV_AUTOMATIONS, DISABLE_THREADING: process.env.DISABLE_THREADING, + QUERY_THREAD_TIMEOUT: process.env.QUERY_THREAD_TIMEOUT, _set(key, value) { process.env[key] = value module.exports[key] = value diff --git a/packages/server/src/integrations/dynamodb.ts b/packages/server/src/integrations/dynamodb.ts index 0bc3a1273a..bfcac3184d 100644 --- a/packages/server/src/integrations/dynamodb.ts +++ b/packages/server/src/integrations/dynamodb.ts @@ -80,6 +80,17 @@ module DynamoModule { }, }, }, + describe: { + type: QueryTypes.FIELDS, + customisable: true, + readable: true, + fields: { + table: { + type: DatasourceFieldTypes.STRING, + required: true, + }, + }, + }, get: { type: QueryTypes.FIELDS, customisable: true, @@ -180,6 +191,13 @@ module DynamoModule { return response } + async describe(query: { table: string }) { + const params = { + TableName: query.table, + } + return new AWS.DynamoDB().describeTable(params).promise() + } + async get(query: { table: string; json: object }) { const params = { TableName: query.table, diff --git a/packages/server/src/integrations/rest.ts b/packages/server/src/integrations/rest.ts index 3199ce3bde..ea40dfb609 100644 --- a/packages/server/src/integrations/rest.ts +++ b/packages/server/src/integrations/rest.ts @@ -43,8 +43,8 @@ const coreFields = { enum: Object.values(BodyTypes), }, pagination: { - type: DatasourceFieldTypes.OBJECT - } + type: DatasourceFieldTypes.OBJECT, + }, } module RestModule { @@ -178,12 +178,17 @@ module RestModule { headers, }, pagination: { - cursor: nextCursor - } + cursor: nextCursor, + }, } } - getUrl(path: string, queryString: string, pagination: PaginationConfig | null, paginationValues: PaginationValues | null): string { + getUrl( + path: string, + queryString: string, + pagination: PaginationConfig | null, + paginationValues: PaginationValues | null + ): string { // Add pagination params to query string if required if (pagination?.location === "query" && paginationValues) { const { pageParam, sizeParam } = pagination @@ -217,14 +222,22 @@ module RestModule { return complete } - addBody(bodyType: string, body: string | any, input: any, pagination: PaginationConfig | null, paginationValues: PaginationValues | null) { + addBody( + bodyType: string, + body: string | any, + input: any, + pagination: PaginationConfig | null, + paginationValues: PaginationValues | null + ) { if (!input.headers) { input.headers = {} } if (bodyType === BodyTypes.NONE) { return input } - let error, object: any = {}, string = "" + let error, + object: any = {}, + string = "" try { if (body) { string = typeof body !== "string" ? JSON.stringify(body) : body @@ -333,7 +346,7 @@ module RestModule { requestBody, authConfigId, pagination, - paginationValues + paginationValues, } = query const authHeaders = this.getAuthHeaders(authConfigId) @@ -352,7 +365,13 @@ module RestModule { } let input: any = { method, headers: this.headers } - input = this.addBody(bodyType, requestBody, input, pagination, paginationValues) + input = this.addBody( + bodyType, + requestBody, + input, + pagination, + paginationValues + ) this.startTimeMs = performance.now() const url = this.getUrl(path, queryString, pagination, paginationValues) diff --git a/packages/server/src/integrations/s3.ts b/packages/server/src/integrations/s3.ts index bcbcdf342a..25b439fd58 100644 --- a/packages/server/src/integrations/s3.ts +++ b/packages/server/src/integrations/s3.ts @@ -8,6 +8,8 @@ module S3Module { region: string accessKeyId: string secretAccessKey: string + s3ForcePathStyle: boolean + endpoint?: string } const SCHEMA: Integration = { @@ -18,7 +20,7 @@ module S3Module { datasource: { region: { type: "string", - required: true, + required: false, default: "us-east-1", }, accessKeyId: { @@ -29,6 +31,15 @@ module S3Module { type: "password", required: true, }, + endpoint: { + type: "string", + required: false, + }, + signatureVersion: { + type: "string", + required: false, + default: "v4" + }, }, query: { read: { @@ -46,16 +57,16 @@ module S3Module { class S3Integration implements IntegrationBase { private readonly config: S3Config private client: any - private connectionPromise: Promise constructor(config: S3Config) { this.config = config - this.connectionPromise = this.connect() - this.client = new AWS.S3() - } + if (this.config.endpoint) { + this.config.s3ForcePathStyle = true + } else { + delete this.config.endpoint + } - async connect() { - AWS.config.update(this.config) + this.client = new AWS.S3(this.config) } async read(query: { bucket: string }) { diff --git a/packages/server/src/integrations/utils.ts b/packages/server/src/integrations/utils.ts index b1b1faf776..c0c5877c7a 100644 --- a/packages/server/src/integrations/utils.ts +++ b/packages/server/src/integrations/utils.ts @@ -1,12 +1,8 @@ import { SourceNames, SqlQuery } from "../definitions/datasource" import { Datasource, Table } from "../definitions/common" - -const { DocumentTypes, SEPARATOR } = require("../db/utils") -const { - FieldTypes, - BuildSchemaErrors, - InvalidColumns, -} = require("../constants") +import { SourceNames } from "../definitions/datasource" +import { DocumentTypes, SEPARATOR } from "../db/utils" +import { FieldTypes, BuildSchemaErrors, InvalidColumns } from "../constants" const DOUBLE_SEPARATOR = `${SEPARATOR}${SEPARATOR}` const ROW_ID_REGEX = /^\[.*]$/g @@ -163,7 +159,12 @@ function copyExistingPropsOver( if (!existingTableSchema.hasOwnProperty(key)) { continue } - if (existingTableSchema[key].type === "link") { + if ( + existingTableSchema[key].type === FieldTypes.LINK || + existingTableSchema[key].type === FieldTypes.OPTIONS || + ((!table.schema[key] || table.schema[key].type === FieldTypes.NUMBER) && + existingTableSchema[key].type === FieldTypes.BOOLEAN) + ) { table.schema[key] = existingTableSchema[key] } } diff --git a/packages/server/src/utilities/index.js b/packages/server/src/utilities/index.js index 77d3d46282..0dba11141c 100644 --- a/packages/server/src/utilities/index.js +++ b/packages/server/src/utilities/index.js @@ -34,7 +34,7 @@ exports.checkSlashesInUrl = url => { * @return {string} The base URL of the object store (MinIO or S3). */ exports.objectStoreUrl = () => { - if (env.SELF_HOSTED) { + if (env.SELF_HOSTED || env.MINIO_URL) { // can use a relative url for this as all goes through the proxy (this is hosted in minio) return OBJ_STORE_DIRECTORY } else { diff --git a/packages/string-templates/package.json b/packages/string-templates/package.json index da61e06da9..bbfa64d01a 100644 --- a/packages/string-templates/package.json +++ b/packages/string-templates/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/string-templates", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "description": "Handlebars wrapper for Budibase templating.", "main": "src/index.cjs", "module": "dist/bundle.mjs", diff --git a/packages/worker/package.json b/packages/worker/package.json index aff597b925..d6bb45f9c4 100644 --- a/packages/worker/package.json +++ b/packages/worker/package.json @@ -1,7 +1,7 @@ { "name": "@budibase/worker", "email": "hi@budibase.com", - "version": "1.0.27-alpha.13", + "version": "1.0.27-alpha.17", "description": "Budibase background service", "main": "src/index.js", "repository": { @@ -29,8 +29,8 @@ "author": "Budibase", "license": "GPL-3.0", "dependencies": { - "@budibase/backend-core": "^1.0.27-alpha.13", - "@budibase/string-templates": "^1.0.27-alpha.13", + "@budibase/backend-core": "^1.0.27-alpha.17", + "@budibase/string-templates": "^1.0.27-alpha.17", "@koa/router": "^8.0.0", "@sentry/node": "^6.0.0", "@techpass/passport-openidconnect": "^0.3.0",