diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 5929a0540a..c33665c964 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -6,7 +6,7 @@ Welcome to the budibase CI pipelines directory. This document details what each ## All CI Pipelines ### Note -- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. +- When running workflow dispatch jobs, ensure you always run them off the `master` branch. It defaults to `develop`, so double check before running any jobs. The exception to this case is the `deploy-release` job which requires the develop branch. ### Standard CI Build Job (budibase_ci.yml) Triggers: @@ -115,4 +115,75 @@ This job is responsible for deploying to our production, cloud kubernetes enviro ### Rollback A Bad Cloud Deployment - Kick off cloud deploy job - Ensure you are running off master -- Enter the version number of the last known good version of budibase. For example `1.0.0` \ No newline at end of file +- Enter the version number of the last known good version of budibase. For example `1.0.0` + +## Pro + +### Installing Pro + +The pro package is always installed from source in our CI jobs. + +This is done to prevent pro needing to be published prior to CI runs in budiabse. This is required for two reasons: +- To reduce developer need to manually bump versions, i.e: + - release pro, bump pro dep in budibase, now ci can run successfully +- The cyclic dependency on backend-core, i.e: + - pro depends on backend-core + - server depends on pro + - backend-core lives in the monorepo, so it can't be released independently to be used in pro + - therefore the only option is to pull pro from source and release it as a part of the monorepo release, as if it were a mono package + +The install is performed using the same steps as local development, via the `yarn bootstrap` command, see the [Contributing Guide#Pro](../CONTRIBUTING.md#pro) + +The branch to install pro from can vary depending on ref of the commit that triggered the budibase CI job. This is done to enable branches which have changes in both the monorepo and the pro repo to have their CI pass successfully. + +This is done using the [pro/install.sh](../../scripts/pro/install.sh) script. The script will: +- Clone pro to it's default branch (`develop`) +- Check if the clone worked, on forked versions of budibase this will fail due to no access + - This is fine as the `yarn` command will install the version from NPM + - Community PRs should never touch pro so this will always work +- Checkout the `BRANCH` argument, if this fails fallback to `BASE_BRANCH` + - This enables the more complex case of a feature branch being merged to another feature branch, e.g. + - I am working on a branch `epic/stonks` which exists on budibase and pro. + - I want to merge a change to this branch in budibase from `feature/stonks-ui`, which only exists in budibase + - The base branch ensures that `epic/stonks` in pro will still be checked out for the CI run, rather than falling back to `develop` +- Run `yarn setup` to build and install dependencies + - `yarn` + - `yarn bootstrap` + - `yarn build` + - The will build .ts files, and also update the `main` and `types` of `package.json` to point to `dist` rather than src + - The build command will only ever work in CI, it is prevented in local dev + +#### `BRANCH` and `BASE_BRANCH` arguments +These arguments are supplied by the various budibase build and release pipelines +- `budibase_ci` + - `BRANCH: ${{ github.event.pull_request.head.ref }}` -> The branch being merged + - `BASE_BRANCH: ${{ github.event.pull_request.base.ref}}` -> The base branch +- `release-develop` + - `BRANCH: develop` -> always use the `develop` branch in pro +- `release` + - `BRANCH: master` -> always use the `master` branch in pro + + +### Releasing Pro +After budibase dependencies have been released we will release the new version of pro to match the release version of budibase dependencies. This is to ensure that we are always keeping the version of `backend-core` in sync in the pro package and in budibase packages. Without this we could run into scenarios where different versions are being used when installed via `yarn` inside the docker images, creating very difficult to debug cases. + +Pro is released using the [pro/release.sh](../../scripts/pro/release.sh) script. The script will: +- Inspect the `VERSION` from the `lerna.json` file in budibase +- Determine whether to use the `latest` or `develop` tag based on the command argument +- Go to pro directory + - install npm creds + - update the version of `backend-core` to be `VERSION`, the version just released by lerna + - publish to npm. Uses a `lerna publish` command, pro itself is a mono repo. + - force the version to be the same as `VERSION` to keep pro and budibase in sync + - reverts the changes to `main` and `types` in `package.json` that were made by the build step, to point back to source + - commit & push: `Prep next development iteration` +- Go to budibase + - Update to the new version of pro in `server` and `worker` so the latest pro version is used in the docker builds + - commit & push: `Update pro version to $VERSION` + + +#### `COMMAND` argument +This argument is supplied by the existing `release` and `release:develop` budibase commands, which invoke the pro release +- `release` will supply no command and default to use `latest` +- `release:develop` will supply `develop` + diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 961e13ee33..531ed05749 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -190,6 +190,21 @@ yarn mode:account ``` ### CI An overview of the CI pipelines can be found [here](./workflows/README.md) + +### Pro + +@budibase/pro is the closed source package that supports licensed features in budibase. By default the package will be pulled from NPM and will not normally need to be touched in local development. If you require to update code inside the pro package it can be cloned to the same root level as budibase, e.g. + +``` +. +|_ budibase +|_ budibase-pro +``` + +Note that only budibase maintainers will be able to access the pro repo. + +The `yarn bootstrap` command can be used to replace the NPM supplied dependency with the local source aware version. This is achieved using the `yarn link` command. To see specifically how dependencies are linked see [scripts/link-dependencies.sh](../scripts/link-dependencies.sh). The same link script is used to link dependencies to account-portal in local dev. + ### Troubleshooting Sometimes, things go wrong. This can be due to incompatible updates on the budibase platform. To clear down your development environment and start again follow **Step 6. Cleanup**, then proceed from **Step 3. Install and Build** in the setup guide above to create a fresh Budibase installation. diff --git a/hosting/letsencrypt/certificate-renew.sh b/hosting/letsencrypt/certificate-renew.sh new file mode 100644 index 0000000000..df88b44322 --- /dev/null +++ b/hosting/letsencrypt/certificate-renew.sh @@ -0,0 +1,13 @@ +#!/bin/bash +CUSTOM_DOMAIN="$1" + +if [[ ! -z "${CUSTOM_DOMAIN}" ]]; then + certbot certonly --webroot --webroot-path="/var/www/html" \ + --register-unsafely-without-email \ + --domains $CUSTOM_DOMAIN \ + --rsa-key-size 4096 \ + --agree-tos \ + --force-renewal + + nginx -s reload +fi diff --git a/hosting/letsencrypt/certificate-request.sh b/hosting/letsencrypt/certificate-request.sh new file mode 100644 index 0000000000..d029da265f --- /dev/null +++ b/hosting/letsencrypt/certificate-request.sh @@ -0,0 +1,24 @@ +#!/bin/bash +CUSTOM_DOMAIN="$1" +# Request from Lets Encrypt +certbot certonly --webroot --webroot-path="/var/www/html" \ + --register-unsafely-without-email \ + --domains $CUSTOM_DOMAIN \ + --rsa-key-size 4096 \ + --agree-tos \ + --force-renewal + +if (($? != 0)); then + echo "ERROR: certbot request failed for $CUSTOM_DOMAIN use http on port 80 - exiting" + nginx -s stop + exit 1 +else + cp /app/letsencrypt/options-ssl-nginx.conf /etc/letsencrypt/options-ssl-nginx.conf + cp /app/letsencrypt/ssl-dhparams.pem /etc/letsencrypt/ssl-dhparams.pem + cp /app/letsencrypt/nginx-ssl.conf /etc/nginx/sites-available/nginx-ssl.conf + sed -i 's/CUSTOM_DOMAIN/$CUSTOM_DOMAIN/g' /etc/nginx/sites-available/nginx-ssl.conf + ln -s /etc/nginx/sites-available/nginx-ssl.conf /etc/nginx/sites-enabled/nginx-ssl.conf + + echo "INFO: restart nginx after certbot request" + nginx -s reload +fi diff --git a/hosting/letsencrypt/nginx-ssl.conf b/hosting/letsencrypt/nginx-ssl.conf new file mode 100644 index 0000000000..c1a1d91917 --- /dev/null +++ b/hosting/letsencrypt/nginx-ssl.conf @@ -0,0 +1,94 @@ +server { + listen 443 ssl default_server; + listen [::]:443 ssl default_server; + server_name _; + ssl_certificate /etc/letsencrypt/live/CUSTOM_DOMAIN/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/CUSTOM_DOMAIN/privkey.pem; + include /etc/letsencrypt/options-ssl-nginx.conf; + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; + client_max_body_size 1000m; + ignore_invalid_headers off; + proxy_buffering off; + # port_in_redirect off; + + location ^~ /.well-known/acme-challenge/ { + default_type "text/plain"; + root /var/www/html; + break; + } + location = /.well-known/acme-challenge/ { + return 404; + } + + location /app { + proxy_pass http://127.0.0.1:4001; + } + + location = / { + proxy_pass http://127.0.0.1:4001; + } + + location ~ ^/(builder|app_) { + proxy_http_version 1.1; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_pass http://127.0.0.1:4001; + } + + location ~ ^/api/(system|admin|global)/ { + proxy_pass http://127.0.0.1:4002; + } + + location /worker/ { + proxy_pass http://127.0.0.1:4002; + rewrite ^/worker/(.*)$ /$1 break; + } + + location /api/ { + # calls to the API are rate limited with bursting + limit_req zone=ratelimit burst=20 nodelay; + + # 120s timeout on API requests + proxy_read_timeout 120s; + proxy_connect_timeout 120s; + proxy_send_timeout 120s; + + proxy_http_version 1.1; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + + proxy_pass http://127.0.0.1:4001; + } + + location /db/ { + proxy_pass http://127.0.0.1:5984; + rewrite ^/db/(.*)$ /$1 break; + } + + location / { + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + proxy_connect_timeout 300; + proxy_http_version 1.1; + proxy_set_header Connection ""; + chunked_transfer_encoding off; + proxy_pass http://127.0.0.1:9000; + } + + client_header_timeout 60; + client_body_timeout 60; + keepalive_timeout 60; + + # gzip + gzip on; + gzip_vary on; + gzip_proxied any; + gzip_comp_level 6; + gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml; +} diff --git a/hosting/letsencrypt/options-ssl-nginx.conf b/hosting/letsencrypt/options-ssl-nginx.conf new file mode 100644 index 0000000000..52fdfde245 --- /dev/null +++ b/hosting/letsencrypt/options-ssl-nginx.conf @@ -0,0 +1,13 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +ssl_session_cache shared:le_nginx_SSL:10m; +ssl_session_timeout 1440m; + +ssl_protocols TLSv1.2 TLSv1.3; +ssl_prefer_server_ciphers off; + +ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"; diff --git a/hosting/letsencrypt/ssl-dhparams.pem b/hosting/letsencrypt/ssl-dhparams.pem new file mode 100644 index 0000000000..088f9673dc --- /dev/null +++ b/hosting/letsencrypt/ssl-dhparams.pem @@ -0,0 +1,8 @@ +-----BEGIN DH PARAMETERS----- +MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz ++8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a +87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7 +YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi +7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD +ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg== +-----END DH PARAMETERS----- \ No newline at end of file diff --git a/hosting/scripts/healthcheck.sh b/hosting/scripts/healthcheck.sh new file mode 100644 index 0000000000..fa6f511eb9 --- /dev/null +++ b/hosting/scripts/healthcheck.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +healthy=true + +if [[ $(curl -Lfk -s -w "%{http_code}\n" http://localhost/ -o /dev/null) -ne 200 ]]; then + echo 'ERROR: Budibase is not running'; + healthy=false +fi + +if [[ $(curl -s -w "%{http_code}\n" http://localhost:4001/health -o /dev/null) -ne 200 ]]; then + echo 'ERROR: Budibase backend is not running'; + healthy=false +fi + +if [[ $(curl -s -w "%{http_code}\n" http://localhost:4002/health -o /dev/null) -ne 200 ]]; then + echo 'ERROR: Budibase worker is not running'; + healthy=false +fi + +if [[ $(curl -s -w "%{http_code}\n" http://localhost:5984/ -o /dev/null) -ne 200 ]]; then + echo 'ERROR: CouchDB is not running'; + healthy=false +fi +if [[ $(redis-cli -a $REDIS_PASSWORD --no-auth-warning ping) != 'PONG' ]]; then + echo 'ERROR: Redis is down'; + healthy=false +fi +# mino, clouseau, + +if [ $healthy == true ]; then + exit 0 +else + exit 1 +fi diff --git a/hosting/single/Dockerfile b/hosting/single/Dockerfile index f0df9373c5..24e90fc818 100644 --- a/hosting/single/Dockerfile +++ b/hosting/single/Dockerfile @@ -1,7 +1,7 @@ FROM node:14-slim as build # install node-gyp dependencies -RUN apt-get update && apt-get install -y --no-install-recommends g++ make python +RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends apt-utils cron g++ make python # add pin script WORKDIR / @@ -20,32 +20,36 @@ RUN node /pinVersions.js && yarn && yarn build && /cleanup.sh FROM couchdb:3.2.1 +ARG TARGETARCH amd64 + COPY --from=build /app /app COPY --from=build /worker /worker -ENV DEPLOYMENT_ENVIRONMENT=docker \ - POSTHOG_TOKEN=phc_fg5I3nDOf6oJVMHSaycEhpPdlgS8rzXG2r6F2IpxCHS \ +ENV \ + APP_PORT=4001 \ + ARCHITECTURE=amd \ + BUDIBASE_ENVIRONMENT=PRODUCTION \ + CLUSTER_PORT=80 \ COUCHDB_PASSWORD=budibase \ COUCHDB_USER=budibase \ COUCH_DB_URL=http://budibase:budibase@localhost:5984 \ - BUDIBASE_ENVIRONMENT=PRODUCTION \ - MINIO_URL=http://localhost:9000 \ - REDIS_URL=localhost:6379 \ - WORKER_URL=http://localhost:4002 \ + CUSTOM_DOMAIN=budi001.custom.com \ + DEPLOYMENT_ENVIRONMENT=docker \ INTERNAL_API_KEY=budibase \ JWT_SECRET=testsecret \ MINIO_ACCESS_KEY=budibase \ MINIO_SECRET_KEY=budibase \ - SELF_HOSTED=1 \ - CLUSTER_PORT=10000 \ + MINIO_URL=http://localhost:9000 \ + POSTHOG_TOKEN=phc_fg5I3nDOf6oJVMHSaycEhpPdlgS8rzXG2r6F2IpxCHS \ REDIS_PASSWORD=budibase \ - ARCHITECTURE=amd \ - APP_PORT=4001 \ - WORKER_PORT=4002 + REDIS_URL=localhost:6379 \ + SELF_HOSTED=1 \ + WORKER_PORT=4002 \ + WORKER_URL=http://localhost:4002 # install base dependencies RUN apt-get update && \ - apt-get install software-properties-common wget -y && \ + apt-get install -y software-properties-common wget nginx && \ apt-add-repository 'deb http://security.debian.org/debian-security stretch/updates main' && \ apt-get update @@ -53,20 +57,19 @@ RUN apt-get update && \ WORKDIR /nodejs RUN curl -sL https://deb.nodesource.com/setup_16.x -o /tmp/nodesource_setup.sh && \ bash /tmp/nodesource_setup.sh && \ - apt-get install libaio1 nodejs nginx openjdk-8-jdk redis-server unzip -y && \ + apt-get install -y libaio1 nodejs nginx openjdk-8-jdk redis-server unzip && \ npm install --global yarn pm2 # setup nginx ADD hosting/single/nginx.conf /etc/nginx -RUN mkdir /etc/nginx/logs && \ - useradd www && \ - touch /etc/nginx/logs/error.log && \ - touch /etc/nginx/logs/nginx.pid +RUN mkdir -p /var/log/nginx && \ + touch /var/log/nginx/error.log && \ + touch /var/run/nginx.pid WORKDIR / RUN mkdir -p scripts/integrations/oracle ADD packages/server/scripts/integrations/oracle scripts/integrations/oracle -RUN /bin/bash -e ./scripts/integrations/oracle/instantclient/linux/x86-64/install.sh +RUN /bin/bash -e ./scripts/integrations/oracle/instantclient/linux/install.sh # setup clouseau WORKDIR / @@ -87,20 +90,41 @@ ADD hosting/single/vm.args ./etc/ # setup minio WORKDIR /minio -RUN wget https://dl.min.io/server/minio/release/linux-${ARCHITECTURE}64/minio && chmod +x minio +ADD scripts/install-minio.sh ./install.sh +RUN chmod +x install.sh && ./install.sh # setup runner file WORKDIR / ADD hosting/single/runner.sh . RUN chmod +x ./runner.sh +ADD hosting/scripts/healthcheck.sh . +RUN chmod +x ./healthcheck.sh # cleanup cache RUN yarn cache clean -f -EXPOSE 10000 +EXPOSE 80 +EXPOSE 443 VOLUME /opt/couchdb/data VOLUME /minio +# setup letsencrypt certificate +RUN apt-get install -y certbot python3-certbot-nginx +ADD hosting/letsencrypt /app/letsencrypt +RUN chmod +x /app/letsencrypt/certificate-request.sh /app/letsencrypt/certificate-renew.sh +# Remove cached files +RUN rm -rf \ + /root/.cache \ + /root/.npm \ + /root/.pip \ + /usr/local/share/doc \ + /usr/share/doc \ + /usr/share/man \ + /var/lib/apt/lists/* \ + /tmp/* + +HEALTHCHECK --interval=15s --timeout=15s --start-period=45s CMD "/healthcheck.sh" + # must set this just before running ENV NODE_ENV=production WORKDIR / diff --git a/hosting/single/README.md b/hosting/single/README.md new file mode 100644 index 0000000000..d62359a628 --- /dev/null +++ b/hosting/single/README.md @@ -0,0 +1,105 @@ +# Docker Single Image for Budibase + +## Overview +As an alternative to running several docker containers via docker-compose, the files under ./hosting/single can be used to build a docker image containing all of the Budibase components (minio, couch, clouseau etc). +We call this the 'single image' container as the Dockerfile adds all the components to a single docker image. + + +## Usage + +- Amend Environment Variables +- Build Requirements +- Build the Image +- Run the Container + +### Amend Environment Variables + +Edit the Dockerfile in this directory amending the environment variables to suit your usage. Pay particular attention to changing passwords. +The CUSTOM_DOMAIN variable will be used to request a certificate from LetsEncrypt and if successful you can point traffic to port 443. If you choose to use the CUSTOM_DOMAIN variable ensure that the DNS for your custom domain points to the public IP address where you are running Budibase - otherwise the certificate issuance will fail. +If you have other arrangements for a proxy in front of the single image container you can omit the CUSTOM_DOMAIN environment variable and the request to LetsEncrypt will be skipped. You can then point traffic to port 80. + +### Build Requirements +We would suggest building the image with 6GB of RAM and 20GB of free disk space for build artifacts. The resulting image size will use approx 2GB of disk space. + +### Build the Image +The guidance below is based on building the Budibase single image on Debian 11. If you use another distro or OS you will need to amend the commands to suit. +Install Node +Budibase requires a recent version of node (14+) than is in the base Debian repos so: + +``` +curl -sL https://deb.nodesource.com/setup_16.x | sudo bash - +apt install -y nodejs +node -v +``` +Install yarn and lerna: +``` +npm install -g yarn jest lerna +``` +Install Docker +``` +apt install -y docker.io +apt install -y python3-pip +pip3 install docker-compose +``` +Check the versions of each installed version. This process was tested with the version numbers below so YMMV using anything else: + +- Docker: 20.10.5 +- docker-compose: 1.29.2 +- node: 16.15.1 +- yarn: 1.22.19 +- lerna: 5.1.4 + +Clone the Budibase repo +``` +git clone https://github.com/Budibase/budibase.git +cd budibase +``` +Node setup: +``` +node ./hosting/scripts/setup.js +yarn +yarn bootstrap +yarn build +``` + +Build the image from the Dockerfile: + +``` +yarn build:docker:single +``` +If the docker build step fails run that step again manually with: +``` +docker build --no-cache -t budibase:latest -f ./hosting/single/Dockerfile . +``` + +### Run the Container +``` +docker run -d -p 80:80 -p 443:443 --name budibase budibase:latest +``` +Where: +- -d runs the container in detached mode +- -p forwards ports from your host to the ports inside the container. If you are already using port 80 on your host for something else you can try running with an alternative port e.g. `-p 8080:80` +- --name is the name for the container as shown in `docker ps` and can be used with other docker commands e.g. `docker restart budibase` + +When the container runs you should be able to access the container over http at your host address e.g. http://1.2.3.4/ or using your custom domain e.g. https://my.custom.domain/ + +When the Budibase UI appears you will be prompted to create an account to get started. + +### Check +There are many things that could go wrong so if your container is not building or running as expected please check the following before opening a support issue. +Verify the healthcheck status of the container: +``` +docker ps +``` +Check the container logs: +``` +docker logs budibase +``` + + +### Support +This single image build is still a work-in-progress so if you open an issue please provide the following information: +- The OS and OS version you are building on +- The versions you are using of docker, docker-compose, yarn, node, lerna +- For build errors please provide zipped output +- For container errors please provide zipped container logs diff --git a/hosting/single/nginx.conf b/hosting/single/nginx.conf index 86938ced4e..42d20dd14a 100644 --- a/hosting/single/nginx.conf +++ b/hosting/single/nginx.conf @@ -1,6 +1,6 @@ -user www www; -error_log /etc/nginx/logs/error.log; -pid /etc/nginx/logs/nginx.pid; +user www-data www-data; +error_log /var/log/nginx/error.log; +pid /var/run/nginx.pid; worker_processes auto; worker_rlimit_nofile 8192; @@ -33,14 +33,23 @@ http { } server { - listen 10000 default_server; - listen [::]:10000 default_server; + listen 80 default_server; + listen [::]:80 default_server; server_name _; client_max_body_size 1000m; ignore_invalid_headers off; proxy_buffering off; # port_in_redirect off; + location ^~ /.well-known/acme-challenge/ { + default_type "text/plain"; + root /var/www/html; + break; + } + location = /.well-known/acme-challenge/ { + return 404; + } + location /app { proxy_pass http://127.0.0.1:4001; } diff --git a/hosting/single/runner.sh b/hosting/single/runner.sh index fab8431796..6f3d247842 100644 --- a/hosting/single/runner.sh +++ b/hosting/single/runner.sh @@ -2,6 +2,15 @@ redis-server --requirepass $REDIS_PASSWORD & /opt/clouseau/bin/clouseau & /minio/minio server /minio & /docker-entrypoint.sh /opt/couchdb/bin/couchdb & +/etc/init.d/nginx restart +if [[ ! -z "${CUSTOM_DOMAIN}" ]]; then + # Add monthly cron job to renew certbot certificate + echo -n "* * 2 * * root exec /app/letsencrypt/certificate-renew.sh ${CUSTOM_DOMAIN}" >> /etc/cron.d/certificate-renew + chmod +x /etc/cron.d/certificate-renew + # Request the certbot certificate + /app/letsencrypt/certificate-request.sh ${CUSTOM_DOMAIN} +fi + /etc/init.d/nginx restart pushd app pm2 start --name app "yarn run:docker" @@ -10,7 +19,6 @@ pushd worker pm2 start --name worker "yarn run:docker" popd sleep 10 -URL=http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@localhost:5984 -curl -X PUT ${URL}/_users -curl -X PUT ${URL}/_replicator -sleep infinity \ No newline at end of file +curl -X PUT ${COUCH_DB_URL}/_users +curl -X PUT ${COUCH_DB_URL}/_replicator +sleep infinity diff --git a/lerna.json b/lerna.json index b74beefaa1..8348f248e6 100644 --- a/lerna.json +++ b/lerna.json @@ -1,5 +1,5 @@ { - "version": "1.0.207-alpha.6", + "version": "1.0.207-alpha.10", "npmClient": "yarn", "packages": [ "packages/*" diff --git a/package.json b/package.json index 014cafcfc1..9c35af497f 100644 --- a/package.json +++ b/package.json @@ -62,6 +62,7 @@ "build:docker:develop": "node scripts/pinVersions && lerna run build:docker && npm run build:docker:proxy:compose && cd hosting/scripts/linux/ && ./release-to-docker-hub.sh develop && cd -", "build:docker:airgap": "node hosting/scripts/airgapped/airgappedDockerBuild", "build:digitalocean": "cd hosting/digitalocean && ./build.sh && cd -", + "build:docker:single:multiarch": "docker buildx build --platform linux/arm64,linux/amd64 -f hosting/single/Dockerfile -t budibase:latest .", "build:docker:single:image": "docker build -f hosting/single/Dockerfile -t budibase:latest .", "build:docker:single": "lerna run build && lerna run predocker && npm run build:docker:single:image", "build:docs": "lerna run build:docs", diff --git a/packages/backend-core/package.json b/packages/backend-core/package.json index 7d9348ddd4..1098a24469 100644 --- a/packages/backend-core/package.json +++ b/packages/backend-core/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/backend-core", - "version": "1.0.207-alpha.6", + "version": "1.0.207-alpha.10", "description": "Budibase backend core libraries used in server and worker", "main": "dist/src/index.js", "types": "dist/src/index.d.ts", @@ -20,7 +20,7 @@ "test:watch": "jest --watchAll" }, "dependencies": { - "@budibase/types": "^1.0.207-alpha.6", + "@budibase/types": "^1.0.207-alpha.10", "@techpass/passport-openidconnect": "0.3.2", "aws-sdk": "2.1030.0", "bcrypt": "5.0.1", diff --git a/packages/backend-core/src/db/pouch.js b/packages/backend-core/src/db/pouch.js index 76390ac644..59b7ff8ae7 100644 --- a/packages/backend-core/src/db/pouch.js +++ b/packages/backend-core/src/db/pouch.js @@ -1,21 +1,42 @@ const PouchDB = require("pouchdb") const env = require("../environment") -function getUrlInfo() { - let url = env.COUCH_DB_URL - let username, password, host - const [protocol, rest] = url.split("://") - if (url.includes("@")) { - const hostParts = rest.split("@") - host = hostParts[1] - const authParts = hostParts[0].split(":") - username = authParts[0] - password = authParts[1] - } else { - host = rest +exports.getUrlInfo = (url = env.COUCH_DB_URL) => { + let cleanUrl, username, password, host + if (url) { + // Ensure the URL starts with a protocol + const protoRegex = /^https?:\/\//i + if (!protoRegex.test(url)) { + url = `http://${url}` + } + + // Split into protocol and remainder + const split = url.split("://") + const protocol = split[0] + const rest = split.slice(1).join("://") + + // Extract auth if specified + if (url.includes("@")) { + // Split into host and remainder + let parts = rest.split("@") + host = parts[parts.length - 1] + let auth = parts.slice(0, -1).join("@") + + // Split auth into username and password + if (auth.includes(":")) { + const authParts = auth.split(":") + username = authParts[0] + password = authParts.slice(1).join(":") + } else { + username = auth + } + } else { + host = rest + } + cleanUrl = `${protocol}://${host}` } return { - url: `${protocol}://${host}`, + url: cleanUrl, auth: { username, password, @@ -24,7 +45,7 @@ function getUrlInfo() { } exports.getCouchInfo = () => { - const urlInfo = getUrlInfo() + const urlInfo = exports.getUrlInfo() let username let password if (env.COUCH_DB_USERNAME) { diff --git a/packages/backend-core/src/db/tests/pouch.spec.js b/packages/backend-core/src/db/tests/pouch.spec.js new file mode 100644 index 0000000000..30cdd0f5ec --- /dev/null +++ b/packages/backend-core/src/db/tests/pouch.spec.js @@ -0,0 +1,62 @@ +require("../../../tests/utilities/TestConfiguration") +const getUrlInfo = require("../pouch").getUrlInfo + +describe("pouch", () => { + describe("Couch DB URL parsing", () => { + it("should handle a null Couch DB URL", () => { + const info = getUrlInfo(null) + expect(info.url).toBeUndefined() + expect(info.auth.username).toBeUndefined() + }) + it("should be able to parse a basic Couch DB URL", () => { + const info = getUrlInfo("http://host.com") + expect(info.url).toBe("http://host.com") + expect(info.auth.username).toBeUndefined() + }) + it("should be able to parse a Couch DB basic URL with HTTPS", () => { + const info = getUrlInfo("https://host.com") + expect(info.url).toBe("https://host.com") + expect(info.auth.username).toBeUndefined() + }) + it("should be able to parse a basic Couch DB URL with a custom port", () => { + const info = getUrlInfo("https://host.com:1234") + expect(info.url).toBe("https://host.com:1234") + expect(info.auth.username).toBeUndefined() + }) + it("should be able to parse a Couch DB URL with auth", () => { + const info = getUrlInfo("https://user:pass@host.com:1234") + expect(info.url).toBe("https://host.com:1234") + expect(info.auth.username).toBe("user") + expect(info.auth.password).toBe("pass") + }) + it("should be able to parse a Couch DB URL with auth and special chars", () => { + const info = getUrlInfo("https://user:s:p@s://@://:d@;][~s@host.com:1234") + expect(info.url).toBe("https://host.com:1234") + expect(info.auth.username).toBe("user") + expect(info.auth.password).toBe("s:p@s://@://:d@;][~s") + }) + it("should be able to parse a Couch DB URL without a protocol", () => { + const info = getUrlInfo("host.com:1234") + expect(info.url).toBe("http://host.com:1234") + expect(info.auth.username).toBeUndefined() + }) + it("should be able to parse a Couch DB URL with auth and without a protocol", () => { + const info = getUrlInfo("user:s:p@s://@://:d@;][~s@host.com:1234") + expect(info.url).toBe("http://host.com:1234") + expect(info.auth.username).toBe("user") + expect(info.auth.password).toBe("s:p@s://@://:d@;][~s") + }) + it("should be able to parse a Couch DB URL with only username auth", () => { + const info = getUrlInfo("https://user@host.com:1234") + expect(info.url).toBe("https://host.com:1234") + expect(info.auth.username).toBe("user") + expect(info.auth.password).toBeUndefined() + }) + it("should be able to parse a Couch DB URL with only username auth and without a protocol", () => { + const info = getUrlInfo("user@host.com:1234") + expect(info.url).toBe("http://host.com:1234") + expect(info.auth.username).toBe("user") + expect(info.auth.password).toBeUndefined() + }) + }) +}) diff --git a/packages/bbui/package.json b/packages/bbui/package.json index 17592216d7..ffffb267c7 100644 --- a/packages/bbui/package.json +++ b/packages/bbui/package.json @@ -1,7 +1,7 @@ { "name": "@budibase/bbui", "description": "A UI solution used in the different Budibase projects.", - "version": "1.0.207-alpha.6", + "version": "1.0.207-alpha.10", "license": "MPL-2.0", "svelte": "src/index.js", "module": "dist/bbui.es.js", @@ -38,7 +38,7 @@ ], "dependencies": { "@adobe/spectrum-css-workflow-icons": "^1.2.1", - "@budibase/string-templates": "^1.0.207-alpha.6", + "@budibase/string-templates": "^1.0.207-alpha.10", "@spectrum-css/actionbutton": "^1.0.1", "@spectrum-css/actiongroup": "^1.0.1", "@spectrum-css/avatar": "^3.0.2", diff --git a/packages/bbui/src/Typography/Detail.svelte b/packages/bbui/src/Typography/Detail.svelte index bb5c78c11e..76437ffb3c 100644 --- a/packages/bbui/src/Typography/Detail.svelte +++ b/packages/bbui/src/Typography/Detail.svelte @@ -1,9 +1,7 @@ diff --git a/packages/builder/cypress/support/commands.js b/packages/builder/cypress/support/commands.js index e638eb6cbb..29aabc4611 100644 --- a/packages/builder/cypress/support/commands.js +++ b/packages/builder/cypress/support/commands.js @@ -361,7 +361,7 @@ Cypress.Commands.add("createTable", (tableName, initialTable) => { cy.get(`[data-cy="new-table"]`).click() } cy.wait(5000) - cy.get(".spectrum-Dialog-grid") + cy.get(".item") .contains("Budibase DB") .click({ force: true }) .then(() => { diff --git a/packages/builder/package.json b/packages/builder/package.json index e096675be4..07d76f2693 100644 --- a/packages/builder/package.json +++ b/packages/builder/package.json @@ -1,6 +1,6 @@ { "name": "@budibase/builder", - "version": "1.0.207-alpha.6", + "version": "1.0.207-alpha.10", "license": "GPL-3.0", "private": true, "scripts": { @@ -69,10 +69,10 @@ } }, "dependencies": { - "@budibase/bbui": "^1.0.207-alpha.6", - "@budibase/client": "^1.0.207-alpha.6", - "@budibase/frontend-core": "^1.0.207-alpha.6", - "@budibase/string-templates": "^1.0.207-alpha.6", + "@budibase/bbui": "^1.0.207-alpha.10", + "@budibase/client": "^1.0.207-alpha.10", + "@budibase/frontend-core": "^1.0.207-alpha.10", + "@budibase/string-templates": "^1.0.207-alpha.10", "@sentry/browser": "5.19.1", "@spectrum-css/page": "^3.0.1", "@spectrum-css/vars": "^3.0.1", diff --git a/packages/builder/src/builderStore/store/frontend.js b/packages/builder/src/builderStore/store/frontend.js index 641e2c2472..ec810e5c31 100644 --- a/packages/builder/src/builderStore/store/frontend.js +++ b/packages/builder/src/builderStore/store/frontend.js @@ -190,6 +190,7 @@ export const getFrontendStore = () => { // Build array of promises to speed up bulk deletions const promises = [] + let deleteUrls = [] screensToDelete.forEach(screen => { // Delete the screen promises.push( @@ -199,14 +200,10 @@ export const getFrontendStore = () => { }) ) // Remove links to this screen - promises.push( - store.actions.components.links.delete( - screen.routing.route, - screen.props._instanceName - ) - ) + deleteUrls.push(screen.routing.route) }) + promises.push(store.actions.links.delete(deleteUrls)) await Promise.all(promises) const deletedIds = screensToDelete.map(screen => screen._id) store.update(state => { @@ -578,89 +575,38 @@ export const getFrontendStore = () => { }) await store.actions.preview.saveSelected() }, - links: { - save: async (url, title) => { - const layout = get(mainLayout) - if (!layout) { - return - } + }, + links: { + save: async (url, title) => { + const layout = get(mainLayout) + if (!layout) { + return + } - // Add link setting to main layout - if (layout.props._component.endsWith("layout")) { - // If using a new SDK, add to the layout component settings - if (!layout.props.links) { - layout.props.links = [] - } - layout.props.links.push({ - text: title, - url, - }) - } else { - // If using an old SDK, add to the navigation component - // TODO: remove this when we can assume everyone has updated - const nav = findComponentType( - layout.props, - "@budibase/standard-components/navigation" - ) - if (!nav) { - return - } + // Add link setting to main layout + if (!layout.props.links) { + layout.props.links = [] + } + layout.props.links.push({ + text: title, + url, + }) - let newLink - if (nav._children && nav._children.length) { - // Clone an existing link if one exists - newLink = cloneDeep(nav._children[0]) + await store.actions.layouts.save(layout) + }, + delete: async urls => { + const layout = get(mainLayout) + if (!layout?.props.links?.length) { + return + } - // Set our new props - newLink._id = Helpers.uuid() - newLink._instanceName = `${title} Link` - newLink.url = url - newLink.text = title - } else { - // Otherwise create vanilla new link - newLink = { - ...store.actions.components.createInstance("link"), - url, - text: title, - _instanceName: `${title} Link`, - } - nav._children = [...nav._children, newLink] - } - } + // Filter out the URLs to delete + urls = Array.isArray(urls) ? urls : [urls] + layout.props.links = layout.props.links.filter( + link => !urls.includes(link.url) + ) - // Save layout - await store.actions.layouts.save(layout) - }, - delete: async (url, title) => { - const layout = get(mainLayout) - if (!layout) { - return - } - - // Add link setting to main layout - if (layout.props._component.endsWith("layout")) { - // If using a new SDK, add to the layout component settings - layout.props.links = layout.props.links.filter( - link => !(link.text === title && link.url === url) - ) - } else { - // If using an old SDK, add to the navigation component - // TODO: remove this when we can assume everyone has updated - const nav = findComponentType( - layout.props, - "@budibase/standard-components/navigation" - ) - if (!nav) { - return - } - - nav._children = nav._children.filter( - child => !(child.url === url && child.text === title) - ) - } - // Save layout - await store.actions.layouts.save(layout) - }, + await store.actions.layouts.save(layout) }, }, settings: { diff --git a/packages/builder/src/builderStore/store/screenTemplates/newRowScreen.js b/packages/builder/src/builderStore/store/screenTemplates/newRowScreen.js index 2b9d2bc663..dd97c511e5 100644 --- a/packages/builder/src/builderStore/store/screenTemplates/newRowScreen.js +++ b/packages/builder/src/builderStore/store/screenTemplates/newRowScreen.js @@ -15,7 +15,7 @@ export default function (tables) { name: `${table.name} - New`, create: () => createScreen(table), id: NEW_ROW_TEMPLATE, - table: table.name, + table: table._id, } }) } diff --git a/packages/builder/src/builderStore/store/screenTemplates/rowDetailScreen.js b/packages/builder/src/builderStore/store/screenTemplates/rowDetailScreen.js index 8ab4a2bea7..a1916769c9 100644 --- a/packages/builder/src/builderStore/store/screenTemplates/rowDetailScreen.js +++ b/packages/builder/src/builderStore/store/screenTemplates/rowDetailScreen.js @@ -17,7 +17,7 @@ export default function (tables) { name: `${table.name} - Detail`, create: () => createScreen(table), id: ROW_DETAIL_TEMPLATE, - table: table.name, + table: table._id, } }) } diff --git a/packages/builder/src/builderStore/store/screenTemplates/rowListScreen.js b/packages/builder/src/builderStore/store/screenTemplates/rowListScreen.js index c369f99f68..39e88ae69e 100644 --- a/packages/builder/src/builderStore/store/screenTemplates/rowListScreen.js +++ b/packages/builder/src/builderStore/store/screenTemplates/rowListScreen.js @@ -10,7 +10,7 @@ export default function (tables) { name: `${table.name} - List`, create: () => createScreen(table), id: ROW_LIST_TEMPLATE, - table: table.name, + table: table._id, } }) } diff --git a/packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte b/packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte index 14adbfbd02..8d34c292f3 100644 --- a/packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte +++ b/packages/builder/src/components/backend/DatasourceNavigator/modals/CreateDatasourceModal.svelte @@ -5,12 +5,13 @@ Body, Layout, Detail, + Heading, notifications, } from "@budibase/bbui" import { onMount } from "svelte" import ICONS from "../icons" import { API } from "api" - import { IntegrationNames, IntegrationTypes } from "constants/backend" + import { IntegrationTypes } from "constants/backend" import CreateTableModal from "components/backend/TableNavigator/modals/CreateTableModal.svelte" import DatasourceConfigModal from "components/backend/DatasourceNavigator/modals/DatasourceConfigModal.svelte" import GoogleDatasourceConfigModal from "components/backend/DatasourceNavigator/modals/GoogleDatasourceConfigModal.svelte" @@ -118,7 +119,7 @@ - - All apps need data. You can connect to a data source below, or add data - to your app using Budibase's built-in database. - + + Get started with Budibase DB
selectIntegration(IntegrationTypes.INTERNAL)} class="item hoverable" > -
- - Budibase DB +
+ +
+ Budibase DB + Non-relational +
- -
- Connect to data source -
+ + Connect to an external data source
{#each Object.entries(integrations).filter(([key]) => key !== IntegrationTypes.INTERNAL) as [integrationType, schema]}
selectIntegration(integrationType)} class="item hoverable" > -
+
- - - {schema.name || IntegrationNames[integrationType]} +
+ {schema.friendlyName} + {#if schema.type} + {schema.type || ""} + {/if} +
{/each} @@ -178,13 +177,6 @@ diff --git a/packages/builder/src/components/design/NavigationPanel/DatasourceModal.svelte b/packages/builder/src/components/design/NavigationPanel/DatasourceModal.svelte index 1cb3856165..bd9b6a1741 100644 --- a/packages/builder/src/components/design/NavigationPanel/DatasourceModal.svelte +++ b/packages/builder/src/components/design/NavigationPanel/DatasourceModal.svelte @@ -14,14 +14,14 @@ let selectedScreens = [...initalScreens] const toggleScreenSelection = (table, datasource) => { - if (selectedScreens.find(s => s.table === table.name)) { + if (selectedScreens.find(s => s.table === table._id)) { selectedScreens = selectedScreens.filter( - screen => screen.table !== table.name + screen => screen.table !== table._id ) } else { let partialTemplates = getTemplates($store, $tables.list).reduce( (acc, template) => { - if (template.table === table.name) { + if (template.table === table._id) { template.datasource = datasource.name acc.push(template) } @@ -88,7 +88,7 @@
x.table === table.name + x => x.table === table._id )} on:click={() => toggleScreenSelection(table, datasource)} > @@ -102,8 +102,7 @@ {table.name} - - {#if selectedScreens.find(x => x.table === table.name)} + {#if selectedScreens.find(x => x.table === table._id)} @@ -116,7 +115,7 @@
x.table === datasource.entities[table_key].name + x => x.table === datasource.entities[table_key]._id )} on:click={() => toggleScreenSelection( @@ -134,8 +133,7 @@ {datasource.entities[table_key].name} - - {#if selectedScreens.find(x => x.table === datasource.entities[table_key].name)} + {#if selectedScreens.find(x => x.table === datasource.entities[table_key]._id)} diff --git a/packages/builder/src/components/design/NavigationPanel/ScreenWizard.svelte b/packages/builder/src/components/design/NavigationPanel/ScreenWizard.svelte index 0a3c9611bc..5f36034b93 100644 --- a/packages/builder/src/components/design/NavigationPanel/ScreenWizard.svelte +++ b/packages/builder/src/components/design/NavigationPanel/ScreenWizard.svelte @@ -66,7 +66,7 @@ // Add link in layout for list screens if (screen.props._instanceName.endsWith("List")) { - await store.actions.components.links.save( + await store.actions.links.save( screen.routing.route, screen.routing.route.split("/")[1] ) @@ -131,6 +131,7 @@ const screens = selectedTemplates.map(template => { let screenTemplate = template.create() screenTemplate.datasource = template.datasource + screenTemplate.autoTableId = template.table return screenTemplate }) await createScreens({ screens, screenAccessRole }) diff --git a/packages/builder/src/components/design/PropertiesPanel/PropertyControls/ButtonActionEditor/actions/ExportData.svelte b/packages/builder/src/components/design/PropertiesPanel/PropertyControls/ButtonActionEditor/actions/ExportData.svelte index 062b9abd4c..aa3bf2a36b 100644 --- a/packages/builder/src/components/design/PropertiesPanel/PropertyControls/ButtonActionEditor/actions/ExportData.svelte +++ b/packages/builder/src/components/design/PropertiesPanel/PropertyControls/ButtonActionEditor/actions/ExportData.svelte @@ -1,27 +1,18 @@