SOP of Downtime upgrade

This section introduces the downtime upgrade SOP for Univer backend services. If you are familiar with docker compose and K8s, Univer recommends you to develop a more suitable SOP for your production environment.

SOP of Downtime upgrade using docker compose

Preparation before upgrade

Custom configuration file Before preparing for the upgrade, you should check our version introduction to determine the version you want to upgrade to, and check our [production deployment(/guides/sheets/pro-features/server/deploy), which will introduce which version has added new configuration items and which version no longer supports old configuration items. According to the document, check what needs to be modified and what new configurations need to be customized in your old version custom configuration file .env.custom. After confirming, prepare a new custom configuration file .env.custom and keep the old version of .env.custom for safe rollback if need.
License file You also need to prepare your license file.
Upgrade notification Inform users of the expected downtime upgrade period so that they are prepared

Upgrade operation

Stop the old version of the service

Assumed that the old version of the Univer service is deployed in the directory univer-old.

prevent writing first

Note: Due to the special nature of collaborative editing, in order to ensure that all collaborative editing requests are saved normally, it is best not to directly stop the service, but to first stop the service after a period of write prohibition. You can use your own infrastructure to achieve this goal, or implement it through nginx in the univer service. The following introduces the way to implement it using nginx deployed by univer. Modify universe-old/nginx/universer.conf as follows, add the routing configuration location~ ^/universe-api {return 403;} to disable all new requests

server {
  listen 8000;
 
  client_max_body_size 100m;
 
  location ~ ^/universer-api {return 403;} # Add this sentence to disable all requests through nginx
 
  location / {
    proxy_pass http://universer;
  }
 
  location /universer-api/comb/connect {
    proxy_pass http://universer;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
  }
 
  location /universer-api/metrics {
    return 403;
  }
}

After modification, you need to let nginx reload this configuration to take effect. Execute the following command to reload the configuration: docker exec univer-lb nginx -s reload, where universe-lb is the default configuration of the nginx container name, if you have modified it, replace with your new container name.

prevent writing for a period of time, it is recommended at least 1 minute
Stop the old version service

Execute the command cd universer-old && bash run.sh stop to stop the old service

Data backup

Univer suggests that you backup important data before upgrading. The data that needs to be backed up includes units metadata in RDS. You can use the corresponding RDS data backup tool to backup it yourself. Data in Object Storage will only be added and not modified, so there is no need to backup.

Deploy the selected version

Execute the custom installation command bash -c "$(curl -fsSL https://get.univer.ai/product)" [-- version] to download to the version you specified, if no version is specified, the latest version will be downloaded
Copy your new custom configuration file .env.custom to the new version folder and place it in the same directory as the default .env file
Copy your license files licenseKey.txt and licenseKey.txt to the directory /configs/ which is under the new version folder
If you choose to use your own maintained RDS, go to download the Database upgrade Script and complete the upgrade of RDS
Modify the load balancing configuration to only allow test accounts to use the new version of the system

If you are using nginx built into Univer, you can configure it as follows:

Edit the /nginx/universer.conf under the new version folder, and add the following configuration before the server routing configuration:

server {
  listen 8000;
 
  # Configure to allow only requests with header x-request-env=test
  # Add this configuration to verify the new version when users are not available
  underscores_in_headers on;
  if ($http_x_request_env != "test") {
    return 403;
  }
 
  client_max_body_size 100m;
 
  location / {
    proxy_pass http://universer;
  }
 
  ...
}

Start the new version services: bash run.sh start, after start success, you can login with the test accounts, carry the header x-request-env=test to test.

test after upgrade

Now, you have started the new version services and users cannot access it temporarily. Next, you need to comprehensively verify whether the new version services is configured and deployed correctly, and whether it can run stably as expected.

Enable new version of service

If the new version services test accepted and decided to be adopted, delete the configure in /nginx/universer.conf that only allows specific header requests, and execute the command to reload the configuration of nginx: docker exec universe-lb nginx -s reload

rollback to old version

If the new version services does not accepted, or there is not enough time to complete the test, you can rollback to the old version first.

Stop the new version services first, because there are no users using it at this time, you can stop it directly. Execute bash run.sh stop to stop it.
Restore the nginx configuration of the old version: remove location ~ ^/universer-api {return 403;}(Do you remember that we disabled writing before stopping the old service?)
Restart the old version services.

SOP of Downtime upgrade using k8s

Preparation before upgrade

Custom configuration file

Before preparing for the upgrade, you should check our version introduction to determine the version you want to upgrade to, and check our [production deployment(/guides/sheets/pro-features/server/deploy), which will introduce which version has added new configuration items and which version no longer supports old configuration items. According to the document, check what needs to be modified and what new configurations need to be customized in your old version custom configuration file values.yaml. After confirming, prepare a new custom configuration file values.yaml and keep the old version of values.yaml for safe rollback if need.

** Please note **: If you find that a custom configuration item in the old version does not need to be modified, the new version of the custom configuration still needs to include it, otherwise this configuration item will use the default value when upgrading. This is because helm dynamically renders the resource list required by K8s with’values.yaml ‘and templates during deployment.

License

You also need to prepare your license file.

Upgrade notification

Inform users of the expected downtime upgrade period so that they are prepared

Upgrade operation

prevent writing first

Note: Due to the special nature of collaborative editing, in order to ensure that all collaborative editing requests are saved normally, it is best not to directly stop the services, but to prevent user access. If the K8s cluster you maintain has a method to configure to reject all requests that contain the prefix ‘/univer-api/’ in the path, then configure it directly. If not, you can achieve this goal by disabling the ingress of the universer service, which can be configured as follows:

universer:
  ingress:
    enabled: false  # Disable universer ingress

Because helm dynamically renders the k8s resources list, in order to enable the above configuration, you need to apply the above configuration changes based on your old custom configuration (assuming values-1.0.0), generate a new configuration file (assuming values-1.0.0-forbid.yaml), and then use helm to perform an update to apply this configuration:

helm upgrade univer-stack \
    oci://univer-acr-registry.cn-shenzhen.cr.aliyuncs.com/helm-charts/univer-stack \
    --version your-current-version \
    -n univer \
    -f values-1.0.0-forbid.yaml \
    --set-file universer.license.licenseV2=your-license.txt-path \
    --set-file universer.license.licenseKeyV2=your-licenseKey.txt-path

In the command, use -f to specify the values-1.0.0-forbid.yaml. Please note that we only need configuring to prevent access to Univer at this time, and cannot upgrade to the new version now. Therefore, we need to use --version to specify the chart version is still the currently deployed version. You can use the following command to obtain the currently deployed chart version:

helm list --all-namespaces --filter univer-stack

The execution result is shown in the following figure, and the version of the current deployment can be obtained from the field chart. In this case, it is 0.1.3

prevent writing for a period of time, it is recommended at least 1 minute

Data backup

Univer suggests backup important data before upgrading. The data which needs to be backed up includes units metadata in RDS. You can use the corresponding RDS data backup tool to backup it yourself. Data in Object Storage will only be added and not modified, so there is no need to backup when upgrading.

Upgrade to the new version

If you choose to use your own maintained RDS, go to download the Database upgrade Script and complete the upgrade of RDS
Since we don’t want all users to directly access the new version services, and we need a test account to be able to access it for testing, you need to configure it to allow only specific users to access. If your infrastructure can support this, configure through it. If not, you can configure it through ingress as follows:

If your k8s cluster uses nginx ingress controller, the following configuration will only allow requests with header x-request-env=test. You can merge it into the new version of the custom configuration values.yaml to support testing and verification with this header:

universer:
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/configuration-snippet: |
        if ($http_x_request_env != "test") {
          return 403;
        }

Execute the helm command to upgrade to the specified version

helm upgrade --install univer-stack \
    oci://univer-acr-registry.cn-shenzhen.cr.aliyuncs.com/helm-charts/univer-stack \
    --version target-version \
    -n univer \
    -f your-values.yaml-path \
    --set-file universer.license.licenseV2=your-license.txt-path \
    --set-file universer.license.licenseKeyV2=your-licenseKey.txt-path

In the command, use -f to specify the new version of the custom configuration values.yaml, use --version to specify the target version upgrade to, if no --version specified, it will upgrade to the latest version.

test after upgrade

Enable new version services

If the new version is test accepted and decided to be adopted, update the configuration to make it accessible to all users. If you use ingress to control which user can access it, you need to remove the corresponding configuration and execute the helm upgrade cmd to apply it.

rollback to old version if need

Use the old chart version number and custom configuration to rollback:

helm upgrade --install univer-stack \
    oci://univer-acr-registry.cn-shenzhen.cr.aliyuncs.com/helm-charts/univer-stack \
    --version old-version \
    -n univer \
    -f old-values.yaml \
    --set-file universer.license.licenseV2=your-license.txt-path \
    --set-file universer.license.licenseKeyV2=your-licenseKey.txt-path

use --version to specify the old chart version, -f to specify the old custom configuration.

Production Deploy SRE Manual