Release and Deploy

2025-07-17

Summary¶

This guide provides step by step instructions for releasing and deploying a new version of SlideRule to the public cluster. All steps are performed via Makefile targets running on an EC2 instance provisioned in the SlideRule AWS environment. The EC2 instance must be setup with access to the SlideRule S3 buckets, container registries, and Git Hub repositories.

SlideRule uses an immutable deployment strategy where the new release is first deployed as a parallel cluster at “sliderule-.slideruleearth.io” and put through a series of tests. Once all tests have passed, the “sliderule.slideruleearth.io” domain is updated to point to the cluster, thereby switching it to become the live cluster. The previously deployed cluster is then left running until the time-to-live on the DNS entries expire and all processing requests have completed. This should take at most 15 minutes (TTL of 5 minutes + Maximum Request Time of 10 minutes). The old cluster is then destroyed.

Checklist:¶

• Create (and check in) release notes for the new release by reviewing all commits, PRs, and GitHub issues. (See sliderule/docs/developer_guide/release_notes).

• Make sure there are no uncommitted changes and the local main branch is in sync with the remote repository.

git checkout main
git status
git pull

• Make sure you have AWS credentials and are logged into the AWS Elastic Container Registry. A part of the deployment process includes access to AWS resources like pushing the recently built and tagged container images to the registry so that they can be pulled in the production system. Note {profile} is the account role.

# aws configure sso --profile {profile} --use-device-code # first time setup only
aws sso login --profile {profile} --use-device-code
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 742127912612.dkr.ecr.us-west-2.amazonaws.com
eval $(aws configure export-credentials --profile {profile} --format env)
export AWS_PROFILE={profile}

• Login to GitHub. GitHub has the concept of a “release” which is a construct on top of a repository tag and includes release artifacts and its own description. The GitHub release is performed from the command line using the gh tool.

gh auth login

• Select version of npm. If the Node Version Manager tool is used, then a compatible version of node must be selected and active in the environment being used for the release.

nvm use 20

• Verify the presence of the conda environments needed to run the different pytest suites: sliderule, ams.

conda env create -f clients/python/environment.yml # creates sliderule environment
conda env create -f apps/ams/environment.yml # creates ams environment
conda env list # lists what conda environments are available

Guide¶

(1) Run Local Self-Tests¶

• Compile the code in debug mode, which runs through a slew of static analysis using the clang tidy and cppcheck tools.

• Run the Lua selftest, which executes all of the component level test scripts with address sanitizer enabled.

From sliderule/targets/slideruleearth

make distclean
make config-debug
make
make selftest

• Run the provisioner PyTests. The provisioner application has its own set of pytests that run under the sliderule conda environment and test the basic functionality and APIs of the provisioner lambdas.

From sliderule/targets/slideruleearth

make provisioner-test

• Run the ams PyTests. The Asset Metadata Service (AMS) application has its own set of pytests that run under the ams conda environment and test the basic functionality and APIs of the ams container.

From sliderule/targets/slideruleearth

make ams-test

(2) Test Local Deployment (Pytests and Jest)¶

• Run a debug version of the sliderule server code locally. Make sure the ilb, and ams containers are running in the background.

From sliderule/targets/slideruleearth

docker compose up ilb ams -d

From sliderule/targets/slideruleearth

make distclean
make config-debug
make
make run

• Run the pytest suite of tests against it, which checks for memory leaks and invalid memory access across the entire pytest suite. Note that tests involving the provisioning system will fail when running against a locally running instance of sliderule.

From sliderule/targets/slideruleearth in a new terminal window

make python test

• Run the Node.js client suite of tests using jest, which test the client and also try to test some of the functionality needed by the web client. Note that the authentication tests will fail when running against a locally running instance of sliderule.

From sliderule/targets/slideruleearth

CLUSTER=null make nodejs-test

(3) Test Deployment to Private Cluster¶

Prior to a full release, it is good to test the full deployment to a private cluster. This catches issues related to the infrastructure in AWS and errors that don’t surface in a local environment but do surface when run as a cluster in the cloud. The cluster can also be deployed using the --desired_nodes and --time_to_live parameters of the pytests, or by using the sliderule-provisioner command line tool.

• Build the latest code base and deploy it to a private cluster (typically developers).

From sliderule/targets/slideruleearth

make cluster-docker
make cluster-docker-push
sliderule-provisioner deploy --node_capacity 3 --ttl 120

• Execute the pytest tests against the deployed private cluster.

From sliderule/targets/slideruleearth

CLUSTER=developers make python-test

• Execute the jest tests against the deployed private cluster.

From sliderule/targets/slideruleearth

CLUSTER=developers make nodejs-test

• Run benchmarks and baseline

While the latest private cluster is still deployed, run the benchmarks and baseline scripts against it. The private cluster should be running 7 nodes. From sliderule/clients/python

conda activate sliderule
pip install .
python utils/benchmarks.py --organization developers
python utils/baseline.py --organization developers

• Update the release notes and make sure all changes are committed and pushed. Other than the release notes, there should be no new changes in the repository at this point; if there are, then the steps above need to be rerun.

(4) Create New Release¶

From sliderule/targets/slideruleearth

make release RELEASE=<version>

where version is in the form vX.Y.Z.

This directly performs at least the following:

• Builds the latest sliderule docker build environment sliderule-buildenv

• Updates the build dependencies in the libdep.lock file

• Updates the ams lock file

• Tags the repository

• Creates a GitHub release

• Builds and pushes/uploads the cluster (docker containers, AMI, terraform)

• Builds and pushes/uploads the static website

This indirectly kicks off a series of GitHub actions:

• Publishes the updated PyPI package for the SlideRule Python Client

• Creates a Pull Request in the conda-forge feedstock for the SlideRule Python Client

• Builds and runs the self-tests

• Deployes the code, using the unstable tag, to the cicd cluster, and runs the pytest suite against it

Note - when a new release is created, there is a GitHub action that automatically creates and publishes a NPM package an a PyPI package. For PyPI, once published, the Conda Forge feedstock for the SlideRule Python Client package automatically kicks off the process of building and releasing the updated conda-forge package. Once the package is built, the final step of merging the PR is manual and must be done by going to: https://github.com/conda-forge/sliderule-feedstock

(5) Deploy the Newly Released Version to the Public Cluster¶

• Deploy the new release. This will always deploy the docker container images tagged as latest. The output of this command will contain the name of the cluster in the form: sliderule-<timestamp>.

From sliderule/targets/slideruleearth/

make cluster-deploy-candidate

• Test out the new cluster, using sliderule-<timestsamp> as the organization.

From sliderule/clients/python

conda activate sliderule
pip install .
pytest --organization sliderule-<timestamp>

• Switch to the new cluster by pointing the sliderule.slideruleearth.io domain to the newly deployed cluster. The <ip> address supplied in the command is the public ip address of the ilb from the deployment of the new cluster.

From sliderule/targets/slideruleearth/

make cluster-go-live PUBLIC_IP=<ip>

• Wait 30 minutes. This will allow DNS caches to flush and any active requests on the old cluster to complete.

• Destroy the old cluster. The <old-timestamp> is the timestamp of the old cluster that we want to now destroy.

sliderule-provisioner destroy --cluster sliderule-<old-timestamp>

(6) Update the Documentation¶

Make sure documentation, examples, and OpenAPI specifications are up to date, and then generate and update the live documentation.

• Update the local sliderule conda environment (which will be used to build the Python API reference)

make python

• Build and install a local version of the sliderule server (which will be used to build the OpenAPI specification)

make config-release
make

• Build the documentation and check out a local running version

make documentation
make documentation-run

• Update the documentation on the live website

make documentation-live-update