Deployment HOWTO

This document covers how to deploy and update our stack in Azure (using terraform and ansible). The full workflow for creating a fresh deployment into a fresh subscription is as follows:

1. Prepare new subscription by creating service principal for terraform & ansible
2. Prepare deployment-specific IaC (which usually means making a copy of /customer/exampleexplore and filling out deployment-specific variables)
3. Prepare deployment container
4. Run the bootstrap ansible role
5. Provision infrastructure with terraform (included in step 4)
6. Prepare and run Ansible to configure the infrastructure created in 5.
7. Prepare your environment to access your deployment

In most cases you only need a subset of the above:

• If you are deploying a new internal development/testing environment on our main subscription, using an existing deployment container, it is merely steps 4. + 6.-7. that is needed.
• If you are re-creating an existing deployment (e.g. for internal engineering purposes) using an existing deployment container it is merely steps 5.-7. that is needed.

In the following, gentzen is used as the customer/deployment name.

Specify the customer/deployment name used in the commands below

This variable needs to be defined for most of the commands used in this guide:

# Our example-name used in this guide
export CUSTOMER_NAME=gentzen"

⚠️NOTE you need to specify this both when inside and outside of the deployment-container created in step 3

1 Prepare new subscription

To be able to deploy in a new subscription, the roles assigned to our service principal needs to be updated. More specifically, the new subscription needs to be included in the scopes of the roles such that it is possible to, e.g., fetch images from our container registry when deploying in the new subscription.

To update the roles, you need to edit the file terraform_modules/sp_roles/roles.tf in the deployment repo and add the new subscription id as a data azurerm_subscription block, like with data "azurerm_subscription" "staging" {} block. Then you need to add a reference to the subscription ID to the assignable_scopes field of the following roles:

• assign_acr_roles_role
• ReadWriteRoles

After that, you need to provision the new roles and assignments using terraform. The user needs to have RBAC assignment priviliges over all subscriptions for this, be it Owner, User Access Administrator or similar. Run the following commands. Read through the generated graph after terraform apply to make sure the changes you are making are correct. It is easy to revert accidental changes but it might have mission critical consequences, so be careful.

terraform init
terraform apply

and hopefully you will be able to provision you infrastructure with terraform. Note that the terraform-state that is created in this step needs to be preserved in the terraformbackendfiles storage account under the azuread-custom-roles container.

2 Prepare deployment-specific IaC

Start by copying the template directory.

lga@frege:~/deployments_repo$ cp -r customers/exampleexplore customers/$CUSTOMER_NAME

The resulting directory will contain the configuration files specific to your deployment. You also need to allocate a subnet for your deployment. Go to Azure Cloud Networking document and pick a free /24 subnet and add it to the table. It should be of the form 10.9.xyz.0/24 for internal environments. Each VM gets a /27 subnet from that. Both deployment and VM subnet is configured by replacing example values in

customers/$CUSTOMER_NAME/terraform/main.tf

Then replace exampleexplore with the customer/deployment name (in this example gentzen) in the following files:

customers/$CUSTOMER_NAME/terraform/main.tf
customers/$CUSTOMER_NAME/bootstrap.tf
customers/$CUSTOMER_NAME/bootstrap.yml

If the environment is used for production, the environment variables (with value development) in all the files above should be changed to production as well.

which contain some default values. The configuration parameters in this file highly depend on the purpose of the deployment.

Note: The host.yml in the customers/$CUSTOMER_NAME folder is merely for example purposes as this is auto-generated by terraform and will be overwritten.

3 Prepare deployment container

The image builded from Dockerfile.mentat is used as the environment for running ansible and terraform. It's based on azure cloud shell and is equipped with all the necessary dependencies. It's heavy, but it's maintained by microsoft.

You need to login to cloudsmith through cli or provide api key taking from website. Credentials need to be in the $HOME/.cloudsmith/ folder. Terraform config file also need to have the cloudsmith token. website (To-do: Give the API key as a enviromental variable to the container and create necessary files from that in the script)

Assuming that a service principal has been created (see step 1.)(service principals are not working for now), the commands to create a deployment-container is as follows (remember to log into Azure, by executing the az login command):

# Define the necessary parameters used by your scripts.
export DEPLOYMENTS_REPOSITORY_PATH="/home/$USER/deployments/" # Location (absolute path!) of your clone of the deployments repo
export ANSIBLE_REPOSITORY_PATH="/home/$USER/ansible/" # Location (absolute path!) of your clone of the ansible repo
export TERRAFORM_REPOSITORY_PATH="/home/$USER/terraform/" # Location (absolute path!) of your clone of the terraform repo

cd $DEPLOYMENTS_REPOSITORY_PATH
./scripts/launch-mentat-with-mounts.sh

the last command does:

1. Fetches the certificates of the deployment Service Principal (service principals are not working for now)
2. Creates a container that mounts your deployments repo into /repo, mounts $HOME/.ssh into /root/.ssh
3. Starts a shell (see the launch-mentat-with-mounts.sh script for details) where you can deploy away!

When you need to store ssh-keys to be used in your deployment container, you can place them on your host in $HOME/.ssh.

4 Run bootstrap ansible playbook

The commands below are executed within the deployment-container created in step 3. First you need to log into Azure as our Service Principal:(For now log in with own azure account, service principals are not working)

With service principles:

# /repo/customers/login-sp.sh

or own azure account:

# az login

and then run the bootstrap playbook:

# ansible-playbook /repo/ansible/playbooks/bootstrap.yml -i /repo/customers/$CUSTOMER_NAME/bootstrap.yml

The playbook creates the storage accounts for the terraform state, runs the bootstrap terraform, creates ssh-keys used to access the VM's, and lastly it runs terraform to provision the entire deployment (described in step 5 below). If succesful, it creates a hosts.yml and ssh_config as described in step 5 below.

5 Provision infrastructure with terraform

⚠️⚠️⚠️ NOTE: This step is also performed by the bootstrap playbook above. However, if you are re-creating an existing environment and do not mind to keep ssh-keys etc., then it is not necessary to run the full bootstrap role again. This step suffices. ⚠️⚠️⚠️

The commands below are executed within the deployment-container created in step 3.

Now it is time to run terraform and provision the ressources of the deployment. Note that you have to be in the terraform-folder for your specific deployment, i.e.,

root@test:~# cd /repo/customers/$CUSTOMER_NAME/terraform

If it is the first time running terraform for this deployment, you need to initialize the terraform state:

root@test:/repo/customers/$CUSTOMER_NAME/terraform# terraform init

Remember that terraform commands are idempotent, so if you are unsure whether or not this has been done, just run the command above to be sure.

Once terraform have been initialized, provision the ressources for your deployment

root@test:/repo/customers/$CUSTOMER_NAME/terraform# terraform apply

and remember to verify that the provisioning plan that terraform proposes is correct before typing "yes".

Once terraform is done, it creates a hosts.yml and ssh_config in customers/$CUSTOMER_NAME. The former is used by ansible in the next step. The latter can be used when you want to acces the deployment (see step 7). The content should also be outputted by terraform when it finishes.

6 Run the main ansible playbook to configure the infrastructure

The commands below are executed within the deployment-container created in step 3. Now it is time to configure the VM's provisioned in the previous step, using ansible.

Firstly, you need to fetch the private ssh-key stored in the $CUSTOMER_NAME-corti keyvault under secret name $CUSTOMER_NAME-privatekey and place it in the file /root/.ssh/$CUSTOMER_NAME. Atm. this is done manually. Now you can run the main ansible role that configures the deployment assuming you are running explore 3101 for example:

# If you are deploying to a seperate subscription run az account set --subscription "<Subscription Name here>"
# If you are unsure of the name of the subscription try running az account list to see available subscriptions.
root@test:~# ansible-playbook -i /repo/customers/$CUSTOMER_NAME/bootstrap.yml -i /repo/customers/$CUSTOMER_NAME/hosts.yml -i /repo/customers/$CUSTOMER_NAME/explore3101_internal.yml -i /repo/customers/$CUSTOMER_NAME/services.yml /repo/ansible/playbooks/explore.yml /repo/ansible/playbooks/loki_prometheus_and_grafana.yml

Note that the hosts.yml file is auto-generated by terraform.

7 Prepare your environment to access your deployment

From step 5 you should have produced a ssh_config that you can use to update your .ssh/config with. Now copy the private key, which can be found in the $CUSTOMER_NAME-corti keyvault, into the file ~/.ssh/$CUSTOMER_NAME and you should be able access all the VMs.

Stack versioning: VMs and services description

In the following we give a brief overview of the stack config used by ansible to deploy to the different VMS and their services. Your bootstrap.yml you need to set the following variables:

    product:
      name: explore
      version: 2107_internal
      deployment_tag: v1.0.0

They refer to predefined versions from our Stack repo, a collection of version matrices that guarantee that the specific versions work together.

This block describes the following: Stack name is the stack you will deploy. As of writing the only supported stack is explore.

version points to a specific service version matrix for that stack. The code block above then points to the explore2107_internal.yml file found in the stack repo.

The optional deployment_tag then refers to the tag for the deployment repo. When deploying production environments we want some guarantee that the infrastructure we deploy on is the same everytime, or if we want to upgrade it that we do so in a controlled versioned way. This field is however optional since it is not as important when are developing deployments or want to test new unreleased versions. If left undefined it will simply skip the check for a git tag.

When developing new services or testing a new versions this stack file can be edited and used with the stack specific playbook, fx. the explore playbook. Running bootstrap fetches this file again, so if you have edited the versions in this file they will get overwritten by that, so tread lightly.

Virtual Machines

Each virtual machine (VM) will be listed in the stack version file as a separated entry. To show the formatting of a VM sepcification, we will give an example:

insights:
   version:
     db_adapter: 2.4.0
     ...
     detectotron: 0.0.1
   modeldata:
     sev: "1234"

   detectotron_models:
     ...

   elastic_index_name: "transcriptions"
   elastic_api:
     ...

   log_level: INFO

Specifications on the to-be-deployed services and VM-specific variables (possibly required by the deployed services) has to be specified for each VM:

• version lists all the services deployed in a VM together with their respective versions. The version should correspond to the Docker Image tag for the relative service on the ACR.
• It's possible to specify other VM-specific variables if needed, e.g., modeldata, elastic_api etc.

The configuration options pertaining to each VM is given in the sections below. NOTE atm. we only have a description for the insights VM

Insights

modeldata

Lists which modeldata should be copied from blob storage. For each key/value pair, data will be copied from {key}-data/{value} to {modeldata_folder}/{key.upper()}/{value}". I.e.,

    insights:
      version:
        db_adapter: 2.4.0
        elastic_adapter: 2.4.2
        elastic_api: 2.1.4
        detectotron: 0.0.1
      modeldata:
        sev: "12721"

together with a modeldata_folder value of /opt/corti/insights/modeldata will result in the data being stored in /opt/corti/insights/modeldata/SEV/12721.

detectotron_models

Specifies which Machine Learning model to deploy alongside detectotron following the formatting

detectotron_models:
  model-1:
    detection_type: "DETECTION_TYPE_HERE"
    host: "{{ hostvars['insights']['ansible_default_ipv4']['address'] }}"
    port: MODEL_PORT_HERE
    model_name: "MODEL_NAME_HERE"

Each of the specified model will be deployed using a tensorflow/serving image, which will look for model data in "{modeldata_folder}/{model_name}". If multiple data is found, the one with the highest numerical ID will be mounted.

PLEASE NOTE: due to the data copying process, it is necessary to assign as model_name the same name specified in modeldata but uppercase.

modeldata:
  sev: "1234"

detectotron_models:
  insights_sev:
    ...
    model_name: "SEV"