Developing on deployment process

This document covers how to deploy and update our stack in Azure in a way that you can make direct changes on the ansible and terraform code and make deployments. All the ansible and terraforms and repos are included to this workflow. The full workflow for creating a fresh deployment is as follows:

1. Prepare deployment-specific IaC (which usually means making a copy of /customer/exampleexplore and filling out deployment-specific variables)
2. Prepare deployment container
3. Run the bootstrap ansible role
4. Provision infrastructure with terraform (included in step 3)
5. Prepare and run Ansible to configure the infrastructure created in 5.
6. Prepare your environment to access your deployment

In the following, gentzen is used as the customer/deployment name.

Specify the customer/deployment name used in the commands below

This variable needs to be defined for most of the commands used in this guide:

# Our example-name used in this guide
export CUSTOMER_NAME=gentzen"

⚠️NOTE you need to specify this both when inside and outside of the deployment-container created in step 2

Prerequisites

• Cloudsmith CLI and token
• Terraformrc file

Fetch a token if you do not have one already and add to ~/.terraformrc

$ cloudsmith token --login <me>@corti.ai
# ~/.terraformrc
credentials "terraform.cloudsmith.io"{
  token = "cops/packages/1ef3..."
}

1. Prepare deployment-specific IaC

Start by copying the template directory.

$ pushd /repo/deployments
$ cp -r customers/exampleexplore customers/$CUSTOMER_NAME
$ popd

The resulting directory will contain the configuration files specific to your deployment. You also need to allocate a subnet for your deployment. Go to Azure Cloud Networking document and pick a free /24 subnet and add it to the table. It should be of the form 10.9.xyz.0/24 for internal environments. Each VM gets a /27 subnet from that. Both deployment and VM subnet is configured by replacing example values in

customers/$CUSTOMER_NAME/terraform/main.tf

Then replace exampleexplore with the customer/deployment name (in this example gentzen) in the following files:

customers/$CUSTOMER_NAME/terraform/main.tf
customers/$CUSTOMER_NAME/bootstrap.tf
customers/$CUSTOMER_NAME/bootstrap.yml

If the environment is used for production or want to use production configurations, the environment variables (with value development) in all the files above should be changed to production as well.

which contain some default values. The configuration parameters in this file highly depend on the purpose of the deployment.

Note: The host.yml in the customers/$CUSTOMER_NAME folder is merely for example purposes as this is auto-generated by terraform and will be overwritten.

2. Prepare deployment container

The image builded from Dockerfile.mentat is used as the environment for running ansible and terraform. It's based on azure cloud shell and is equipped with all the necessary dependencies. It's heavy, but it's maintained by microsoft. You need to login for having permission to pull images from azure registry

$ az acr login --name corti

The commands to create a deployment-container is as follows (remember to log into Azure, by executing the az login command):

# Define the necessary parameters used by your scripts.
$ export DEPLOYMENTS_REPOSITORY_PATH="/home/$USER/deployments/" # Location (absolute path!) of your clone of the deployments repo
$ export ANSIBLE_REPOSITORY_PATH="/home/$USER/ansible/" # Location (absolute path!) of your clone of the ansible repo
$ export TERRAFORM_REPOSITORY_PATH="/home/$USER/terraform/" # Location (absolute path!) of your clone of the terraform repo

$ cd $DEPLOYMENTS_REPOSITORY_PATH
$ /bin/bash ./scripts/launch-mentat-with-mounts-development.sh

the last command does:

1. Creates a container that mounts your deployments repo into /repo, mount your ansible repo into /ansible, mount your terraform repo into /terraform,
2. Mounts ssh credentials from $HOME/.ssh into /root/.ssh
3. Mounts credentials for cloudsmith
4. Starts a shell (see the launch-mentat-with-mounts-development.sh script for details) where you can deploy away!

When you need to store ssh-keys to be used in your deployment container, you can place them on your host in $HOME/.ssh.

3. Run bootstrap ansible playbook

The commands below are executed within the deployment-container created in step 2. First you need to log into Azure using device login:

$ az logout
$ az login

and then run the bootstrap playbook:

$ pushd /repo/deployments/customers/$CUSTOMER_NAME
$ ansible-playbook /repo/ansible/stack/playbooks/bootstrap.yml -i bootstrap.yml
$ popd

The playbook creates the storage accounts for the terraform state, runs the bootstrap terraform, creates ssh-keys used to access the VM's, and lastly it runs terraform to provision the entire deployment (described in step 4 below). If succesful, it creates a hosts.yml and ssh_config as described in step 5 below.

4. Provision infrastructure with terraform

⚠️⚠️⚠️ NOTE: This step is also performed by the bootstrap playbook above. However, if you are re-creating an existing environment and do not mind to keep ssh-keys etc., then it is not necessary to run the full bootstrap role again. This step suffices. ⚠️⚠️⚠️

The commands below are executed within the deployment-container created in step 2.

Now it is time to run terraform and provision the resources of the deployment. Note that you have to be in the terraform-folder for your specific deployment, i.e.,

$ pushd /repo/deployments/customers/$CUSTOMER_NAME/terraform

If it is the first time running terraform for this deployment, you need to initialize the terraform state:

$ terraform init

Remember that terraform commands are idempotent, so if you are unsure whether or not this has been done, just run the command above to be sure.

Once terraform have been initialized, provision the resources for your deployment

$ terraform plan -out tf_plan

Verify that the provisioning plan that terraform proposes is correct and run apply

$ terraform apply -auto-approve tf_plan

Once terraform is done, it creates a hosts.yml and ssh_config in customers/$CUSTOMER_NAME. The former is used by ansible in the next step. The latter can be used when you want to acces the deployment (see step 7). The content should also be outputted by terraform when it finishes.

5. Run the main ansible playbook to configure the infrastructure

The commands below are executed within the deployment-container created in step 3. Now it is time to configure the VM's provisioned in the previous step, using ansible.

Firstly, you need to fetch the private ssh-key stored in the $CUSTOMER_NAME-corti keyvault. You can use the script for this.

$ ./repo/deployments/scripts/get-sshkey-for-customer.sh $CUSTOMER_NAME

Now you can run the main ansible role that configures the deployment assuming you are running explore 2110 for example:

If you are deploying to a seperate subscription run az account set --subscription "" If you are unsure of the name of the subscription try running az account list to see available subscriptions. explore2110_internal.yml inventory files name changes over time and according to prod or dev config, check it from the customer folder.

$ pushd /repo/deployments/customers/$CUSTOMER_NAME
$ ansible-playbook /repo/ansible/stack/playbooks/explore.yml \
   -i bootstrap.yml \
   -i hosts.yml \
   -i explore2110_internal.yml \
   --ssh-extra-args="-F ssh_config"

$ popd

Note that the hosts.yml file is auto-generated by terraform.

To deploy loki & grafana you need to run there playbook for having the monitoring.

$ pushd /repo/deployments/customers/$CUSTOMER_NAME
$ ansible-playbook /repo/ansible/stack/playbooks/loki_prometheus_and_grafana.yml \
   -i bootstrap.yml \
   -i hosts.yml \
   -i explore2110_internal.yml \
   --ssh-extra-args="-F ssh_config"
$ popd