Sign inSign up

mirantis/magma-ci

Verified Publisher

By Mirantis Inc

•Updated almost 6 years ago

Image
Integration & delivery
0

1.9K

mirantis/magma-ci repository overview

⁠magma-ci-infra

Few notes about building FB Magma CICD.

⁠Requirements

Few necessary tools for building the AKS cluster using this repository:

⁠Diagram

Magma CI/CD

⁠Create AKS cluster

Follow these steps to install the AKS cluster using Terraform.

⁠Variables

Export your environment to access the Azure. If you do not know how how to fill these variables please look at Configuring the Service Principal in Terraform⁠ or use Google to find the details. You can also look here: https://ruzickap.github.io/k8s-flagger-istio-flux/part-01/#prepare-the-azure-environment⁠

export ARM_CLIENT_ID="axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8"
export ARM_CLIENT_SECRET="dxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx0"
export ARM_SUBSCRIPTION_ID="exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb"
export ARM_TENANT_ID="5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8"
export TF_VAR_client_id="${ARM_CLIENT_ID}"
export TF_VAR_client_secret="${ARM_CLIENT_SECRET}"
export TF_VAR_subscription_id="${ARM_SUBSCRIPTION_ID}"
export TF_VAR_tenant_id="${ARM_TENANT_ID}"

The variables above are mandatory and needs to be properly set before doing next steps.

There are few additional variables which you can use to configure Terraform without touching it's code. The following variables are "defaults" and you may change them if necessary (otherwise just copy&paste them to your terminal):

export TF_VAR_admin_email_address="[email protected]"
export TF_VAR_dns_zone_name="magma.mirantis.cloud"
export TF_VAR_kubernetes_cluster_name="k8s"
export TF_VAR_kubernetes_version="1.15.11"
export TF_VAR_location="francecentral"
export TF_VAR_prefix="magmaci"
export TF_VAR_resource_group_name="${TF_VAR_prefix}"
export TF_VAR_resource_group_name_dns="magma-dns"
export TF_VAR_letsencrypt_environment="staging"  # staging / production
export TF_VAR_argocd_application_git_repository="Mirantis/magma-argocd-apps"
# bcrypt hashed password https://github.com/argoproj/argo-helm/blob/master/charts/argo-cd/values.yaml#L747
export TF_VAR_argocd_server_admin_password="htpasswd_generated_password"
export TF_VAR_harbor_admin_password="harbor_admin_password"
export TF_VAR_prow_git_repository="Mirantis/magma"
# GitHub Personal access token generated for bot user
export TF_VAR_prow_oauth_token='my_prow_oauth_token'
# Slack application access token: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier
export TF_VAR_prow_slack_access_token='my_prow_slack_access_token'
export TF_VAR_oidc_client_id="1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxr.apps.googleusercontent.com"
export TF_VAR_oidc_client_secret="_xxxxxxxxxxxxxxxxxxxxxx4"
export TF_VAR_alertmanager_slack_api_url='https://hooks.slack.com/services/TxxxxxxxT/BxxxxxxxM/Mxxxxxxxxxxxxxxxxxxxxxxr'
export TF_VAR_ssh_public_key="../../secrets/k8s-nodes-ssh-key/id_rsa.pub"
export TF_VAR_default_node_pool="{
  node_count                     = 3,
  vm_size                        = \"Standard_D2as_v4\",
  vm_disk_size                   = 40,
  zones                          = [\"1\"],
  cluster_auto_scaling           = false,
  cluster_auto_scaling_min_count = null,
  cluster_auto_scaling_max_count = null
}"
export TF_VAR_additional_node_pools="{
  p1 = {
    node_count                     = 2,
    vm_size                        = \"Standard_D2as_v4\",
    vm_disk_size                   = 40,
    zones                          = [\"1\"],
    cluster_auto_scaling           = false,
    cluster_auto_scaling_min_count = null,
    cluster_auto_scaling_max_count = null,
  },
}"
export STORAGE_ACCOUNT_NAME="${TF_VAR_prefix}tf"
export CONTAINER_NAME="${TF_VAR_prefix}-tfstate"

For detailed description of the variables please consult terraform/azure/variables.tf⁠.

(Note: In case you want to create second cluster - you should have unique TF_VAR_prefix variable)

Note: In case you have access to decrypted content in secrets/environment_variables⁠ please use the command below, because all the necessary variables are already set in the file.

source secrets/environment_variables

Handy: Best practice is storing all the variables in the file and source⁠ it before running Terraform.

⁠Create Terraform state backend in Azure

The following steps must be completed before running Terraform, because Terraform will use the newly created "storage container" for saving the "state file".

Login to Azure:

az login --service-principal --username "${ARM_CLIENT_ID}" --password "${ARM_CLIENT_SECRET}" --tenant "${ARM_TENANT_ID}" | jq

Output:

[
  {
    "cloudName": "AzureCloud",
    "id": "exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
    "user": {
      "name": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
      "type": "servicePrincipal"
    }
  }
]

Create resource group:

az group create --name "${TF_VAR_resource_group_name}" --location "${TF_VAR_location}" --tags "Owner=${TF_VAR_admin_email_address}" "Environment=Testing" | jq

Output:

{
  "id": "/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourceGroups/magmaci",
  "location": "eastus",
  "managedBy": null,
  "name": "magmaci",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": {
    "Environment": "Testing",
    "Owner": "[email protected]"
  },
  "type": "Microsoft.Resources/resourceGroups"
}

Create storage account:

az storage account create --resource-group "${TF_VAR_resource_group_name}" --name "${STORAGE_ACCOUNT_NAME}" --access-tier Cool --kind StorageV2 --sku Standard_LRS --tags "Owner=${TF_VAR_admin_email_address}" "Environment=Testing" | jq

Output:

{
  "accessTier": "Cool",
  "azureFilesIdentityBasedAuthentication": null,
  "creationTime": "2019-12-11T14:22:31.572168+00:00",
  "customDomain": null,
  "enableHttpsTrafficOnly": true,
  "encryption": {
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "services": {
      "blob": {
        "enabled": true,
        "lastEnabledTime": "2019-12-11T14:22:31.634694+00:00"
      },
      "file": {
        "enabled": true,
        "lastEnabledTime": "2019-12-11T14:22:31.634694+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "failoverInProgress": null,
  "geoReplicationStats": null,
  "id": "/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourceGroups/magmaci/providers/Microsoft.Storage/storageAccounts/magmacitf",
  "identity": null,
  "isHnsEnabled": null,
  "kind": "StorageV2",
  "largeFileSharesState": null,
  "lastGeoFailoverTime": null,
  "location": "eastus",
  "name": "magmacitf",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://magmacitf.blob.core.windows.net/",
    "dfs": "https://magmacitf.dfs.core.windows.net/",
    "file": "https://magmacitf.file.core.windows.net/",
    "queue": "https://magmacitf.queue.core.windows.net/",
    "table": "https://magmacitf.table.core.windows.net/",
    "web": "https://magmacitf.z28.web.core.windows.net/"
  },
  "primaryLocation": "eastus",
  "provisioningState": "Succeeded",
  "resourceGroup": "magmaci",
  "secondaryEndpoints": null,
  "secondaryLocation": null,
  "sku": {
    "capabilities": null,
    "kind": null,
    "locations": null,
    "name": "Standard_LRS",
    "resourceType": null,
    "restrictions": null,
    "tier": "Standard"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": null,
  "tags": {
    "Environment": "Testing",
    "Owner": "[email protected]"
  },
  "type": "Microsoft.Storage/storageAccounts"
}

Create container:

az storage container create --name "${CONTAINER_NAME}" --account-name "${STORAGE_ACCOUNT_NAME}" | jq

Output:

{
  "created": true
}
⁠Create cluster using Terraform

Clone git repository magma-ci-infra:

test -d ".git" || ( git clone [email protected]:Mirantis/magma-ci-infra.git && cd magma-ci-infra )

Initialize Terraform:

cd terraform/azure
terraform init \
  -backend-config="container_name=${CONTAINER_NAME}" \
  -backend-config="storage_account_name=${STORAGE_ACCOUNT_NAME}" \
  -backend-config="resource_group_name=${TF_VAR_resource_group_name}"

Output:

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.37.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.local: version = "~> 1.3"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Execute Terraform and create the AKS cluster:

terraform apply -auto-approve

Output:

...
...
...
Apply complete! Resources: 44 added, 0 changed, 0 destroyed.
Releasing state lock. This may take a few moments...

Outputs:

acr_admin_password = Vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1
acr_admin_username = magmacik8smagmamirantiscloud
acr_login_server = magmacik8smagmamirantiscloud.azurecr.io
azure_kubernetes_cluster_login = az aks get-credentials --name magmaci-k8s-magma-mirantis-cloud --resource-group magmaci
kubeconfig_export_command = export KUBECONFIG=./kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
kubeconfig_file = ./kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
location = eastus
url_alertmanager = https://alertmanager.magmaci.magma.mirantis.cloud
url_argocd = https://argocd.magmaci.magma.mirantis.cloud
url_grafana = https://grafana.magmaci.magma.mirantis.cloud
url_prometheus = https://prometheus.magmaci.magma.mirantis.cloud
url_prow = https://prow.magmaci.magma.mirantis.cloud
url_tekton-dashboard = https://tekton-dashboard.magmaci.magma.mirantis.cloud

The k8s cluster should be up and running. Check it by running:

export KUBECONFIG=$PWD/kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
kubectl get nodes

Output:

NAME                                  STATUS   ROLES   AGE   VERSION
aks-magmacik8sd-34239724-vmss000000   Ready    agent   49m   v1.15.11
aks-magmacik8sd-34239724-vmss000001   Ready    agent   43m   v1.15.11

⁠Delete AKS cluster

You must have defined the same variables as you defined when you created the cluster. (See "Variables" section above)

Login to Azure:

az login --service-principal --username "${ARM_CLIENT_ID}" --password "${ARM_CLIENT_SECRET}" --tenant "${ARM_TENANT_ID}" | jq

Output:

[
  {
    "cloudName": "AzureCloud",
    "id": "exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
    "user": {
      "name": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
      "type": "servicePrincipal"
    }
  }
]

Go to the terraform/azure directory:

cd terraform/azure

Delete AKS cluster and it's dependencies:

terraform destroy -auto-approve

Output:

data.azurerm_resource_group.resource_group: Refreshing state...
azurerm_kubernetes_cluster.aks: Refreshing state... [id=/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourcegroups/magmaci/providers/Microsoft.ContainerService/managedClusters/magmaci-k8s-magma-mirantis-cloud]
local_file.file: Refreshing state... [id=e90e46639b01fedc353cbb70664748b643f823a0]
local_file.file: Destroying... [id=e90e46639b01fedc353cbb70664748b643f823a0]
local_file.file: Destruction complete after 0s
...
...
...
Destroy complete!

Remove the whole Resource group including the container/storage used by Terraform:

az group delete --yes --name "${TF_VAR_resource_group_name}"

You can also clean all the Terraform+Azure files from your laptop:

rm -rf .terraform
az logout
rm -rf ~/.azure

⁠Example building the Magma CICD environment from scratch

Real example. You need to have access to the repositories using the PGP+SSH key. There should be a proper keys/files in these two directories:

  • ~/.gnupg/ - PGP keys
  • ~/.ssh/ - ssh keys

Commands:

ssh-agent
docker run --rm -it -e SSH_AUTH_SOCK -v /var/tmp:/mnt -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK -v $HOME/.gnupg:/root/.gnupg ubuntu:latest
apt-get update -qq && apt install -qq -y curl git git-crypt jq unzip > /dev/null
curl -sL https://aka.ms/InstallAzureCLIDeb | bash

curl "https://releases.hashicorp.com/terraform/0.12.23/terraform_0.12.23_linux_amd64.zip" --output /tmp/terraform_linux_amd64.zip
unzip -o /tmp/terraform_linux_amd64.zip -d /usr/local/bin/
rm -f /tmp/terraform_linux_amd64.zip

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/local/bin/

cd /mnt
git clone [email protected]:Mirantis/magma-ci-infra.git
cd magma-ci-infra
git-crypt unlock
source secrets/environment_variables
scripts/run-create.sh

⁠Handy URLs

Build Status

After the build you should be able to reach these URLs:

⁠Notes

If you need to access the AKS worker nodes, please follow the official guide: https://docs.microsoft.com/en-us/azure/aks/ssh⁠

  • Use the ssh key located in secrets/k8s-nodes-ssh-key/id_rsa
  • Username for accessing the node is ubuntu by default (defined in terraform/azure/variables.tf as admin_username)

Example:

kubectl run --generator=run-pod/v1 -it --rm aks-ssh --image=debian
apt-get update && apt-get install -y openssh-client
kubectl cp secrets/k8s-nodes-ssh-key/id_rsa $(kubectl get pod -l run=aks-ssh -o jsonpath='{.items[0].metadata.name}'):/id_rsa
ssh -i id_rsa ubuntu@aks-magmacik8sd-34239724-vmss000000

Tag summary

Content type

Image

Digest

Size

60.5 MB

Last updated

almost 6 years ago

docker pull mirantis/magma-ci

This week's pulls

Pulls:

130

Last week

Learn more⁠