In my last article, we created a log server using Docker, Elastic Search, Kibana and Nginx. This article is the second part, we are going to focus on the log server hosting. More exactly we are going to host it in Microsoft Azure using a virtual Machine.
 

Hosting a containerized application in Azure

Microsoft Azure offers actually several services to manage Docker containers. We can split theses offers in 3 types:

 

In this post we will use the Azure IaaS solution. It’s the solution that permit us to have a full control over our infrastructure and containers, that’s why we will explore this solution first. Besides this solution do not create a strong dependency to Azure, we could easily install our log engine on a virtual machine hosted by another cloud provider.
 

Target Azure Architecture

In Azure, a virtual machine is contained in a virtual network (vnet). A network security group (NSG) can be placed in front of the vnet or the virtual machine, it permits to define inbound and outbound rules.  To make our log server work, we need to allow inbound connections on the port 80. In the virtual machine, two tools are required: Docker and Docker-compose.

Here is an overview of the target Azure architecture:

Working environment

We want to deploy and make run the log Server on Azure. In the last article, we created a folder named “log_server”. It contains all Docker files. We are going to add a shell script at the same level as the folder.

Working environment screenshot:

The development machine can be a Linux or a Windows one. On Windows we need to use bash (easily usable on Windows 10) and we need the azure cli tool.
 

Deployment script

We want to deploy our log server using a minimum of human steps. We can actually achieve it using a single shell script (deploy.sh). We are going to explain each part of the script.

The deployment script contains 3 steps:

  • Create the Azure virtual machine
  • Install Docker tools (Docker & Docker compose) on the virtual machine
  • Push Docker files on the virtual machine and run the dockerized log server

 

Let’s create a function for each step (to improve readability we will use global variables and functions without parameters).
 

Script input variables

The script will create an Azure Virtual Machine. We weed a name and a location for the machine, a resource group (Azure logical container) name, an SSH user name and the entry point (port) on the vm.


rgName='*****'

vmName='****'

vmUser='****'

vmLocation='westeurope'

vmOpenPort=80 #docker compose nginx open port

 

Manage the .htpasswd file content

In the previous article, we explained that the .htpasswd file must contain the login and the password used to set up the Basic Authentication to the log server. For security reason, we do not want to store authentication information on this file on the development machine (or anywhere!). To solve that, we can write the .htpasswd content from the deployment script.  By the way we can store our log server source files (Docker files + deployment script) on a source control without sensitive data.

We can store the password in a bash variable, and write the bash variable in the file. 2 lines of script are needed:


passwordHash='tranise:$apr1$4t5rn1oy$G****************'

echo $passwordHash > ./log_server/nginx/.htpasswd

 

1. Create the Azure VM

We are going to host our log server in a Linux virtual machine (Ubuntu). The first step consists to create this virtual machine. To deploy it, we are going to use the az vm create * command. Our log server must be accessible on the port 80, that’s why we need to open this port on the vm using the az vm open-port command.

* This command will create all components necessary to manage an Azure virtual machine: a virtual network, a storage account and a network security group.

Create Azure virtual machine script :


create_AzureVm(){
 echo -- create the resource group
 az group create --name $rgName --location $vmLocation
 echo -- create the vm
 az vm create --resource-group $rgName --name $vmName --public-ip-address-dns-name $vmName --image UbuntuLTS --admin-username $vmUser --generate-ssh-keys
 echo -- open port $vmOpenPort on the vm
 az vm open-port --port $vmOpenPort --resource-group $rgName --name $vmName
}

We use the –generate-ssh-keys option. This option will generate SSH files (private and public keys) in the ~/ssh directory.

 

2. Install Docker tools

Once the virtual machine is created, we need to install Docker tools (Docker CE & Docker compose) and start the Docker Daemon. To run scripts on the target machine, we will use the az vm run-command command. This command let us define an inline bash script which will be upload and execute on the remote virtual machine.


install_DockerTools (){
 echo install dependencies docker, docker compose and start docker service
 #install Docker
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable' && sudo apt-get update && apt-cache policy docker-ce && sudo apt-get install -y docker-ce"
 #install Docker Compose
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "sudo curl -L https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose && sudo chmod +x /usr/local/bin/docker-compose"
 #start Docker service
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "sudo service docker start"
}

 

3. Run the log server

Once the Docker is up and running on the target virtual machine, we have the runtime ready to host and execute our log server. First we need to upload the Docker files on the remote machine *. All Docker files are gathered in the “log_server” directory, with the tool SCP, we can transfer the folder and its content on the target machine using an SSH connection.

Then we need to run a “docker-compose” command on the remote machine to run the log server.  We can achieve it with the az vm run-command.

 

* We could have used at least 2 other solutions to transfer Docker files:

  • GIT
  • Docker repository

Those solutions would have worked perfectly. In this script version, I decided to use the SCP tool because it just need one line of script to transfer files.

 

Run log server script :

run_LogServer(){
 echo copy 'log_server' folder content in the remote vm
 scp -o StrictHostKeyChecking=no -r ./log_server $vmUser@$vmName.$vmLocation.cloudapp.azure.com:/home/$vmUser/log_server
 echo run 'docker-compose' file
 az vm run-command invoke --debug -g $rgName -n $vmName --command-id RunShellScript --scripts "cd /home/"$vmUser"/log_server && sudo docker-compose up -d"
}

With the SCP tool, we use the -r option to copy the folder and its content recursively.

 

Final Script

In the final script, we are going to use variables and functions we just explained.


#!/bin/bash

rgName='logservertestrg'
vmName='logservertra'
vmUser='tranise'
vmLocation='westeurope'
vmOpenPort=80 # read docker compose nginx host port
#https://www.web2generators.com/apache-tools/htpasswd-generator
passwordHash='tranise:$apr1$4t5rn1oy$G7uJ*********'

echo write the basic auth credentials in nginx file
echo $passwordHash > ./log_server/nginx/.htpasswd

create_AzureVm(){
 echo -- create the resource group
 az group create --name $rgName --location $vmLocation

 echo -- create the vm
 az vm create --resource-group $rgName --name $vmName --public-ip-address-dns-name $vmName --image UbuntuLTS --admin-username $vmUser --generate-ssh-keys

 echo -- open port $vmOpenPort on the vm
 az vm open-port --port $vmOpenPort --resource-group $rgName --name $vmName
}

install_DockerTools (){
 echo install dependencies docker, docker compose and start docker service
 #install Docker
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable' && sudo apt-get update && apt-cache policy docker-ce && sudo apt-get install -y docker-ce"
 #install Docker Compose
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "sudo curl -L https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose && sudo chmod +x /usr/local/bin/docker-compose"
 #start Docker service
 az vm run-command invoke -g $rgName -n $vmName --command-id RunShellScript --scripts "sudo service docker start"
}

run_LogServer(){
  echo copy 'log_server' folder content in the remote vm
  scp -o StrictHostKeyChecking=no -r ./log_server $vmUser@$vmName.$vmLocation.cloudapp.azure.com:/home/$vmUser/log_server
  echo run 'docker-compose' file
  az vm run-command invoke --debug -g $rgName -n $vmName --command-id RunShellScript --scripts "cd /home/"$vmUser"/log_server && sudo docker-compose up -d"
}

 create_AzureVm
 install_DockerTools
 run_LogServer
 
 echo log server available on $vmName.$vmLocation.cloudapp.azure.com

By executing this script we are able to deploy a conterized log server based on Docker in few minutes. That’s great but we can improve it. In my next article, we will update this current architecture (Docker files and deployment script) to store logs data in an Azure File Share using a Docker volume driver.
Source code is available on my Github !

 

Happy coding 🙂


Create a log engine using Docker, Elastic Search, Kibana and Nginx – Azure IaaS Hosting (part 2) Create a log engine using Docker, Elastic Search, Kibana and Nginx – architecture & local work

Leave a Reply

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *