database strategies for microservices

Last blog I have talked about the problem of database lockup but how can we solve it?

Shared Tables: Shared Tables could be a easy to go and a dirty solution that is very common. But be aware that it is high maintenance.
Using mysql we can use FEDERATED ENGINE (http://dev.mysql.com/doc/refman/5.1/en/federated-storage-engine.html) to do this. We have to create a federated table based on the table at another remote location that we want.

CREATE TABLE remote_user (
  username varchar(20) NOT NULL,
  password varbinary(20) NOT NULL,
  PRIMARY KEY(username)
) ENGINE=FEDERATED DEFAULT CHARSET=utf8 CONNECTION='mysql://username:password@someip:port/db/user’;

Database View : A database view is a comparatively better approach when the cases are simple because it allows another representation of database model which is more suitable. Most amazing thing about database view is that it supports wide range of databases. But for heavy use cases we can see performance issues. While considering database view we must ensure that both of the databases can connect with each other without any network or firewall issue. Most of the database views are read only, updating them according to need might get tricky.

CREATE TABLE federated_table (
    [column definitions go here]
)
ENGINE=FEDERATED
CONNECTION='mysql://username:password@someip:port/db/user’;

Triggers:
Database triggers might come handy where one database operation will trigger another database update. We can bind to AFTER INSERT, AFTER UPDATE, and AFTER DELETE triggers.

CREATE TRIGGER user_bi BEFORE INSERT ON user FOR EACH ROW
BEGIN
  INSERT INTO remote_user (username,password) VALUES (NEW.username,NEW.password);
END

Data Virtualization: When we are dealing with micro services possibly some of our databases are running using Mysql while other services are running other DBMS. In that case Data Virtualization strategy is necessary. One open source data virtualization platform is Teiid. But when dealing with data virtualization strategy we must know that if we are dealing with stale data or not, as it will have serious performance issue as it will add another hop as the data is not being accessed directly from database.

Event sourcing: Rather then making database operatins we can consider designing it as a stream of events that goes one after another through as message broker. So it does not matter how many users are accessing your database it will never lock up your database but it would take more time to process the data.

Change Data Capture: Another approach is to use Change Data Capture (CDC), is an integration strategy that captures the changes that are being made to a data and makes them available as a sequence of events in other databases that needs to know about these changes. It can be implemented using Apache Kafka, Debezium and so on.

Simple trick that can can help us to achieve Zero Downtime when dealing with DB migration

Currently we are dealing with quite a few deployment processes. For a company that enables DevOps culture, deployment happens many many times a day. Tiny fraction of code change goes to deployment, and as the change size is so small it gets easier to spot a bug and if the bug is crucial maybe it is time to rollback to an older version and to be able to have a database that accepts rollback, yet we have to do it with zero downtime so that the user do not understand a thing. It is often is not as easy as it sounds in principal.

Before describing about few key idea to solve this common problem lets discuss few of our most common deployment architectures.

In a blue/green deployment architecture, it consists of two different version of application running concurrently, one of them can be the production stage and another one can be development platform, but we need to note that both of the version of the app must be able to handle 100% of the requests. We need to configure the proxy to stop forwarding requests to the blue deployment and start forwarding them to the green one in a manner that it works on-the-fly so that no incoming requests will be lost between the changes from blue deployment to green.

Canary Deployment is a deployment architecture where rather than forwarding all the users to a new version, we migrate a small percentage of users or a group of users to new version. Canary Deployment is a little bit complicated to implement, because it would require smart routing Netflix’s OSS Zuul can be a tool that helps. Feature toggles can be done using FF4J and Togglz.

As we can see that most of the deployment processes requires 2 version of the application running at the same time but the problem arises when there is database involved that has migration associated with it because both of the application must be compatible with the same database.So the schema versions between consecutive releases must be mutually compatible.

Now how can we achieve zero downtime on these deployment strategies?

So we can’t do database migrations that are destructive or can potentially cause us to lose data. In this blog we will be discussing how can we approach database migrations:

One of the most common problem that we face during UPDATE TABLE is that it locks up the database. We don’t control the amount of time it will take to ALTER TABLE but most popular DBMSs available in the market, issuing an ALTER TABLE ADD COLUMN statement won’t lead to locking. For example if we want to change the type of field of database field rather than changing the field type we can add a new column.

When adding column we should not be adding a NOT NULL constraint at the very beginning of the migration even if the model requires it because this new added column will only be consumed by the new version of the application where as the new version still doesn’t provide any value for this newly added column and it breaks the INSERT/UPDATE statements from current version. We need to assure that the new version reads values from the old column but writes on both.  This is to assure that all new rows will have both columns populated with correct values. Now that new columns are being populated in a new way, it is time to deal with the old data, we need to copy the data from the old column to the new column so that all of your current rows also have both columns populated, but the locking problem arises when we try to UPDATE.

Instead of just issuing a single statement to achieve a single column rename, we’ll need to get used to breaking these big changes into multiple smaller changes. One of the solution could be taking baby steps like this:

ALTER TABLE customers ADD COLUMN correct VARCHAR(20); UPDATE customers SET correct = wrong

WHERE id BETWEEN 1 AND 100; UPDATE customers SET correct = wrong

WHERE id BETWEEN 101 AND 200;
ALTER TABLE customers DELETE COLUMN wrong;

When we are done with old column data population. Finally when we would have enough confidence that we will never need the old version, we can delete a column, as it is a destructive operation the data will be lost and no longer recoverable.

As a precaution, we should delete only after a quarantine period. After quarantined period when we are enough confident that we would no longer need our old version of schema or even a rollback that does require that version of schema then we can stop populating the old column.  If you decide to execute this step, make sure to drop any NOT NULL constraint or else you will prevent your code from inserting new rows.

Higher Level View of RDMA programming and its vocabularies

Recently I have come across a pretty cool tool called RDMA. It enables direct memory access from the memory of one computer into that of another computer without involving the burden of either one’s operating system. This permits high-throughput, low-latency networking, which is especially useful in massively parallel computer clusters. In this blog I will be noting down few vocabularies that comes in handy when dealing with RDMA.

Queue Pair (QP) consists of a Send Queue (SQ) and Receive Queue (RQ). When we expect it to send data we would send it to SQ and when we expect it to receive data, we would sends it to RQ. Both of them can but put on a Completion Queue (CQ).  Completion queue (CQ) is used by network adapter to notify the status of the completed Work Request. Each entry in Completion Queue entry (CQE) holds information of completion status of one or more completed work requests.

When we want an adapter to send or receive, we need to post a request these are called work requests. In a Send Request we need to assign how much data will be sent for connected and unconnected transport and the memory buffer where data is located for connected and unconnected transport, to where the data should be send and the type of the send request and in a receive requests, the maximum data size to be received and memory buffer where data should be. Completion of a send queue and a receive queue can be assigned to same or different completion queues.

Work queue maintains order of their posted time however in different work queue does not maintain orders. Every work queue has ids own user defined id wr_id and flags, for example wr.send_flags = IBV_SEND_SIGNALED  defines generation of a completion element once the data is transmitted. it can be handled in a chain manner by assigning another work queue in wr.next

ibv_create_cq is the command that helps to create CQ. Transportations can be completed successfully or with error result is reported through a completion queue entry (CQE) polling a CQ is used to retrieve the CQE from the CQ outcome is reported in the status field of the completion entry.

We create a QP using ibv_create_qp function. In the parameter it takes a Protection Domain(PD) and a set of attributes. Protection domain is gathering resources in groups. Resource from same protection domain are allowed to communicate with each other. Eg: QP, MP. Resource from outside protection domain are not allowed to communicate. To allocate protection domain by calling ibv_alloc_pd. Attribute struct would look something like this:

struct ibv_qp_init_attr qp_init_attr;
struct ibv_cq *cq;
qp_init_attr.send_cq = cq; 
qp_init_attr.recv_cq = cq; 
qp_init_attr.qp_type = IBV_QPT_UD; 
qp_init_attr.cap.max_send_wr = 2; 
qp_init_attr.cap.max_recv_wr = 2; 
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;

more at: https://www.rdmamojo.com/2012/12/21/ibv_create_qp/

Where max_send_wr maximum number entities that we want to allow in a send queue before completion. By the way it would be wise to note that it should be less than max cqe.
max_recv_wr maximum scatter queue that we want to allow.
At max_send_sge, max_recv_sge, sge is a short hand for Scatter Gather Entries. The maximum number of scatter gather entries can be queried using ibv_device_query.

As you can see we have set qp_type to IBV_QPT_UD which is there for Unreliable Data. In reliable context, QP is possible between two RCs, but when it is about Unreliable QP, it allows one to many Unreliable Queue Pair without requiring any previous connection setup.

Like many things in network programming, QP goes through a series of steps before it ends up processing send a receive.

RESET: By default, upon the creation of QP, it is at it’s reset state. Although it is at its ready to receive data but it can’t process any work request at this state.
INIT: After RESET it goes to INIT state after its initial configuration. When QP moves from RESET to INIT, QP starts receiving receive buffers in the receive queues using ipv_recv commands. this data won’t be used until QP is in RTR state.
RTR: After that it goes to Ready To Receive state, at this state it is configured to receive data.
RTS: After that it goes to Ready To Send, at this stage it is configured to send data. At this stage device can post using ipv_post_send commend.

After creation if you want you can modify QP using ibv_modify_qp. When modifying QP, pkey_index, port_num, qkey (for unrelieable datagram only) might be necessary. All QP that wants to communicate on unreliable datagram must share same q_key.

To make RDMA do things, it is necessary for network adapters to ask for permissions to access local data. this is done through MR (memory region). A memory region has address, size and set of permissions. that control access to the memory pointed out by the region.

To register memory region we need to call ibv_reg_mr. It takes the Protection Domain, Start Virtual Memory Address, size, access bit information like local read, local write, remote read, remote write, atomic operation. Local read access is necessary when the adapter has to access local pc to gather data when rdma operating is being processed. Local write access is necessary when adapter has to scatter data when recieving a send operation. Remote access is necessary when the adapter has to access local data from rdma operation recieved by remote process.

to open a communication we would need to call ipv_open_device, where we can assign a context and a pointer. cq_context, channel, comp_vector is necessary when dealing with completion events.

If we want to send data from a to b, we would be needing a source and a destination address or destination_gid, which is known as address vector.

We can collect our device details using ibstat command. But please note that we would need to have connect two devices, install mlnx_ofed, ibstat command, change port type ib/eth, check ports are enabled state LinkUp, if running using infiniBand opensm must be running. Also we can collect that information programmatically using ibv_get_device_list function.

Under the hood libverb handles rdma network related operations, like creating, modifying, querying, destroying resources. it handles sending receiving data from QPs, and recieving Completion Queues.

As Ipdump does not work when we are dealing with infiniBand as it bypasses OS layer we can use ibdump for debugging.

Running a on premise local mysql replica with AWS RDS Aurora master

To solve our problem we are running a hybrid cloud. Few of our services are running on cloud and some of our services are running in premise locally in our country where we have our users and where AWS does not provide service. To able to do that we need a database replica that has read facility.

We need to creating replica user:

CREATE USER 'replica'@'%' IDENTIFIED BY 'slavepass'; 
GRANT REPLICATION SLAVE ON *.* TO 'replica'@'%';

Then create a new DB Cluster parameter group and set binlog_format to MIXED. Modify the Aurora cluster and select the custom parameter group. Restart your db to apply those changes. Now if you run following command you will be able to see the bin log file name and position.

show master status

Now we need to dump our master user data to sql dump so that we can feed our slave database.

mysqldump --single-transaction --routines --triggers --events -h XXX.azhxxxxxx2zkqxh3j.us-east-1.rds.amazonaws.com -u bhuvi –-password='xxx' my_db_name > my_db_name.sql

It can be GB to TB of data depending on your database size. So it will take time to download.

Run follwoing to know your mysql configuration file:

mysqld --help -verbose | grep my.cnf

For me it is /usr/local/etc/my.cnf

vi /usr/local/etc/my.cnf

and change server-id to:

 [mysqld] server-id = 2

now lets import these data into our mysql.

mysql -u root –-password='xxx' my_db_name < my_db_name.sql

Now we need to let our slave database know who is the master:

CHANGE MASTER TO  
MASTER_HOST = 'RDS END Point name',  
MASTER_PORT = 3306,  
MASTER_USER = '',  
MASTER_PASSWORD = '',  
MASTER_LOG_FILE='',  
MASTER_LOG_POS=;

Now we need to start the slave.

start slave;

Setting up (comodo) ssl for your website on aws

We have bought our ssl from comodo from name.com as we got a better deal there. After sending them our signed key. comodo sent us following files via email, against my private key. Now I would blog about how I setted the whole thing up on AWS.

First of all, before purchasing I had to send them a key which I had generated using OpenSSL using following command:

openssl req \
       -newkey rsa:2048 -nodes -keyout domain.key \
       -out domain.csr

Which was pretty easy. And as we had bought Comodo Essential SSL Wildcard so we could buy it without verifying our company, in fairly easy in less than 5 min.

After our successful purchase comodo sent us following files as zip in my email:
domain_com.crt
COMODORSAAddTrustCA.crt
domain_com.crt os our Primary Certificate, COMODORSAAddTrustCA.crt is our Intermediate Certificate, and AddTrustExternalCAROOT.crt is the The Root Certificate.

Now it gets a little bit tricky because currently our certificates are in .crt format, but we want it to be in *.pem format. So we would need to convert them in *.pem.

openssl x509 -in ./AddTrustExternalCARoot.crt -outform pem -out ./pem/AddTrustExternalCARoot.pem
openssl x509 -in ./COMODORSAAddTrustCA.crt -outform pem -out ./pem/COMODORSAAddTrustCA.pem
openssl x509 -in ./COMODORSADomainValidationSecureServerCA.crt -outform pem -out ./pem/COMODORSADomainValidationSecureServerCA.pem
openssl x509 -in ./domain_com.crt -outform pem -out ./domain.pem

We would also need to keys that was used to create these certificates by comodo.

openssl rsa -in ./domain.key -outform PEM -out domain.key.pem

Lets create the chain first:

$ cat ./COMODORSADomainValidationSecureServerCA.pem > ./CAChain.pem
$ cat ./COMODORSAAddTrustCA.pem >> ./CAChain.pem
$ cat ./AddTrustExternalCARoot.pem >> ./CAChain.pem

Now you need to login to your aws console and search for ACM (Amazon Certificate Manager). and if it is your first time you need to click on Provision certificates.

It is time to import your certificate to ACM. At the form where it says Certificate body* please paste domain.pem and domain.key.pem and at Certificate chain paste CAChain.pem.

So thats it we are done importing.

Now if you have a load balancer you can take advantages of this ssl. If you have an existing load balancer or feel free to create one, where at the place of listener add https instead of http and for certificate choose acm and your domain.

You are good to go.

Configuring django for centralised log monitoring with ELK stack with custom logging option (eg. client ip, username, request & response data)

When you are lucky enough to have enough users that you decide to roll another cloud instance for your django app, logging becomes a little bit tough because in your architecture now you would be needing a load balancer which will be proxying request from one instance to another instance based on requirement. Previously we had log in one machine to log monitoring was easier, when someone reported a error we went to that instance and looked for errors, but now as we have multiple instance we have to go to all the instance, regardless of security risks, i would say it is a lot of work. So I think it would be wise to have a centralized log aggregating service.

For log management and monitoring we are using Elastic Logstash and Kibana popularly known as ELK stack. For this blog we will be logging pretty much all the request and its corresponding responses so that debugging process gets handy for us. To serve this purpose we will leverage django middlewares and python-logstash.

First of all let’s configure our settings.py for logging:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
        'logstash': {
            '()': 'proj_name.formatter.SadafNoorLogstashFormatter',
        },
    },
    'handlers': {
        'default': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/proj_name/django.log',
            'maxBytes': 1024*1024*5, # 5 MB
            'backupCount': 5,
            'formatter':'standard',
        },  
        'logstash': {
          'level': 'DEBUG',
          'class': 'logstash.TCPLogstashHandler',
          'host': 'ec2*****.compute.amazonaws.com',
          'port': 5959, # Default value: 5959
          'version': 1, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
          'message_type': 'logstash',  # 'type' field in logstash message. Default value: 'logstash'.
          'fqdn': False, # Fully qualified domain name. Default value: false.
          #'tags': ['tag1', 'tag2'], # list of tags. Default: None.
          'formatter': 'logstash',
      },

        'request_handler': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/proj_name/django.log',
            'maxBytes': 1024*1024*5, # 5 MB
            'backupCount': 5,
            'formatter': 'standard',
        },
    },
    'loggers': {
        'sadaf_logger': {
            'handlers': ['default', 'logstash'],
            'level': 'DEBUG',
            'propagate': True
        },
    }
}

As you can see we are using a custom logging format. We can leave this configuration and by default LogstashFormatterVersion1 is the logging format that will work just fine. But I chose to define my own logging format because my requirement is different, I am running behind a proxy server, I want to log who has done that and from which IP. So roughly my Log Formatter looks like following:

from logstash.formatter import LogstashFormatterVersion1

from django.utils.deprecation import MiddlewareMixin
class SadafNoorLogstashFormatter(LogstashFormatterVersion1):
    def __init__(self,*kargs, **kwargs):
        print(*kargs, **kwargs)
        super().__init__(*kargs, **kwargs)


    def format(self, record,sent_request=None):
        print(record)
        print(sent_request, "old req")
        caddr = "unknown"
        #print(record.request.META)

        if 'HTTP_X_FORWARDED_FOR' in record.request.META:
            caddr = record.request.META['HTTP_X_FORWARDED_FOR'] #.split(",")[0].strip()
        
#        print(record.request.POST,record.request.GET, record.request.user)
        message = {
            '@timestamp': self.format_timestamp(record.created),
            '@version': '1',
            'message': record.getMessage(),
            'host': self.host,
            
            'client': caddr,
            'username': str(record.request.user),

            'path': record.pathname,
            'tags': self.tags,
            'type': self.message_type,
            #'request': self.record

            # Extra Fields
            'level': record.levelname,
            'logger_name': record.name,
        }

        # Add extra fields
#        print(type(self.get_extra_fields(record)['request']))
        message.update(self.get_extra_fields(record))

        # If exception, add debug info
        if record.exc_info:
            message.update(self.get_debug_fields(record))

        return self.serialize(message)

As our requirement is to log every request our middleware may look like following:

import logging

request_logger = logging.getLogger('sadaf_logger')
from datetime import datetime
from django.utils.deprecation import MiddlewareMixin
class LoggingMiddleware(MiddlewareMixin):
    """
    Provides full logging of requests and responses
    """
    _initial_http_body = None
    def __init__(self, get_response):
        self.get_response = get_response

    def process_request(self, request):
        self._initial_http_body = request.body # this requires because for some reasons there is no way to access request.body in the 'process_response' method.


    def process_response(self, request, response):
        """
        Adding request and response logging
        """
#        print(response.content, "xxxx")
        if request.path.startswith('/') and \
                (request.method == "POST" and
                         request.META.get('CONTENT_TYPE') == 'application/json'
                 or request.method == "GET"):
            status_code = getattr(response, 'status_code', None)
            print(status_code)

            if status_code:
                if status_code >= 400:
                    log_lvl = logging.ERROR
                else:
                    log_lvl = logging.INFO

            #request_logger.log(logging.DEBUG,)
            request_logger.log(log_lvl,
                               "GET: {}"
                               ""
                               .format(
                                   request.GET,
                                   ), 
                                   extra ={
                                       'request': request,
                                       'request_method': request.method,
                                       'request_url': request.build_absolute_uri(),
                                       'request_body': self._initial_http_body.decode("utf-8"),
                                       'response_body':response.content,
                                       'status': response.status_code
                                   }
                                       #extra={
                    #'tags': {
                    #    'url': request.build_absolute_uri()
                    #}
                #}
                )
#            print(request.POST,"fff")
        print("hot")
        return response

So pretty much you are done. Go login to your Kibana dashboard, make index pattern that you are interest and see your log:

Sample AWS CodeDeploy configuration for django

AWS has its own continuous integration tool known as CodeDeploy, using a simple command you would be able to deploy on multiple servers when you want to change something on code base.

Installing code deploy to instance

If code deploy client is not installed at your instance, you would need to do that:

sudo yum install -y ruby wget
cd /opt
wget https://aws-codedeploy-ap-south-1.s3.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto

Create CodeDeploy Application

You have to create Code Deploy application with Deployment type to Inplace deployment, and deployment Configuration set to CodeDeployDefault.OneAtATime.
Give it a name under Ec2 configuration and Amazon ec2 instance, say the name is Code deploy instance. Now you have to add the same tag to all your code deploy instances.

Set IAM Permissions

Now that we are done with installation, we would need to setup IAM rules:
First create an IAM group called CodeDeployGroup. This group needs AmazonS3FullAccess and AWSCodeDeployFullAccess permissions. Create a user and add it to this group. This user only needs programmatic access.Save key and key id to somewhere safe.

Create role that has Trusted entities and Policies are ec2.amazonaws.com and AWSCodeDeployRole AmazonS3FullAccess, respectively.

Edit trust relationship to following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
            "ec2.amazonaws.com",
            "codedeploy.ap-south-1.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create new s3 bucket with previously created IAM rules.

CodeDeploy configuration

My codebase  structure is something like following:

- src
  - <django project>
- scripts
   install_dependencies
   start_server
   stop_server
appspec.yml
codedeploy_deploy.py
deploy.sh

appspec.yml is the file that contains our hooks and configuration for code deploy.

version: 0.0
os: linux
files:
  - source: src
    destination: /home/centos/proj_name
hooks:
  BeforeInstall:
    - location: scripts/install_dependencies
      timeout: 300
      runas: root
  ApplicationStop:
    - location: scripts/stop_server
      timeout: 300
      runas: root
  ApplicationStart:
    - location: scripts/start_server
      timeout: 300
      runas: root

for django scripts/install_dependencies may look like following:

sudo yum install -y gcc openssl-devel bzip2-devel wget
sudo yum install -y make git
cd /opt
command -v python3.6 || {
    wget https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz
    tar xzf Python-3.6.3.tgz
    cd Python-3.6.3
    sudo ./configure --enable-optimizations
    sudo make altinstall
}
sudo yum install -y mysql-devel

for scripts/start_server I have following:

cd /home/centos/evaly
pip3.6 install -r requirements.txt
nohup uwsgi --http :80 --module evaly.wsgi > /dev/null 2>&1 &

for scripts/stop_server I have following:

pkill uwsgi

I have borrowed a python script from bitbucket team which looks like following:

# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
# except in compliance with the License. A copy of the License is located at
#
#     http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed on an "AS IS"
# BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under the License.
"""
A BitBucket Builds template for deploying an application revision to AWS CodeDeploy
narshiva@amazon.com
v1.0.0
"""
from __future__ import print_function
import os
import sys
from time import strftime, sleep
import boto3
from botocore.exceptions import ClientError

VERSION_LABEL = strftime("%Y%m%d%H%M%S")
BUCKET_KEY = os.getenv('APPLICATION_NAME') + '/' + VERSION_LABEL + \
    '-bitbucket_builds.zip'

def upload_to_s3(artifact):
    """
    Uploads an artifact to Amazon S3
    """
    try:
        client = boto3.client('s3')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False
    try:
        client.put_object(
            Body=open(artifact, 'rb'),
            Bucket=os.getenv('S3_BUCKET'),
            Key=BUCKET_KEY
        )
    except ClientError as err:
        print("Failed to upload artifact to S3.\n" + str(err))
        return False
    except IOError as err:
        print("Failed to access artifact.zip in this directory.\n" + str(err))
        return False
    return True

def deploy_new_revision():
    """
    Deploy a new application revision to AWS CodeDeploy Deployment Group
    """
    try:
        client = boto3.client('codedeploy')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False

    try:
        response = client.create_deployment(
            applicationName=str(os.getenv('APPLICATION_NAME')),
            deploymentGroupName=str(os.getenv('DEPLOYMENT_GROUP_NAME')),
            revision={
                'revisionType': 'S3',
                's3Location': {
                    'bucket': os.getenv('S3_BUCKET'),
                    'key': BUCKET_KEY,
                    'bundleType': 'zip'
                }
            },
            deploymentConfigName=str(os.getenv('DEPLOYMENT_CONFIG')),
            description='New deployment from BitBucket',
            ignoreApplicationStopFailures=True
        )
    except ClientError as err:
        print("Failed to deploy application revision.\n" + str(err))
        return False     
           
    """
    Wait for deployment to complete
    """
    while 1:
        try:
            deploymentResponse = client.get_deployment(
                deploymentId=str(response['deploymentId'])
            )
            deploymentStatus=deploymentResponse['deploymentInfo']['status']
            if deploymentStatus == 'Succeeded':
                print ("Deployment Succeeded")
                return True
            elif (deploymentStatus == 'Failed') or (deploymentStatus == 'Stopped') :
                print ("Deployment Failed")
                return False
            elif (deploymentStatus == 'InProgress') or (deploymentStatus == 'Queued') or (deploymentStatus == 'Created'):
                continue
        except ClientError as err:
            print("Failed to deploy application revision.\n" + str(err))
            return False      
    return True

def main():
    if not upload_to_s3('/Users/sadafnoor/Projects/evaly/artifact.zip'):
        sys.exit(1)
    if not deploy_new_revision():
        sys.exit(1)

if __name__ == "__main__":
    main()

I have written a script to zip up my source code so that the script can upload it to s3 and eventually all my ec2 instances will be downloading that zip from s3.

export APPLICATION_NAME="CodeDeployApplicationName" 
export AWS_ACCESS_KEY_ID="IAMUserKeyId"
export AWS_DEFAULT_REGION="ap-south-1"

export AWS_SECRET_ACCESS_KEY="IAMUserSecretKey"
export DEPLOYMENT_CONFIG="CodeDeployDefault.OneAtATime"

export DEPLOYMENT_GROUP_NAME="CodeDeployDeploymentGroup"
export S3_BUCKET="S3BucketName"
zip -r ../artifact.zip src/* appspec.yml scripts/*
python codedeploy_deploy.py

Running a python/django app containerised in docker image pushed on private repo on top of kubernetes cluster

Recently I was looking for more flexible way to ship our code in production,   docker and kubernetes are the sweetheart of the devops engineers. Docker is something that let us containerise our app in an image and then we let that image to run on production. I know that we all have some experience with Virtual Machines, it is easy to get confused about the difference that will stop you from appreciating what docker does. A virtual machine works separately on top of the hypervisor of your computer on the other hand docker creates another layer of abstraction on top of your OS. It lets you share the similarities the images that you already have on your OS, and on top of that it adds another layer that has the differences. Now we can run multiple linux image on top of one machine without costing double. So it optimises, it is intelligent and it saves us. When we are running a cluster of n nodes, the complexity exponentially, what if something broke somewhere in a docker container in a cluster of n nodes? How can we ensure that which docker container should be run on which node? How do we move that docker container to another node because that node is going to be turned off for maintenance? We need a manager, who takes care of them, don’t we? Kubernetes comes along takes the responsibility.

First of all lets setup kubernetes.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

We would turn the selinux off as the documentation says so.

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now we would be installing docker, kubeadm, and in more abstract kubernetes.

yum install -y docker kubelet kubeadm kubectl --disableexcludes=kubernetes

We would need to ensure that these are the service that should be the first thing it should be doing when the computer turns on:

systemctl enable kubelet && systemctl start kubelet
systemctl enable docker && systemctl start docker

Now that we have our kubernetes and docker running on master and slave node. We would need to change one or two things in configuration for a safe initial launching on master.

vi /var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd
#--network-plugin=cni

We are initializing the node on master:

kubeadm init

for future token creation

sudo kubeadm token create --print-join-command 

It should generate a token for you which you would need to copy and paste on your slave node, but before that you would be need to put configuration file in proper directory with proper permission.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

on slave server

kubeadm join 10.0.15.10:6443 --token vzau5v.vjiqyxq26lzsf28e --discovery-token-ca-cert-hash sha256:e6d046ba34ee03e7d55e1f5ac6d2de09fd6d7e6959d16782ef0778794b94c61e

if you are getting something similar:

I0706 07:18:56.609843    1084 kernel_validator.go:96] Validating kernel config
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

 Pulling images required for setting up a Kubernetes cluster

running following would help

for i in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4; do modprobe $i; done

Now if you run following in server, you would know that you have some nodes attached to your kubernetes cluster.

sudo kubectl get nodes
sudo kubectl describe nodes

if you had issues dealing with nodes:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

For this demonstration I will be adding an existing python/django application on this cluster. So first of all I would be needing to dockerize that first.

at my Dockerfile

# set the base image 
FROM python:3.7
# File Author / Maintainer
MAINTAINER Sadaf
#add project files to the usr/src/app folder

#set directoty where CMD will execute 
WORKDIR /usr/src/app
ADD app_name ./app_name
COPY /app_name/requirements.txt .
# Get pip to download and install requirements: --no-cache-dir 
RUN pip install -r requirements.txt
# Expose ports
EXPOSE 8000
# default command to execute
WORKDIR /usr/src/app/app_name
RUN chmod +x app_name/gunicorn.sh
CMD ./app_name/gunicorn.sh
#ENTRYPOINT ["/bin/bash", "app_name/gunicorn.sh"]

Now that we have a dockerfile ready. Lets build that image:

sudo docker build -t app_name_api_server .

Time to run that image and expose it on port 8000

sudo docker run -p 8000:8000 -i -t app_name_api_server

If you like what you see on localhost:8000! Congratulations! Your app is working on docker. Now let’s push that image on docker hub. For me I have created a private repo on docker hub. To be able to push your image on docker hub you would be needing to add tags to that image first then you can push it.

sudo docker tag app_name_api_server sadaf2605/app_name_api_server
sudo docker push sadaf2605/app_name_api_server

Now that you have your image pushed on docker hub. Now we will go back to our kubernetes master. As the image that we want to pull are on private image, no wonder we would need some sort of credential to pull it.

DOCKER_REGISTRY_SERVER=docker.io
DOCKER_USER=Type your dockerhub username, same as when you `docker login`
DOCKER_EMAIL=Type your dockerhub email, same as when you `docker login`
DOCKER_PASSWORD=Type your dockerhub pw, same as when you `docker login`

kubectl create secret docker-registry myregistrykey \
  --docker-server=$DOCKER_REGISTRY_SERVER \
  --docker-username=$DOCKER_USER \
  --docker-password=$DOCKER_PASSWORD \
  --docker-email=$DOCKER_EMAIL

Now lets define and yaml file that we are going to define our kubernetes deployments, app_name.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-name-api-server
spec:
  selector:
    matchLabels:
      run: app-name-api-server
  replicas: 1
  template:
    metadata:
      labels:
        run: app-name-api-server
    spec:

      containers:
      - name: app-name-api-server
        image: index.docker.io/sadaf2605/app_name_api_server:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
          #hostPort: 8000
        env:
        - name: DB_USERNAME
          value: "user"
        - name: DB_PASSWORD
          value: "password"
        - name: DB_NAME
          value: "dbname"
        - name: DB_HOST
          value: "1.2.2.3"
      imagePullSecrets:
      - name: myregistrykey
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 8.8.8.8
      imagePullSecrets:
      - name: myregistrykey
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  upstreamNameservers: |
    ["8.8.8.8"]

Now time to run a deployment using that configuration:

sudo kubectl apply -f app_name.yaml

Lets check if we have a deployment or not:

sudo kubectl get deployments

Lets check if any instance of our docker container is running or not.

sudo kubectl get pods

Now we would be creating a service that would let us access to these pods from outside of pods.

sudo kubectl expose deployment app-name-api-server --type=LoadBalancer --name=app-name-api-server
sudo kubeadm upgrade plan --feature-gates CoreDNS=true

CentOS postgis setup: You need JSON-C for ST_GeomFromGeoJSON

I am actually struggling with this error on our centos server all day, this is how I fixed it. I could not recover everything from my commandline history but this is what I could find out. Hope it will be helpful. I am sure while cleaning up my server for production it will be helpful to me. So cheers!

All this started with this error message on my centos server:

npm-2 Unhandled rejection SequelizeDatabaseError: You need JSON-C for ST_GeomFromGeoJSON
npm-2     at Query.formatError (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
npm-2     at null. (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Query.handleError (/home/centos/jobcue.com/node_modules/pg/lib/query.js:108:8)
npm-2     at null. (/home/centos/jobcue.com/node_modules/pg/lib/client.js:171:26)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Socket. (/home/centos/jobcue.com/node_modules/pg/lib/connection.js:109:12)
npm-2     at emitOne (events.js:77:13)
npm-2     at Socket.emit (events.js:169:7)
npm-2     at readableAddChunk (_stream_readable.js:153:18)
npm-2     at Socket.Readable.push (_stream_readable.js:111:10)
npm-2     at TCP.onread (net.js:531:20)

When I tried to install json-c on server it was like:

sudo yum install json-c
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.eecs.wsu.edu
 * epel: s3-mirror-us-west-2.fedoraproject.org
 * extras: linux.mirrors.es.net
 * updates: mirror.raystedman.net
Package json-c-0.11-4.el7_0.x86_64 already installed and latest version
Nothing to do

Then I started panicing. Tried 5-6 hours long yum battles and figure out a solution that would look like following:

Install some dependencies at first:

yum install geos-devel.x86_64
yum install proj-devel.x86_64
yum install gdal-devel.x86_64
yum install libxml2-devel.x86_64
yum install json-c-devel.x86_64

yum install postgresql92-devel
sudo yum install postgresql-server 

sudo yum install geos geos-devel
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
gzip -d proj-4.8.0.tar.gz
tar -xvf proj-4.8.0.tar
cd proj-4.8.0
./configure
make
sudo make install

I needed to install gdal:

installing gdal:

sudo rpm -Uvh http://elgis.argeo.org/repos/6/elgis-release-6-6_0.noarch.rpm
sudo yum install -y gdal
./configure
make
make install

Obviously I needed to install json c:

sudo yum install json-c-devel

I needed to know where it is located:

rpm -ql json-c json-c-devel

for me it was at:

/usr/include/*

Now it is time to built our postgis like this:

wget http://download.osgeo.org/postgis/source/postgis-2.2.1.tar.gz
tar xvzf postgis-2.2.1.tar.gz
cd postgis-2.2.1
./configure --with-jsonc=/usr/include

make
make install
sudo make install

Install PostGIS on your ubuntu 15.10

It took me quite a lot of time to figure out how do you do that! Though it looks like it is just 3 apt get install command, so simple right? but took quite a toll of my life to figure those 3 out. If you are reading this, probably you are also going through something semilar!

>sudo apt-get install postgresql postgresql-contrib postgis
> psql --version
psql (PostgreSQL) 9.5.2

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

>sudo apt-get install postgresql-9.5-postgis-2.0

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

> sudo apt-get install postgresql-9.5-postgis-scripts
> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
CREATE EXTENSION