Linux namespaces and how docker internally uses those namespaces

While using docker it is quite natural to feel like we are dealing with a virtual machine because we can install something in a container, it has an IP, file tree is different and whatnot. In reality, containers are just a mechanism that allows Linux processes to run with some degree of isolation and separation. Containers basically use few different primitives in Linux combined. By using Linux namespaces containers can make these happen.

 

What are namespaces? Namespaces are another Linux kernel primitive that can be used to control access and visibility of resources. Generally, when properties like names or permissions change, it is visible only within the namespace and it is not visible to the processes outside of it. So it makes it look like that the process has its own isolated copy of a given resource. Docker uses these properties of the namespace to provide isolation. Namespaces can map resources outside a namespace to a resource inside a namespace. Linux has at least 7 kinds of namespaces that cover different kinds of resources: Mount, UTS, IPC, PID, Network, User, Cgroup. Sometimes we these namespaces individually and sometimes we have to use them together because types like IPC, Cgroup, PID use file systems that need to be mounted. A name server does not stay open when there is no running process on it or a bind mount. That’s why a docker gets killed when the process inside it gets killed, it is because of how Linux System deals with its PID 1, which I am going to discuss later in this blog.

How would I find out the namespace of a process? We can find any information about a process by accessing the proc virtual file system. If we check carefully we should be able to find a directory called ns which contains symbolic links to the namespace. These symbolic link structures can provide us the information about the namespace type and inode number that the kernel uses. By comparing this inode number we can confirm if both of the processes are part of the same namespace or not.

readlink /proc/$$/ns/uts

How can we use namespace? A set of Linux syscall is used to work with namespaces, most commonly clone, unshare, etc. Clone is a sort of more general version of the fork system call that allows us to create a new child process.  It is controlled by a set of flags that we can if we want the new child process to be in some new namespaces and we want to move the process to move itself from one namespace that it is currently resident in, to another existing namespace. unshare supports the same flags as clone. unshare is a syscall that a running process can use to move into a new namespace. Namespaces automatically close when nothing is holding them open so it ceases to exist when the running process stops or kills. Creating a user namespace requires no privilege but for another namespace, it requires privilege. So time to time you will be needed to use sudo. We can use commands like setns, nsenter, and ip netns to switch or enter from one namespace to another namespace.

Every docker container has its own hostname. Docker uses UTS namespace to achieve it. UTS namespaces isolate two system identifiers of node name (generally known as hostname) and the NIS domain name. On a  Linux system, there might be multiple UTS namespaces. If one of those processes changes the hostname the change will be visible to the other processes within that namespace.

Lets use unshare command to create new namespaces and execute some command in those namespaces.

echo $$

readlink /proc/$$/ns/uts

sudo unsure -u bash

echo $$

readlink /proc/$$/ns/uts

hostname new_host_name

readlink /proc/$$/ns/mnt

nsenter -t PID -u bash

Another fascinating feature of a Linux container is that it is successful to make us believe that every container has a completely different set of files but in reality, it uses Linux mount namespaces to provide a separate view of the container image mounted on the containers root file system. Mount namespace allows us to isolate a set of mount points seen by a process. Mount points are just tuples that keep track of the device, its pathname, and the paren of the mount. By tracking parents, mount point helps us to maintain a single directory hierarchy. So across namespaces, we can have processes that have a different set of mount points and they see a completely different single directory hierarchy tree. It hides the host file system from view to share data across containers or between the host and a container.  Volumes are really amount added to the container’s mount namespace.  There are ways to share subtrees that allow us to propagate mount points across different namespaces.

Docker makes sure that from one container we don’t access another process of a container. PID namespaces play a great role in it. PID namespace isolates process id. We can only see the id of a process when the process that is part of the same PID namespaces. Since every PID in a namespace is isolated and unique in that isolated space, we don’t need to worry about potential conflicts of id. We can use the PID namespace in many ways, if we want to freeze a process and move it to another core or another machine, we can do that without thinking about the conflict it may cause in terms of PID conflict. PID namespace also allows us to use PID 1 across different namespaces. PID 1 is a special thing on any UNIX system. PID 1 is the parent of all orphaned processes and is responsible for doing certain system initialization. PID namespace is different from all other namespaces. PID namespace maintains a hierarchy. This type of hierarchy has a depth of 32. Each PID namespace has a parent PID namespace that has been initiated by clone() or unshare().  For when the fork happens the process stays at the same level but when clone happens it goes to a lower level. All these 32 generations are initiated from PID 1 on that process. So as a parent of all processes, PID 1 can see all the processes in the namespace, but lower-level PIDs can’t see higher-level PIDs. When PID 1 goes down or gets killed Linux kernel panics and gets restarted. When PID 1 in a namespace goes away, the namespace is considered unusable

sudo unshare -p -f bash
echo $$ 
ls /proc/

We are going to see many processes, but we should not see any processes, because PID should not have any processes yet, right? It is because we unshare only the PID namespace, and totally forgot to unshare mount point, so we can still have access to all processes that are part of PID1 across all namespaces.

Now let’s unshare mount and PID namespace together:

sudo unshare  -m -p -f bash

mount -t proc none /proc

docker typically uses a separate network namespace per container. Network namespace a separated view of the network with different network resources like routing table rules, firewall rules the socket port, some directories in /proc/net and sys/class/net, and so on. Network namespaces can be connected using a virtual ethernet device pair or veth pair and these vets pair are being isolated using network namespace. These veth pair are connected to a Linux bridge to enable outbound connectivity. Kubernetes pods and ECS tasks get a unified view of the network and a single IP address exposed to address all of the containers in the same pod or task. Every container has its own socket port number space. It can use some NAT rules on the container we could run a web server on port 80.

ip link

sudo unshare —net ip link

touch /var/run/netns/new

mount —bind /proc/$$/ns/net /var/run/netns/new

exit


docker run —name busybox  —detach busybox

ps ax | grep busybox

sudo redlink /proc/PID/ns/net

docker inspect busybox | jq .[0].NetworkSettings.IPAddress

nsenter --target PID --net ifconfig eth0

ip link | grep veth

ifconfig vethxxxxxx

 

Since dockers are just a clever use of Linux namespaces, we can interchangeably use all namespace-related commands and tools to debug dockers as well. For example, to enter namespaces we can use commands like setns, nsenter, ip netns. To demonstrate that we can run a docker image, and find the process id of that image, and then let’s use nsenter command to access to enter the namespace and run commands inside that namespace that has been created by docker.

docker run —name redis  —detach redis
ps ax | grep redis
nsenter —target PID —mount /usr/local/bin/redis-server


			
					

How docker internally handles resource limit using linux control groups

While using docker it is quite natural to feel like we are dealing with a virtual machine because we can install something in a container, it has an IP, we can ssh it, we can allocate memory and CPU to it like any virtual machine but in reality, containers are just a mechanism that allows Linux processes to run with some degree of isolation and separation. Containers basically use few different primitives in Linux combined. If containers are just based on Linux primitives how are we being able to set limits to memories and CPUs? The answer is very simple and it is already available in the Linux system, docker takes leverage of Linux Control groups.

What are Control groups? Control groups are commonly known as cgroups. Cgroups are the abstract frameworks in Linux systems for tracking, grouping, and organizing Linux processes. No matter what process it is, every process in a Linux system is tracked by one or more cgroups. Typically cgroups are used to associate processes with a resource. We can leverage cgroups to track how much a particular group of processes is using for a specific type of resource. Cgroup plays a big role when we are dealing with multitenancy because it enables us to limit or prioritize specific resources for a group of processes. It is particularly necessary because we don’t want one of our resources to consume all the CPU or say io bandwidth. We can also lock a process to run a particular CPU core using cgroup as well.

We can interact with the abstract framework of cgroups through subsystems. In fact, the subsystems are the concrete implementations that are bound to resources. Some of the Linux Subsystems are Memory, CPU time, PIDs, Devices, Networks. Each and every subsystem are independent of each other. They have the capability to organize their own processes separately. One process can be part of two independent cgroups. All of the c group subsystems organize their processes in a structured hierarchy. Controllers are responsible for distributing specific types of resources along the hierarchy Each subsystem has an independent hierarchy. Every task or process ID running on the host is represented in exactly one of the c groups within a given subsystem hierarchy. These independent hierarchies allow doing advanced process-specific resource segmentation. For example when two processes share the total amount of memory that they consume but we can provide one process more CPU time than the other.

This hierarchy is maintained in the directory and file structure. by default, it is mounted in /sys/fs/cgroup the directory but it can be mounted in another directory of choice as well. We can also mount multiple cgroup locations in multiple locations as well, which comes in handy when a single instance is using by multiple tenants and we want one tenant cgroup to be mounted on his disk area. Mounting cgroup can be done using the command:

mount -t cgroup2 none $MOUNT_POINT

Now lets explore the virtual filesystem of cgroup:

ls /sys/fs/cgroup
ls /sys/fs/cgroup/devices

Some of the resource controllers apply settings from the parent level to the child level, an example of such controller would be the devices controller. Others consider each level in the hierarchy independently, for example, the memory controller can be configured this way. In each directory, you’ll see a file called tasks. This file holds all of the process IDs for the processes assigned to that particular cgroup.

cat /sys/fs/cgroup/devices/tasks

It shows you all of the processes that are in this particular cgroup inside this particular subsystem but suppose that you have an arbitrary process and you want to find out which c groups it’s assigned to you can do that with the proc virtual file system. Let’s look at that the proc file system contains a directory that corresponds to each process id.

ls /proc

To see our current process we can use the command:

 echo $$
cat /proc/<pid>/cgroups

Let’s say I want to monitor a group for how much memory it is using. We can read the virtual file system and see what it returns.

cat /sys/fs/cgroup/memory/memory.usage_in_bytes

What are seeing these files and directories everywhere? It just interfaces into the kernel’s data structures for cgroups. Each directory has a distinct structure. Even if you create a new directory, it will automatically create a bunch of files to match up. Let’s create a cgroup. To create a cgroup, we just need to create a directory, at least this is how the kernel keeps track of it.

sudo mkdir /sys/fs/cgroup/pids/new_cgroup
ls /sys/fs/cgroup/pids/new_cgroup
cat /sys/fs/cgroup/pids/new_cgroup/tasks

The kernel keeps track of these processes using this directory and files. So adding or removing processes or changing any settings is nothing but changing the content of the files.

cat /sys/fs/cgroup/pids/new_cgroup/tasks
echo $$ | sudo tee /sys/fs/cgroup/pids/new_cgroup/tasks

You have written one line to the task files, haven’t you? Now let’s see what it is inside the tasks file:

cat /sys/fs/cgroup/pids/new_cgroup/tasks

 

We are supposed to see at least 2 files because when a new process is started it begins in the same cgroups as its parent process. When we try to use command cat, our shell starts another process and it appears in the tasks file.

 

We can limit the number of processes that a cgroup is allowed to run by modifying pids.max file.

Echo 2 | sudo tee /sys/fs/cgroup/pids/new_cgroup/pids.max

Now let’s try to run 3 processes instead of two and it is going to crash our shell.

 

Now that we have a basic understanding of cgroups. Let’s investigate cgroups inside our docker containers.

Let’s try to run a docker container with a CPU limit of 512 and explore the cgroups.

docker run —name test —cpu-shares 512 -d —rm buxybox sleep 1000

docker exec demo ls /sys/fs/cgroup/cpu

docker exec demo ls /sys/fs/cgroup/cpu/cpu.shares

 

So basically docker is using my commands to manipulate the group setting files to get things done. Interesting indeed. If it is not a virtual machine, these files are supposed to be on our machine too, isn’t it? yes, you got it right. Usually, these files are located somewhere in /sys/fs/cgroup/cpu/docker. Usually, there is a directory with a 256 hash that contains the full docker id.

ls /sys/fs/cgroup/cpu/docker
cat /sys/fs/cgroup/cpu/docker/<256big>/cpu.shares

Cgroup mechanism is tied to namespaces. If we have a particular namespace defined then we can add the namespace to the cgroup and everything that is a member of the cgoup becomes controlled by the cgroup.

Just want to give a word of warning before modifying anything of a cgroup we should keep in mind that dependencies there might be on the existing hierarchy. For example, amazon ECS or google c advisor uses the secret hierarchy to know where to read CPU and memory utilization information.

CentOS postgis setup: You need JSON-C for ST_GeomFromGeoJSON

I am actually struggling with this error on our centos server all day, this is how I fixed it. I could not recover everything from my commandline history but this is what I could find out. Hope it will be helpful. I am sure while cleaning up my server for production it will be helpful to me. So cheers!

All this started with this error message on my centos server:

npm-2 Unhandled rejection SequelizeDatabaseError: You need JSON-C for ST_GeomFromGeoJSON
npm-2     at Query.formatError (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
npm-2     at null. (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Query.handleError (/home/centos/jobcue.com/node_modules/pg/lib/query.js:108:8)
npm-2     at null. (/home/centos/jobcue.com/node_modules/pg/lib/client.js:171:26)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Socket. (/home/centos/jobcue.com/node_modules/pg/lib/connection.js:109:12)
npm-2     at emitOne (events.js:77:13)
npm-2     at Socket.emit (events.js:169:7)
npm-2     at readableAddChunk (_stream_readable.js:153:18)
npm-2     at Socket.Readable.push (_stream_readable.js:111:10)
npm-2     at TCP.onread (net.js:531:20)

When I tried to install json-c on server it was like:

sudo yum install json-c
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.eecs.wsu.edu
 * epel: s3-mirror-us-west-2.fedoraproject.org
 * extras: linux.mirrors.es.net
 * updates: mirror.raystedman.net
Package json-c-0.11-4.el7_0.x86_64 already installed and latest version
Nothing to do

Then I started panicing. Tried 5-6 hours long yum battles and figure out a solution that would look like following:

Install some dependencies at first:

yum install geos-devel.x86_64
yum install proj-devel.x86_64
yum install gdal-devel.x86_64
yum install libxml2-devel.x86_64
yum install json-c-devel.x86_64

yum install postgresql92-devel
sudo yum install postgresql-server 

sudo yum install geos geos-devel
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
gzip -d proj-4.8.0.tar.gz
tar -xvf proj-4.8.0.tar
cd proj-4.8.0
./configure
make
sudo make install

I needed to install gdal:

installing gdal:

sudo rpm -Uvh http://elgis.argeo.org/repos/6/elgis-release-6-6_0.noarch.rpm
sudo yum install -y gdal
./configure
make
make install

Obviously I needed to install json c:

sudo yum install json-c-devel

I needed to know where it is located:

rpm -ql json-c json-c-devel

for me it was at:

/usr/include/*

Now it is time to built our postgis like this:

wget http://download.osgeo.org/postgis/source/postgis-2.2.1.tar.gz
tar xvzf postgis-2.2.1.tar.gz
cd postgis-2.2.1
./configure --with-jsonc=/usr/include

make
make install
sudo make install

Install PostGIS on your ubuntu 15.10

It took me quite a lot of time to figure out how do you do that! Though it looks like it is just 3 apt get install command, so simple right? but took quite a toll of my life to figure those 3 out. If you are reading this, probably you are also going through something semilar!

>sudo apt-get install postgresql postgresql-contrib postgis
> psql --version
psql (PostgreSQL) 9.5.2

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

>sudo apt-get install postgresql-9.5-postgis-2.0

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

> sudo apt-get install postgresql-9.5-postgis-scripts
> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
CREATE EXTENSION

Apparently your ubuntu can do blue screen too! Here is how to fix blue screen in ubuntu

I did not know that one can have blue screen in ubuntu as well. But if you know it already that don’t panic, this is how to fix blue screen in ubuntu. It is probably happened because of problem in upgrading distro.


$ sudo su
$ sudo apt-get install libgdk-pixbuf2.0-dev
$ cd /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/
$ find 2.10.0/loaders/ > ~/pixbuf-files
$ nano ~/pixbuf-files
# and delete 1st line 2.10.0/loaders/

$ cat ~/pixbuf-files | xargs -n1 gdk-pixbuf-query-loaders > 2.10.0/loaders.cache
$ reboot

How ubuntu server boots up: V init vs upstream

How many time did I boot my labtop today? Well, I booted up twice, usually I never move my ass out, but shockingly today I just did. Don’t get too curious about where did I go, because thats not the point, the point of curiosity is what happens when a ubuntu server boots up. Thats what we will try to figure out in this blog. Unlike any other Linux distributions, Ubuntu uses a different startup process for services, known as Upstart. As it has backward compatibility, the difference remain unnoticed most of the time.

When Ubuntu server starts up, first thing it does is it starts up GRUB boot loader. GRUB boot loader stays at least partially at the boot code on the Master Boot Record (the first 512 bytes of hard drive). It selects which Linux kernel the system will boot from and which options to uses when it boots.

When we look at /boot/grub/grub.cfg or /etc/default/grub, we see references of a program called update-grub. This is a helper program that automates the update of GRUB configuration file when new kernels are added. It executes number of configuration scripts stored at /etc/grub.d. When we select a kernel to boot from GRUB menu, it loads the kernel into memory along with its initrd file (initial RAM disk). The initrd file is actually a gzipped cpio archive known as an initramfs file under Ubuntu.  example gz file is initrd.img-2.6.32-14-generic-pae.

When a kernel boots, it needs to be able to at least mount the root file system so that it can access basic configuration files, kernel modules, and system binaries.

Now with the increase of hardware and supporting file systems, it makes sense to support them only if that is necessary. It keeps kernel smaller and flexible.

It needs to accessthe files to mount root file system. The initramfs file provides the kernel the essential kernel modules and system binaries it needs to have to mount theroot file system and complete the boot process.  Grub provide the information about root file system.

When kernel boots, it extracts the initramfs into RAM and runs a script called init. It basically creates some system mount points and mounts the actual root partition. Finally, after this init script has mountedthe real root file system, its last task is to run the /sbin/init program on theroot file system, which starts the next phase of the boot process.

The /sbin/init program is the parent process of every program running onthe system. This process always has a PID of 1 and is responsible for starting the rest of the processes that make up a running Linux system.

UNIX like OS has few standards to initialize, most of the known distributions are using System V init model but Ubuntu Server has switched to a systemknown as Upstart. Ubuntu still some features of System V init such as runlevels and /etc/rc?.d directories for backwardcompatibility. The good thing about upstart is that it manages everything under the hood.

In this V init system, different system states are  known as runlevels. When V init system starts it goes through the configuration file located at /etc/inittab and discovers its default runlevel. Then it enters to that runlevel and starts processes that has been configured to run at that runlevel. Runlevels are labeled by numbers ranging from 0 to 6. For an instance, runlevel 0 is reserved for a halted system state. When we enter runlevel 0, the system shuts down all running processes, unmounts all file systems, and powers off. Likewise, runlevel 6 is reserved for rebooting the machine. Runlevel 1 is reserved for single-user mode a state where only a single user can log in to the system with only few process running which comes very handy for digonosis. Even in the default GRUB menu you will notice a recovery mode option that boots you into runlevel 1.

Runlevels 2 through 5 are left for distributions and us to define. So we can create our own runlevels. Traditionally in Linux distributions one runlevel is allocated for graphical desktop (eg. runlevel 5 of Red Hat) and another runlevel for a system with no graphics (Eg. runlevel 3 of RedHat). User also has scope to create his own run level, for instance, maybe starting up a system without network access some time could come handy, so we can define it as a runlevel . In that case we need to pass an argument at boot prompt to override the default runlevel with desired runlevel. Once the system is booted, we can change the current runlevel with the init commands using sudo init 1.

/etc/init.d directory contains all of the start-up scripts of all services of all runlevels. This scripts usually contains start and stop commands.

After the choice of runlevel, init goes to /etc/rcS.d and runs each script that begins with an S in numerical order with start as an argument. Finally init is finished but stays running in the background, waiting for the runlevel to change.

init scripts has few draw backs for an instance, if a service dies in before completing the task it does not automatically starts the process. So we need another tool to monitor this process succeedeed or not. Init scripts is are generally affected by either change in runlevel or when the system starts up but some reason not executed. A perfect example would be Init scripts that depend on a network connection. On Ubuntu the init script that establishes the network connection is called networking. As we know it follow a neurmaric sequence, any init scripts that depend on a network connection are named with a higher number than this init script to ensure they run after the networking script has run. Lets imagine a situation where you boot up your system at the time when your network cable is unplugged. So in V init system, network init will run and failed and other connection will time out one by one.

It was designed not only to address some of the shortcomings of the System V init process, but also to provide a more robust system for managing services. Upstart solves this problem because upstart is event driven. Upstart can be configured to take action based on those events. Some sample events might be system start-up, system shutdown, the Ctrl-Alt-Del sequence being pressed, the runlevel changing, or an Upstart script starting or stopping. Upstart also constantly monitors the system for certain events to occur, and when they do,

Upstart does not completely replace System V init, or the functionality of init and the /etc/inittab files or changes of runlevels, but instead more core functionality is being ported to Upstart scripts. The difference is that Upstart now starts and stops services when runlevels changes. Upstart script are defined with either the script or exec options. Exec option keeps track of its PID. The convention is to keep track of these PIDs in the /var/run/ directory. With the script option, Upstart treats the lines that follow as a shell script until it reaches the end script line. Upstart provides methods to check the status of Upstart jobs and start and stop them as appropriate. We can check the status, start, and stop Upstart scripts with the appropriately named status, start, and stop commands. For example we can use sudo /etc/init.d/networking status, ubuntu short hand of this command is sudo service networking status. To disable an init script from starting up, we need to use sudo update-rc.d -f servicename remove, and to enable a script we need to use sudo update-rc.d servicename defaults. When we need to write out own script, we should start from the sceleton provided by ubuntu at /etc/init.d/skeleton. init scripts reside in /etc/init.d and have symlinks in /etc/rc?.d directories, where ? is a number representing a runlevel. So when we create our own script we need to choose a value for rc?.d wisely. The lower the value early it runs. We need to be careful about the dependency.

In Ubunutu services are managed two ways, either through init scripts or using xinetd. Xinetd is an updated and resource efficient version of the classic inetd service. When a system boots a init scripted service starts, the service could sit idly for decades before it gets accessed, wasting valuable server resources. On the other hand xinetd listen to ports its child services uses. If a connection is made on one of the ports, xinetd will then spawn the service that corresponds to that port, and once the connection is finished, the service exits until it is needed again.

Thanks:
1. BRACU Ayesha Abed Library
2. Kyle Rankin and Benjamin Mako Hill
3. My boredom 😛

Ubuntu server directory structure: what and whys

I am writing this blog from a ubuntu desktop distro, but there are other versions of ubuntu, ubuntu cloud, ubuntu server and so on. Today basically I am interested in ubuntu server version in particular. We may or may not know the truth that Ubuntu has always been used as a server. The company which developed ubuntu, Canonical is using ubuntu for their server from the begining. So it has been a goal of this ubuntu. Later the desktop and server got their path separated for their own good.

When I installed my first ubuntu LVM, I actually followed the settings blindly, I followed what some random guy wrote on their blog or on youtube. But today I want to shed some light on linux directories. The trick is very simple, the more we can understand what our directories do, the more we can secure them and save our time to debug what went wrong.

/bin directory holds the core binaries like ps, ls, rm, mv, chmod, df etc that an administrators or a regular users may need to use.

/sbin directory contains only system binaries which is very crucial for mounting the rest of the system and recovering the system if it can’t boot. Programs like fsck, ifconfig, mkfs, route, and init here. Usually this commands are run by root users.

/lib directory usually contains core system libraries the system needs to complete the boot process and use the binaries under /bin and /sbin. All of the kernel’s modules are also found here under /lib/modules/.

/usr is intended to store all of the noncritical binaries and libraries for the system in UNIX like operating system. if we keep only the core binaries and libraries in /bin, /sbin, and /lib, we can keep the root partition small. Without any update /usr remains relatively static. So we could add an extra layer of security by mounting it read-only and remount it read-write when performing updates.

/usr/bin directory is similar to /bin, but it stores the rest of the binaries on the system that aren’t considered critical. Commands like man, gzip, nano, and other binaries intended for use both by administrators or regular users stays here.

/usr/sbin directory is similar to /sbin, only it stores binaries for administrator use that aren’t critical to booting. Commands like traceroute, chroot, and ntpdate along with a majority of the daemons that are started by init scripts like Web servers, ntp daemons, and mail servers are stored here.

/usr/lib holds the program libraries that supports in binaries under /usr/bin and /usr/lib.

/usr/local is used for custom service, binary or script of your own to the system.

/opt and /usr/local share the same purpose, it stores of third-party programs. Usually the program that installs under /opt are installed under their own directory (such as, say,
/opt/someprogram) and then create their own bin, sbin, and lib directories under there. On one hand this means we can remove a program by removing that directory under /opt, but at the same time it let our PATH environment variable grow rapidly. This directory stores kernel images, initramfs files, and also the GRUB configuration files.

/boot directory exists so that we can potentially separate it out into its own small mount point. When we want to experiment with LVM or an experimental file system for our root partition, we can format a separate /boot partition as ext2 or ext3 with no software RAID or LVM configured and be sure that our boot loader could read it.

/etc stores all the configuration files for the system and services. For example system startup scripts are stored at /etc/init and /etc/init.d. Keeping /etc separated makes it easy to back it up easily. It also makes the recovery process is easier.

/var directory was separated to store files and directories that could be vary in size and change frequently.

/var/log stores the system logs. Log has this tendency to grow out of control when the system is either under load or has a problem. When logs grow out of control, it fills up all the space of system and cause a crash.

/var/spool directory contains subdirectories that stores information of user crontabs, printer spools, and mail spools. In mail server this directory becomes very important. Like any other var it can grow quite large if the server spools a lot of mail for delivery.

/var/www directory won’t exist on all systems, but a Web server is run it will be the default place for the Web server’s docroot and cgi-bin directories.

/home holds all of the personal files for user accounts on that machine. It can also grow quite large. The advantage of separating this directory is that when we decide to change distributions, we could install the system on the root partition and overwrite what was there but preserve all user settings here.

/dev directory contains all of the device files of the system. These files include disk devices, keyboards, mice, and any other devices the system detects. On classic Linux systems these files. This directory is also pretty static, but on a modern Ubuntu server device files are often created on the fly by the udev program as devices are added or modules are loaded.

/media directory for these devices so it can keep /mnt for temporary mount points for nonremovable media such as floppy disks, CD-ROMs, and USB drives.

/proc is not an actual area on disk but it is a virtual file systems under Linux, It exists in RAM. Every process gets a directory full of live information, under /proc corresponding to its PID.It also stores virtual files related to the kernel process itself. We can query such things as the options passed to the kernel at boot (/proc/cmdline) or view settings for different kernel devices or other settings. A number of the kernel /proc files can also be used to not only read settings, but set them by writing to the corresponding file.

/sys directory is also a virtual file system. The files within /sys provide information about devices and drivers on your system, and the file system was created in part so that these sorts of files would no longer clutter up /proc. As with /proc, not only do many of the files within /sys provide information about parts of the system, but we can also write to various files to change settings on the fly.

/tmp is to store temporary files that don’t have to persist after a reboot. A common problem, though, is that a program might store far too much data in /tmp, and if it is part of the root partition, the entire root partition can fill up. A common culprit is vi. When you open a file with vi, a temporary copy of the file is placed in /tmp. I can’t count the number of times I’ve been paged because a server’s root partition was full, only to find out that a user used vi to open a 500Mb log file and dumped an equivalently sized temporary file under /tmp. I then had to kill the vi session and instruct the user on using a program such as less or more to view large text files.

Thanks:
1. BRACU Ayesha Abed Library
2. Kyle Rankin and Benjamin Mako Hill
3. My boredom 😛

ssl (https) from python flask

Basically here I am in this blog I am going share a code snippet, and I am going to describe what else stupid things I tried and did not work to do that. Well, don’t miss my point, when I am sharing my stupid ordeal, it does not mean, I am proving myself stupid but I am trying to save your time, I am basically telling what did not work in last couple of hour so that it could save your time.

So using flask when I shared a static file migrated directly from django, while working with django we have figured out that we actually don’t need powerful tool like django, instead we can use something very lightweight like flask and it is much easier to switch from django to flask as for both of them default front end templating is jinja. Now after this shift, I had to face a little bit trouble with https, because when I tried https, it is completely blank, I lost my mind, what the hell is going on here? Then I realized for flask probably I need to define my ssl. They got this nice documentation at their website (http://flask.pocoo.org/snippets/111/), I follwed them, and it did not work! Why it won’t work? Alright after couple of trial and google search I realized this is an old school way of doing this, fine, I need to go to new school, obviously! I got myself a new dress and now context looks more pretty {“cert.crt”,”key.key”}. I am impressed but what the hell? it did not work as well, why it won’t work? I lost my mind! hours of fighting, and I got this error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 602, in inner
    passthrough_errors, ssl_context).serve_forever()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 506, in make_server
    passthrough_errors, ssl_context)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 450, in __init__
    self.socket = ssl_context.wrap_socket(self.socket,
AttributeError: 'OpenSSL.SSL.Context' object has no attribute 'wrap_socket'

It is because I am using a python 2.7 version below 2.7.9.

What else did I try? You won’t want to know, I tried to install pyopenssl on heroku using pip, but looks like it is a ported version and failed to compile on heroku. Now I will write about what I had to do to make it work.

I have to make my cirtificate and keys:

$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
 openssl rsa -passin pass:x -in server.pass.key -out server.key
 rm server.pass.key
 openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

So now we have server.key and server.csr two files in our directory

Now python script:

from flask import Flask, request, send_from_directory
from flask import render_template

#from OpenSSL import SSL
#context = SSL.Context(SSL.SSLv23_METHOD)
#context.use_privatekey_file('server.key')
#context.use_certificate_file('server.crt')

import os

ASSETS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '/static')
app = Flask(__name__, static_url_path='/static')

@app.route('/js/')
def send_js(path):
    return send_from_directory('js', path)

@app.route('/signup')
def signup():
    return render_template('signup.html')

if __name__ == '__main__':
    context = ('server.crt', 'server.key')
    app.run(ssl_context=context, threaded=True, debug=True)

Done! Up and running if you got a version >=2.7.9.!