Sunday, March 30, 2014

How to use Simple Storage Service with Amazon EC2

Simple Storage Service is a pay as you go service from Amazon. There is no size limit for this service and we will be billed according to our usage.
There are many advantages for S3 over EBS since we don't need to associate the volume with any instance for access. S3 can be accessed through many services and APIs.

For creating S3,

Select the S3 option under AWS console. And create Buckets.

Now setup Access/Secret key pair for accessing the S3 buckets through APIs. For that,

Select the IAM (Identity and Access Management) option. Here we can create Users and add permissions to each Buckets.

 Note: The main AWS account Access/Secret key pair can be found and created from the security_credential option in the AWS console.
     
At IAM section we can create Users and access/secret keys and add permissions under user policies. This user policies allows us to restrict that user privileges to certain services only.

Logical Volume Management in linux

LVM is a userspace toolset that provide logical volume management on linux. With this tool we can create logical volumes, expand, delete, take snapshots of existing logical volumes etc.

For LVM to work properly, your kernel should support device-mapper libraries.

To install the latest LVM software, download the package from the link below,

wget https://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_105.tar.gz

tar -zxvf lvm2-2_02_105.tar.gz
cd lvm2-2_02_105

./configure
make && make install

If you are on ubuntu, you can simply install it by issuing the below command.

apt-get install lvm2

First we can create physical volumes that will be used to create the lvm

pvcreate /dev/sda8 /dev/sda9

Try Amazon EC2 for Free

Amazon Webservices is one of the top cloud computing service providers around the globe. They have introduced a Free Tier plan which helps to experience the Amazon cloud services at free of cost for one year. Here I will provide a detailed info about the Free Trier plan and pricing after the Free Tier duration.

Through Free Tier Plan we will be getting an AWS EC2 Micro instance named t1.micro !


[t1.micro] specification under Free Tier is listed below.

Duration - 365 Days

Linux Operating System - Ubuntu and Debian ( 64/32 bit)  AMI images.

Time -750 hours/month

Hardware- 613 MB with 32-bit and 64-bit platform support

15 GB data processing / month through Elastic Load Balancer.

30GB of EBS Storage, 2 million I/Os, and 1GB of snapshot storage

15 GB of bandwidth out.

1 GB of Regional Data Transfer.

Free One Elastic IP address associated with a running instance

Wednesday, February 5, 2014

ModSecurity Installation on Linux

ModSecurity is an open source web application layer firewall which primarily
supported Apache and now extends its security features to IIS and Nginx. Modsecurity
acts as a filter for every request supplied to the webserver which accepts/rejects
the request according to the rule sets.

Here I will explain how to install Modsecurity for Apache and configure OWASP core
rule sets on it.

Apache webserver installation can be followed from the link below,

Apache-Installation

Now download the latest Apache Modsecurity package,

wget https://www.modsecurity.org/tarball/2.7.7/modsecurity-apache_2.7.7.tar.gz

Install the missing dependencies if any,

apt-get install libxml2 libxml2-dev libxml2-utils pcre curl

tar -zxvf modsecurity-apache_2.7.7.tar.gz
cd modsecurity-apache_2.7.7

Thursday, January 16, 2014

Hadoop MapReduce Program Samples

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.

What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.

Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Hadoop single and multi node installations can be followed from the below links respectively,

Single Node Setup

Multi Node Setup

Common commands..

List files -> hadoop fs -ls

Open file -> hadoop fs -cat filename

Copy file -> hadoop fs -copyFromLocal input.txt

Copy file -> hadoop fs -copyFromLocal input.txt /newfolder

Test your installation,

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

This will return the value of pi.

Hadoop Multi Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop on multi node clusters.

h1.hadoop-test.com (master/slave)
h2.hadoop-test.com (slave)

Follow the below steps in all nodes,

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

Hadoop Single Node Setup

Hadoop is an opensource implementation of MapReduce framework which is used for processing BigData by using parallel computation on distributed file system across cluster of commodity systems.
What MapReduce does is to split large data sets and send each process to seperate nodes in cluster which uses local computation and storage and combines the result to an output.
Since it uses local computation and storage (hdfs) of each node it is more reliable and fast.

Here I will explain how to install hadoop in a single node.

Hadoop framework is built in java.  So we need to install java (my case ) in /usr/lib/java/

My java version,

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.1.5-20-146757-1.6.0_29-20111004-1747-linux-ia32, compiled mode)

Add user and group for Hadoop,

addgroup hadoop
adduser --ingroup hadoop hduser

For hadoop to work, the user should need passwordless login, For that ..