Gautam's Blog

I would like to take a look at how to parse libpcap files in Hadoop. A problem is that the files are not easily ‘splittable’.  However, we can parse PCAP files using Java using PcapDumper (sample code in the distribution’s SVN): data needs to be serialized using protocol buffers. Watch for this patch.

I am yet to check out Hivo, but here are the results otherwise. I am loading to an empty table, without much indexes; otherwise, I might need to disable indexing and re-enable indexing after the load data.

From: http://wiki.apache.org/hadoop/HowManyMapsAndReduces

Initial Configuration

  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path src = new Path(srcHDFSFile);
  Path dst = new Path(dstHDFSFile);

Copy a file from the local file system to HDFS

 hdfs.copyFromLocalFile(src, dst);

Create HDFS file

  FSDataOutputStream outputStream = hdfs.create(path);
  outputStream.write(buff, 0, buff.length);

Rename HDFS file

 boolean isRenamed = hdfs.rename(src, dst);

Delete HDFS file

  boolean isDeleted = hdfs.delete(src, false);

Recursive delete:

  boolean isDeleted = hdfs.delete(path, true);

Get HDFS file last modification time

  FileStatus fileStatus = hdfs.getFileStatus(src);
  long modificationTime = fileStatus.getModificationTime

Check if a file exists in HDFS

  boolean isExists = hdfs.exists(path);

Check if a Path is a file

  boolean isExists = hdfs.isFile(path);

Get the locations of a file in the HDFS cluster

  FileStatus fileStatus = hdfs.getFileStatus(src);
  BlockLocation[] blkLoc = hdfs.getFileBlockLocations(path, 0, fileStatus.getLen());
  int blkCount = blkLoc.length;
  for (int i=0; i < blkCount; i++) {
    String[] hosts = blkLoc[i].getHosts();
    // Print the values
  }

Get a list of all the nodes host names in the HDFS cluster

  DistributedFileSystem hdfs = (DistributedFileSystem) fs;
  DatanodeInfo[] dataNodeStats = hdfs.getDataNodeStats();
  String[] names = new String[dataNodeStats.length];
  for (int i = 0; i < dataNodeStats.length; i++) {
      names[i] = dataNodeStats[i].getHostName();
  }

Source: http://myjavanotebook.blogspot.com/2008/05/hadoop-file-system-tutorial.html

File size effects the performance of Hadoop. To evaluate the performance,  I wrote a java program to create specific file sizes and uploaded it into Hadoop for performance.

The sum of all file sizes was 512MB. The hadoop streaming ran cat as mapper and wc as the reducer. The results are attached below:

It can be seen that Hadoop is faster when the file size is larger.

Cut, copied and pasted from: http://www.sun.com/software/sge/ for future personal reference.

Sun Grid Engine software is the world’s leading – and most widely deployed – distributed resource manager. Sun Grid Engine 6.2 Update 5 brings you even more power and the ability to lower the costs even more than ever before.

Why Sun Grid Engine 6.2 Update 5?
(1) It runs Hadoop Apache as a data center application. No dedicated Hadoop cluster needed.
(2) It is able to schedule applications that are sensitive to CPU topologies to maximize their performance, cutting execution times in some cases by over 50%.
(3) Resources can be automatically rented and provisioned from the Amazon EC2 service to keep up with peak demand. It is much better than buying and keeping them idle during low demand.
(4) It does power conservation by turning off idle systems and turning them on, when demand justifies it.

I need a curve-fitting mechanism where a series of (x, y) coordinates have to be put into a order 2 curve in a Mapper module of Map/Reduce code.

f(y) = a x^2+ b *x + c

Ideally, I would like to see that the curve is linear. That is need to verify, a = 0.

I have some links listed here:

• Spline (mathematics)
• An example code: spline code implementation.
• Michael Thomas Flanagan’s Java Scientific Library is fantastic place to start mathematical implementations in Java.

This is a better mechanism to find distance between two IPs. It is faster than using Haversine formula as discussed in my previous post because this is faster than getting latitude and longitude and computing the distance.

import com.maxmind.geoip.Location;
import com.maxmind.geoip.LookupService;
import java.io.IOException;

public class GeoDistanceTwoIPs {
    public static void main(String args[]) throws Exception {
         //args[0] is the geolite.dat  .......... args[1] is ip1 args[2] is ip2
         LookupService lookupService = new LookupService(args[0]);
         Location location1 = lookupService.getLocation(args[1]);
         Location location2 = lookupService.getLocation(args[2]);
         System.out.print("Distance: " + location1.distance(location2) + " kilometers");
     }
}

/**
 * Created by IntelliJ IDEA.
 * User: Gautam
 * Date: Jan 15, 2009
 * Time: 9:35:12 AM
 * To change this template use File | Settings | File Templates.
 */

import java.io.*;

public class threadCommand {
    static String fileName;
    static String fileLines[];
    static int numberLinesToCompute;
    static int numberThreads = 16;

    public static void main(String args[]) {
        for (int i = 0; i < args.length; i++)
            if (args[i].equals("-f"))
                fileName = args[++i];
            else if (args[i].equals("-n"))
                numberLinesToCompute = Integer.parseInt(args[++i]);
        try {
            fileLines = new String[numberLinesToCompute];
            DataInputStream dis = new DataInputStream(new BufferedInputStream(new FileInputStream(new File(fileName))));
            int i = 0;
            while (dis.available() != 0) {
                fileLines[i++] = dis.readLine().trim();
            }
            dis.close();
        } catch (IOException e) {
            System.out.println("java threadCommand -f  -n ");
            System.exit(0);
        }
        for (int i = 0; i < numberThreads; i++) {
            if (i == (numberThreads - 1))
                new threadedExecution("Thread" + i, fileLines, i * (numberLinesToCompute / numberThreads), numberLinesToCompute - 1);
            else
                new threadedExecution("Thread" + i, fileLines, i * (numberLinesToCompute / numberThreads), ((i + 1) * (numberLinesToCompute / numberThreads)) - 1);
        }
    }
}

class threadedExecution extends Thread {
    static String SHELL_SCRIPT_TO_EXECUTE = "./getBGP.sh";
    private static final String ERROR_NODES = "error";
    String threadName;
    String linesForThisThread[];
    int init;
    int finale;

    public threadedExecution(String threadName, String linesForThisThread[], int init, int finale) {
        super(threadName);
        //System.out.println(this);
        this.linesForThisThread = linesForThisThread;
        this.threadName = threadName;
        this.init = init;
        this.finale = finale;
        start();
        File f = new File(ERROR_NODES + threadName);
        if (f.exists())
            f.delete();
    }

    public void run() {
        try {
            for (int i = init; i < finale + 1; i++) {
                runCommand(linesForThisThread[i], threadName);
            }
        } catch (IOException e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        }
        //System.out.println("Exit from Thread " + getName());
    }

    private static void runCommand(String line, String thread) throws IOException {
        //To change body of created methods use File | Settings | File Templates.
        Process process = Runtime.getRuntime().exec(SHELL_SCRIPT_TO_EXECUTE + " " + line);
        System.out.println("Success at:" + line);
        try {
            if (process.waitFor() != 0) {
                FileWriter fw = new FileWriter(ERROR_NODES + thread, true);
                StringBuffer sbuf = new StringBuffer();
                sbuf.append(line + "\n");
                fw.write(sbuf.toString());
                fw.flush();
                fw.close();
            }
        } catch (InterruptedException e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        }
    }
}

Select the project properties in the file tab. Select main class. Build projects: Build->Clean and build main project. Jar file is available.