Today I have installed Hadoop Core on my white macbook (currently I have only one though :)).
I remember my times that I loved to use map and reduce in languages following functional paradigm (like Haskell or Lisp) during my undergraduate years. I know that Google’s Map/Reduce is pretty different than that but I felt the same “love” again after years.
Deploying Hadoop Core was pretty easy though, a very straightforward guide is there on Hadoop wiki.
Running Hadoop On OS X 10.5 64-bit (Single-Node Cluster)
Hadoop needs Java SERE 1.6 to run on, so I updated it first and configured from Java Preferences.
A little problem was that I first set JAVA_HOME in hadoop-env.sh to Java SE 1.5 directory, updated Java RE and forgot to change JAVA_HOME to point to the new RE.
Exception in thread “main” java.lang.UnsupportedClassVersionError: Bad version number in .class file
After starting NameNode, DataNode, JobTracker and TaskTracker (1) on the same machine, Hadoop was good to go for experiments.
The package comes with several examples, very little instances/prototypes of the big problems in real life. Counting the occurrence of words in different files was one of that. In the WordCount example:
Each mapper takes a line as input and breaks it into words. It then emits a key/value pair of the word and 1. Each reducer sums the counts for each word and emits a single key/value with the word and sum. As an optimization, the reducer is also used as a combiner on the map outputs.
This was my first experiment and an initial touch on HDFS. The next experiment might be on HBase :)
