7

Distributed data processing with Hadoop - Part-3: App Build

view full story
linux-howto

http://www.linux.com – With configuration, installation, and the use of Hadoop in single-node and also the use in multi-node architectures under your belt, you can now turn to the task of developing applications within the Hadoop infrastructure. This article explores the Hadoop APIs and data flow and demonstrates their use with a simple mapper and reducer application. (General)

dayu's picture
Submitted by dayu on 07/17/2010 – Made popular on 07/17/2010

Stories similar to Distributed data processing with Hadoop - Part-3: App Build

IBM Developerworks: "This final article in the series explores the Hadoop APIs and data flow and demonstrates their use with a simple mapper and reducer application."

Hey guys Montana here,

I have configured Hadoop for single node cluster, when I run my URLS, the .jsp's they work which means Hadoop is working but upon booting Hadoop by formatting a new distributed filesystem

Code: bin/hadoop namenode -format I get an SSH error, but then when I actually start Hadoop

Code: bin/start-dfs.sh Which should also handle the "${HADOOP_CONF_DIR}&q

Learn advanced setup that uses multiple nodes for parallel processing. It demonstrates the various node types required for multinode clusters and explores MapReduce functionality in a parallel environment. This article also digs into the management aspects of Hadoop—both command line and Web based.

IBM Developerworks: "This article�the first in a series on Hadoop�explores the Hadoop framework, including its fundamental elements, such as the Hadoop file system (HDFS), and node types that are commonly used."

I am required to hack a single node hadoop "cluster" (cloudera psuedo-distributed) to be able to access it remotely. I have successfully installed hadoop and I have updated the localhost identifiers in the configs to the IP address of the machine. I can run hadoop fs -ls / and all is good. I have created a passphraseless key and I can ssh to the hadoop machine.

I have only one Data Node on my dev environment on EC2. I ran heavy MR job and in 6 hours noticed that 100% of mappers and 20% of reducers finished (1 of reducer shows 100% competition, other ones - 0%). Looks like job is hung between 2 reducer runs. I don't see any errors in log files.

What it can be?

P.S.

cp not working in script 41 weeks 6 days ago

Ok, this is the first script I've ever attempted to write so go easy on me. I'm just trying to simple copy a tar.gz file from the directory the script is located to /usr/local/filename.tar.gz The script is run as root so it shouldn't be a permissions issue.

I want to set up a small (6 node) hadoop/hive/pig cluster.

I see the work in the juju space on charms; however, the current status of deploying a single charm per node will not work for me.

I see ServerTeam Hadoop which talks about re-packaging the bigtop packages.

The cloudera CDH3 installation guide talks about Maverick and Lucid, but not precise.

What am I missing?

I am following this tutorial.

http://hadoop.apache.org/docs/mapreduce/current/mapred_tutorial.html

javac -classpath ${HADOOP_HOME}/hadoop-core- ${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-mapred-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-hdfs-${HADOOP_VERSION}.jar -d wordcount_classes

The hadoop version is 0.22.0 and this does not have a hadoop-core-0.22.0.jar though I find hadoop-hdfs-0.22.