Installing and Running Hadoop and Spark on Ubuntu 18

Andrew (he/him) - Dec 13 '19 - - Dev Community

Installing and Running Hadoop and Spark on Ubuntu 18

This is a short guide (updated from my previous guides) on how to install Hadoop and Spark on Ubuntu Linux. Roughly this same procedure should work on most Debian-based Linux distros, at least, though I've only tested it on Ubuntu. No prior knowledge of Hadoop, Spark, or Java is assumed.

I'll be setting all of this up on a virtual machine (VM) using Oracle's VirtualBox, so I first need to get an ISO file to install the Ubuntu operating system (OS). I'll download the most recent Long-Term Support (LTS) version of Ubuntu from their website (as of this writing, that's 18.04.3). Setting up a virtual machine is fairly straightforward and since it's not directly relevant, I won't be replicating those linked instructions here. Instead, let's just start with a clean Ubuntu installation...

Installing Java

Hadoop requires Java to be installed, and my minimal-installation Ubuntu doesn't have Java by default. You can check this with the command:

$ java -version

Command 'java' not found, but can be installed with:

sudo apt install default-jre
sudo apt install openjdk-11-jre-headless
sudo apt install openjdk-8-jre-headless
Enter fullscreen mode Exit fullscreen mode

Note: we're going to ignore these suggestions and install Java in a different way.

Hadoop runs smoothly with Java 8, but may encounter bugs with newer versions of Java. So I'd like to install Java 8 specifically. To manage multiple Java versions, I install SDKMAN! (but first I need to install curl):

$ sudo apt install curl -y
Enter fullscreen mode Exit fullscreen mode

...enter your password, and then install SDKMAN! with

$ curl -s "https://get.sdkman.io" | bash
Enter fullscreen mode Exit fullscreen mode

SDKMAN! is a great piece of software that allows you to install multiple versions of all sorts of different packages, languages, and more. You can see a huge list of available software with:

$ sdk ls # or sdk list
Enter fullscreen mode Exit fullscreen mode

sshot_04.PNG

To make sure you can use SDKMAN! in every new terminal, run the following command to append a line which sources the SDKMAN! initialisation script whenever a new terminal is opened:

$ echo "source ~/.sdkman/bin/sdkman-init.sh" >> ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

We're only going to use SDKMAN! to install one thing -- Java. You can list all available versions of a particular installation candidate with:

$ sdk list <software>
Enter fullscreen mode Exit fullscreen mode

So in our case, that's

$ sdk list java
Enter fullscreen mode Exit fullscreen mode

...we can see all of the different available Java versions:

sshot_05.PNG

To install a specific version, we use the Identifier in the column all the way on the right with:

$ sdk install <software> <Identifier>
Enter fullscreen mode Exit fullscreen mode

I'm going to install AdoptOpenJDK's Java 8.0.232 (HotSpot), so this command, for me, is:

$ sdk install java 8.0.232.hs-adpt
Enter fullscreen mode Exit fullscreen mode

SDKMAN! candidates are installed, by default, at ~/.sdkman/candidates:

$ ls ~/.sdkman/candidates/java
8.0.232.hs-adpt  current
Enter fullscreen mode Exit fullscreen mode

The current symlink always points to whichever Java version SDKMAN! thinks is the version you're currently using, and this is reflected in the java -version command. After the last step, this command returns:

$ java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
Enter fullscreen mode Exit fullscreen mode

If you install multiple Java versions, you can easily switch between them with sdk use:

$ sdk install java 13.0.1.hs-adpt

...

$ sdk use java 13.0.1.hs-adpt

Using java version 13.0.1.hs-adpt in this shell.
Enter fullscreen mode Exit fullscreen mode 
$ java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (AdoptOpenJDK)(build 13.0.1+9)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 13.0.1+9, mixed mode, sharing)
Enter fullscreen mode Exit fullscreen mode

We also need to explicitly define the JAVA_HOME environment variable by adding it to the ~/.bashrc file:

$ echo "export JAVA_HOME=\$(readlink -f \$(which java) | sed 's:bin/java::')" >> ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

echo-ing JAVA_HOME should now give us the path to the SDKMAN! directory:

$ echo $JAVA_HOME
/home/andrew/.sdkman/candidates/java/13.0.1.hs-adpt
Enter fullscreen mode Exit fullscreen mode

Make sure you switch back to Java 8 before continuing with this tutorial:

$ sdk use java 8.0.232.hs-adpt

Using java version 8.0.232.hs-adpt in this shell.
Enter fullscreen mode Exit fullscreen mode

Installing Hadoop

With Java installed, the next step is to install Hadoop. You can get the most recent version of Hadoop from Apache's website. As of this writing, that version is Hadoop 3.2.1 (released 22 Sep 2019). If you click on the link on that webpage, it may redirect you. Click until a *.tar.gz file is downloaded. The link I ended up using was

http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
Enter fullscreen mode Exit fullscreen mode

You can download that in the browser, or by using wget in the terminal:

$ wget http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
Enter fullscreen mode Exit fullscreen mode

Unpack the archive with tar, and redirect the output to the /opt/ directory:

$ sudo tar -xvf hadoop-3.2.1.tar.gz -C /opt/
Enter fullscreen mode Exit fullscreen mode

Remove the archive file and move to the /opt/ directory:

$ rm hadoop-3.2.1.tar.gz && cd /opt
Enter fullscreen mode Exit fullscreen mode

Rename the Hadoop directory and change its permissions so that its owned by you (my username is andrew) and not root or 1001:

$ sudo mv hadoop-3.2.1 hadoop && sudo chown andrew:andrew -R hadoop
Enter fullscreen mode Exit fullscreen mode

Finally, define the HADOOP_HOME environment variable and add the correct Hadoop binaries to your PATH by echoing the following lines and concatenating them to your ~/.bashrc file:

$ echo "export HADOOP_HOME=/opt/hadoop" >> ~/.bashrc
$ echo "export PATH=\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin" >> ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Now, when you source your ~/.bashrc (or open any new shell), you should be able to check that Hadoop has been installed correctly:

$ hadoop version
Hadoop 3.2.1
Source code repository...
Compiled by ...
...
Enter fullscreen mode Exit fullscreen mode

In order for HDFS to run correctly later, we also need to define JAVA_HOME in the file /opt/hadoop/etc/hadoop/hadoop-env.sh. Find the line in that file which begins with:

# export JAVA_HOME=
Enter fullscreen mode Exit fullscreen mode

and edit it to match the JAVA_HOME variable we defined earlier:

export JAVA_HOME=/home/<username>/.sdkman/candidates/java/8.0.232.hs-adpt
Enter fullscreen mode Exit fullscreen mode

Make sure you change the <username> above to the appropriate username for your setup. In my case, I replace <username> with andrew.

Installing Spark

The last bit of software we want to install is Apache Spark. We'll install this in a similar manner to how we installed Hadoop, above. First, get the most recent *.tgz file from Spark's website. I downloaded the Spark 3.0.0-preview (6 Nov 2019) pre-built for Apache Hadoop 3.2 and later with the command:

$ wget http://mirrors.whoishostingthis.com/apache/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop3.2.tgz
Enter fullscreen mode Exit fullscreen mode

As with Hadoop, unpack the archive with tar, and redirect the output to the /opt/ directory:

$ sudo tar -xvf spark-3.0.0-preview-bin-hadoop3.2.tgz -C /opt/
Enter fullscreen mode Exit fullscreen mode

Remove the archive file and move to the /opt/ directory:

$ rm spark-3.0.0-preview-bin-hadoop3.2.tgz && cd /opt
Enter fullscreen mode Exit fullscreen mode

Rename the Spark directory and change its permissions so that its owned by you (my username is andrew) and not root or 1001:

$ sudo mv spark-3.0.0-preview-bin-hadoop3.2 spark && sudo chown andrew:andrew -R spark
Enter fullscreen mode Exit fullscreen mode

Finally, define the SPARK_HOME environment variable and add the correct Spark binaries to your PATH by echoing the following lines and concatenating them to your ~/.bashrc file:

$ echo "export SPARK_HOME=/opt/spark" >> ~/.bashrc
$ echo "export PATH=\$PATH:\$SPARK_HOME/bin" >> ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Now, when you source your ~/.bashrc (or open any new shell), you should be able to check that Spark has been installed correctly:

$ spark-shell --version
...
...version 3.0.0-preview
...
Enter fullscreen mode Exit fullscreen mode

Configuring HDFS

At this point, Hadoop and Spark are installed and running correctly, but we haven't yet set up the Hadoop Distributed File System (HDFS). As its name suggests, HDFS is usually distributed across many machines. If you want to build a Hadoop Cluster, I've previously written instructions for doing that across a small cluster of Raspberry Pis. But for simplicity's sake, we'll just set up a standalone, local installation here.

To configure HDFS, we need to edit several files located at /opt/hadoop/etc/hadoop/. The first such file is core-site.xml. Edit that file so it has the following XML structure:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

The second file is hdfs-site.xml, which gives the locations of the the namenode and datanode directories. Edit that file so it looks like:

<configuration>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/datanode</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

We set dfs.replication to 1 because this is a one-machine cluster -- we can't replicate files any more than once here.

Read more about data replication in HDFS here.

The directories given above (/opt/hadoop_tmp/hdfs/datanode and /opt/hadoop_tmp/hdfs/namenode) must exist and be read/write-able by the current user. So create them now, and adjust their permissions, with:

$ sudo mkdir -p /opt/hadoop_tmp/hdfs/datanode
$ sudo mkdir -p /opt/hadoop_tmp/hdfs/namenode
$ sudo chown andrew:andrew -R /opt/hadoop_tmp
Enter fullscreen mode Exit fullscreen mode

The next configuration file is mapred-site.xml, which you should edit to look like:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

...and finally yarn-site.xml, which you should edit to look like:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

Configuring SSH

If you started with a minimal Ubuntu installation like I did, you may need to first set up your ssh connection (as HDFS connects to localhost:9000). To check if the SSH server is running, enter the command

$ which sshd
Enter fullscreen mode Exit fullscreen mode

If nothing is returned, then the SSH server is not installed (this is the case with the minimal Ubuntu installation). To get this up and running, install openssh-server, which will start the SSH service automatically:

$ sudo apt install openssh-server
Enter fullscreen mode Exit fullscreen mode 
$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
  Loaded: loaded ...
  Actve: active...
  ...
Enter fullscreen mode Exit fullscreen mode

To check that this worked, try ssh-ing into localhost:

$ ssh localhost
...
Are you sure you want to continue connecting (yes/no)? yes
...
Welcome to Ubuntu 18.04.3 LTS...
...
Enter fullscreen mode Exit fullscreen mode

You can exit to escape this superfluous self-connection.

Then, create a public-private keypair (if you haven't already):

$ ssh-keygen
Generating public/private rsa key pair.
...
Enter fullscreen mode Exit fullscreen mode

Hit 'enter' / 'return' over and over to create a key in the default location with no passphrase. When you're back to the normal shell prompt, append the public key to your ~/.ssh/authorized_keys file:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

You should now be able to boot HDFS. Continue to the next section.

Formatting and Booting HDFS

At this point, we can format the distributed filesystem. BE CAREFUL and do not run the following command unless you are sure there is no important data currently stored in the HDFS because IT WILL BE LOST. But if you're setting up HDFS for the first time on this computer, you've got nothing to worry about:

Format the HDFS with

$ hdfs namenode -format -force
Enter fullscreen mode Exit fullscreen mode

You should get a bunch of output and then a SHUTDOWN_MSG:

sshot_06.PNG

We can then boot the HDFS with the following two commands:

$ start-dfs.sh && start-yarn.sh
Enter fullscreen mode Exit fullscreen mode

Note: if you performed a minimal installation, you may need to install openssh-server by following the instructions given here.

You can check that HDFS is running correctly with the command jps:

$ jps
10384 DataNode
11009 NodeManager
4113 ResourceManager
11143 Jps
10218 NameNode
10620 SecondaryNameNode
Enter fullscreen mode Exit fullscreen mode

You should see a NameNode and a DataNode, at minimum, in that list. Check that HDFS is behaving correctly by trying to create a directory, then listing the contents of the HDFS:

$ hdfs dfs -mkdir /test
$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - andrew supergroup          0 2019-12-13 13:56 /test
Enter fullscreen mode Exit fullscreen mode

If you can see your directory, you've correctly configured the HDFS!

Monitoring

Hadoop and Spark come with built-in web-based monitors that you can access by going to http://localhost:8088:

...and http://localhost:9870 in your browser:

Working with Spark and HDFS

One of the benefits of working with Spark and Hadoop is that they're both Apache products, so they work very nicely with each other. It's easy to read a file from HDFS into Spark to analyse it. To test this, let's copy a small file to HDFS and analyse it with Spark.

Spark comes with some example resource files. With the above configuration, they can be found at /opt/spark/examples/src/main/resources. Let's copy the file users.parquet to HDFS:

$ hdfs dfs -put /opt/spark/examples/src/main/resources/users.parquet /users.parquet
Enter fullscreen mode Exit fullscreen mode

Parquet files are another Apache creation, designed for fast data access and analysis.

Next, open the Spark shell and read in the file with read.parquet:

$ spark-shell
...
Welcome to
... version-3.0.0-preview
...
Enter fullscreen mode Exit fullscreen mode 
scala> val df = spark.read.parquet("hdfs://localhost:9000/users.parquet")
df: org.apache.spark.sql.DataFrame = [name: string, favorite_color: string ... 1 more field]

scala> df.collect.foreach(println)
[Alyssa,null,WrappedArray(3,9,15,20)]
[Ben,red,WrappedArray()]
Enter fullscreen mode Exit fullscreen mode

This is just a small example, but it shows how Spark and HDFS can work closely together. You can easily read files from HDFS and analyse them with Spark!

If you want to stop the HDFS, you can run the commands:

$ stop-dfs.sh
Enter fullscreen mode Exit fullscreen mode

and

$ stop-yarn.sh
Enter fullscreen mode Exit fullscreen mode

Conclusion

I hope this guide will be useful for anyone trying to set up a small Hadoop / Spark installation for testing or education. If you're interested in learning more about Hadoop and Spark, please check out my other articles in this series on Dev! Thanks for reading!

Image
Everything You Need To Be Aware Of Car Remote Key Repair
Image
How do I refund a wrong transaction on PhonePe

webdev, javascript
Image
National Skills Development Authority (NSDA)

nsda
Image
5 Qualities People Are Looking For In Every Bifold Door Repairs Near Me
Image
How do I get my money back from Google Pay wrong?
Image
1st Post

writing, truth
Image
Parkinson's Law in Software Development: How to Break the Time-Expansion Trap 🚀
Image
Add Jetbrains Fleet to right click context menu in windows 11

fleet
Image
How do I get my money back from Google Pay wrong?

webdev, javascript, beginners
Image
20 Tools That Will Make You Better At Asbestos Lawsuit History
Image
How Do You Explain ADHD Online Assessment To A 5-Year-Old
Image
Responsible For The Cheap ADHD Assessment Budget? 10 Fascinating Ways To Spend Your Money
Image
Who's The World's Top Expert On Van Hook Locks?
Image
Setting Up a Node.js, TypeScript, and Express Project

node, express, typescript, project
Image
How can I get a refund from Paytm for a wrong transaction?

programming, devops
Image
Innovative Programs at the University of Wisconsin-Madison: Shaping Tomorrow’s Leaders
Image
How To Become A Prosperous ADHD Assessments Even If You're Not Business-Savvy
Image
A Productive Rant About Lost Car Key Service
Image
Mastering SQL Optimization: A Beginner’s Guide to Faster and More Efficient Queries

sqlserver, sql, database, datascience
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player