Tuesday, November 25, 2014

Using spark-submit to send an application to a local Spark cluster

In my last post (Running a local Apache Spark Cluster)
I went over how to spin up a local Spark cluster for development and prototyping.  Now it is time to build the most basic Spark application to submit to your local cluster.  While this example is heavily based on this example, we will tweak a couple of bits to make it just slightly more interesting.

What you should expect:
  1. Pull down and quickly modify the source.
  2. Package the application into a jar file.
  3. Submit the application using spark-submit to your locally running cluster (or any cluster where the sample file exists on all nodes).
  4. View the expected results in your terminal.

The ready to consume application can be found at:

See the README.md file for direction on how to modify the application to run on your environment.

You will need to have Java, Scala, and SBT installed locally.

(From the README.md file)

Step 1:
Move the file tenzingyatso.txt to a known location on your file system (E.g. /tmp/tenzingyatso.txt)

Step 2:
Modify SuperSimple.scala so the path to tenzingyatso.txt is correct for your system.

val compassionFile = "/home/bkarels/tenzingyatso.txt"

val compassionFile = "/tmp/tenzingyatso.txt"

Step 3:
From the root of this project run package from within SBT:

> package
*** Take note of where the application jar is written ***
[info] Packaging /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ...
[info] Done packaging.

> exit

Step 4:
Since this has been designed to run against a local cluster, navigate to your $SPARK_HOME and use spark-submit to send the application to your cluster:

[bkarels@ahimsa spark_1.1.0]$ ./bin/spark-submit --class com.bradkarels.spark.simple.SuperSimple --master spark:// /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar
Talks of peace: 3
Speaks of love: 2



  1. HIRE A SAFE DRIVER @ 80 AED IN DUBAI. Awesome Drive has the best drivers in Dubai and provides the best Safe Driver Services at the rate of AED 80.
    Hire most reliable and professional Safe Driver Dubai that can drive you home safely at reasonable cost. Available at your service in 20 min. Call Professional Drivers for City Tour and ride with
    Cheapest Safe Driver Dubai.Let's move with the comfort of your own car with RAT approved company.

  2. Safe Driver Service Dubai, Hire Safe Driver @ AED 80/day. Hire most reliable and
    professional Safe Driver Dubai that can drive you home safetly at reasonable cost. Available at your service in 20 min. Professional safe driver Dubai drives you home safely, book reliable & designated safer driver. Get driver in 20 minutes starting from 80 AED only.

  3. Sober Driver Dubai Most Trusted Driver In UAE. Sober Driver Dubai . from Awesome Drive. Welcome to Awesome Drive, an emerging chauffeur service.
    Newly launched Sober Driver app is designed to provide you with comforting and fast experience of booking a driver . Book a your Personal Safe Driver that can take you home safely. Customers Comfort Safety and their Vehicle take Caring is our Highest Priority. Just Call Us 24/7

  4. Safer Driver Dubai - Reliable and Safe Drivers .
    We offer you the best Ground Transportation Services all over UAE with UAE Chauffeur. The best Chauffeur Service available in the UAE. Simplify Your Ground Transportation. Sanitized. Book Ride.

  5. Self Storage– Best Storage Solutions providers you with the best facility and service you are looking for and deserve for all of your storage needs in Dubai. Here’s how our Storage in Dubai solution works as – our experienced professional movers Dubai will come to your door step, pack and pick up the items that you want to put in storage space and take them to our warehouse facility.

  6. Avail of reliable storage facilities in Dubai to securely store your belongings. Be it
    furniture storageor office equipment storage, we cover a wide range. Storage Space and Warehouse Facility Dubai. In search of the right managed storage space in Dubai to store and manage your goods and products? Looking for warehouse storage Dubai? MH Technical Services movers have separate warehouses designed for commercial and personal uses, Contact Us today. Get a Free Quote. We offer warehouse facility and cargo services in Dubai & UAE.

  7. Storage in Dubai is now easy with Home Storage Dubai. Affordable & Premium storage facility with exceptional customer services. Self-Storage Dubai, Best storage UAE, Affordable moving company in Dubai, Self-Storage units, Storage space, Storage services, Storage service in Abu Dhabi. Modern Storage Solutions For All Your Personal Needs – Free Up Your Home Space Today. Moving Home? Renovating? Or Just Need More Space? Get A Free Quote from Us Today.

  8. Nice Blog !!

    Are you for in a situation where too exhausted to drive back to home like returning home from a party, long work schedule? Safe driver Dubai is ready to help you…

  9. The most popular betting site in Korea is Private Toto. Private Toto, which occupies the majority of the Toto site market, is attracting the attention of sports Toto users by providing various live betting services. Please use the private Toto site, which has been identified as a safety playground, to make online bets. 토토사이트 웹사이트 바로가기 안전놀이터

  10. Perfect Passport Photos allows you to take regular photos on your smartphone, DSLR, or tablet and transform them into a document-ready format instantly, no matter where you live.