What you should expect:
- Pull down and quickly modify the source.
- Package the application into a jar file.
- Submit the application using spark-submit to your locally running cluster (or any cluster where the sample file exists on all nodes).
- View the expected results in your terminal.
The ready to consume application can be found at:
See the README.md file for direction on how to modify the application to run on your environment.
You will need to have Java, Scala, and SBT installed locally.
(From the README.md file)
Move the file tenzingyatso.txt to a known location on your file system (E.g. /tmp/tenzingyatso.txt)
Modify SuperSimple.scala so the path to tenzingyatso.txt is correct for your system.
val compassionFile = "/home/bkarels/tenzingyatso.txt"
val compassionFile = "/tmp/tenzingyatso.txt"
From the root of this project run package from within SBT:
*** Take note of where the application jar is written ***
[info] Packaging /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar ...
[info] Done packaging.
Since this has been designed to run against a local cluster, navigate to your $SPARK_HOME and use spark-submit to send the application to your cluster:
[bkarels@ahimsa spark_1.1.0]$ ./bin/spark-submit --class com.bradkarels.spark.simple.SuperSimple --master spark://127.0.0.1:7077 /home/bkarels/dev/super-simple-spark-app/target/scala-2.10/super-simple-spark-app_2.10-0.1.jar
Talks of peace: 3
Speaks of love: 2