Monday, January 26, 2015

Spark + Kafka + Cassandra (Part 2 - Spark Kafka [mass] Producer)

Overview

Welcome to the part two of the series 'Spark + Kafka + Cassandra'.

Building on top of part one and preparing for part three, here we'll spin up a little application that does only one thing well - spews out random messages for a number of seconds.  Basically, we wanted to have a tool that could be used to pump a large quantity of messages into Kafka to:
  • Test a Kafka setup
  • Tinker with throughput
  • Feed a consumer
  • etc.
Also, this example can easily be altered to spew whatever one might like (CSV, JSON, data from a flat file or HDFS, etc.).  Make it yours - spew your own stuff.

This second instalment will have you:
  • Run ZooKeeper local (as part of Kafka distribution)
  • Run Kafka local
  • Run a local Spark Cluster
  • Create a Kafka Producer wrapped up in a Spark application
  • Submit the application to generate messages sent to a given topic for some number of seconds.

Prerequisites:

  • Java 1.7+ (Oracle JDK required)
  • Scala 2.10.4
  • SBT 0.13.x
  • A Spark cluster (how to do it here.)
  • ZooKeeper and Kafka running local 
  • git


Set it up...

If you have not walked through Part 1 of this series, go there and walk through it to get ZooKeeper and Kafka up and running on your local machine.  When you've done that, come back here and proceed to the next step.



Clone the Example

Begin by cloning the example project from github - spark-kafka-msg-generator, & cd into the project directory.

[bkarels@rev27 work]$ git clone https://github.com/bradkarels/spark-kafka-msg-generator.git
Cloning into 'spark-kafka-msg-generator'...
remote: Counting objects: 19, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 19 (delta 0), reused 15 (delta 0)
Unpacking objects: 100% (19/19), done.
[bkarels@rev27 work]$ cd spark-kafka-msg-generator/
[bkarels@rev27 spark-kafka-msg-generator]$


Prepare the Application

In a terminal, from the project root, fire up SBT and assembly the project to create your application jar file.
[bkarels@rev27 spark-kafka-msg-generator]$ sbt
Picked up _JAVA_OPTIONS: -Xms1G -Xmx2G -XX:PermSize=512m -XX:MaxPermSize=1G
[info] Loading project definition from /home/bkarels/work/spark-kafka-msg-generator/project
[info] Updating {file:/home/bkarels/work/spark-kafka-msg-generator/project/}spark-kafka-msg-generator-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to MsgSpewer (in build file:/home/bkarels/work/spark-kafka-msg-generator/)
> assembly
...
[info] SHA-1: cddec14059d6a435847f8dd4b4b6f15f6899c0c3
[info] Packaging /home/bkarels/work/spark-kafka-msg-generator/target/scala-2.10/sparkFuProducer.jar ...
[info] Done packaging.
[success] Total time: 53 s, completed Jan 26, 2015 9:10:16 AM

As per normal, take note of where your jar file gets put (highlighted above).  Also, note that we have set the resulting jar file name in assembly.sbt - here we have set it to sparkFuProducer.jar.

Spark it up!

If your local Spark cluster is not up and running, do that now.  Go here to see about getting 'r' done.

Make Sparks fly! (i.e. run it)

Having reviewed the code you have seen that this application accepts one, two, or three argument.  By default, the application will publish to topic sparkfu for 10 seconds without printing more than the normal output to the terminal.  You can optionally set the following three arguments:

[[topic] [duration]] [verbose]]]
Argument Examples:
Publish to topic sparkfu for 20 seconds and do print additional output:
sparkfu 20 true

Publish to topic filthpig for 3600 seconds and do not print additional output:
filthpig 3600

Publish to topic schweinehund for 10 seconds and do not print additional output:
schweinehund


Like Part 1, when this application is submitted to the cluster it will have no clear output.  Here again we can use the local consumer.  Do as below or look back to Part 1 to see about this.
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sparkfu --from-beginning


OK, spark it up:
$SPARK_HOME/bin/spark-submit --class com.bradkarels.simple.RandomMessages --master local[*] ~/dev/spark-kafka-msg-generator/target/scala-2.10/sparkFuProducer.jar sparkfu 20 true
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Picked up _JAVA_OPTIONS: -Xms1G -Xmx2G -XX:PermSize=512m -XX:MaxPermSize=1G
This will run for: 20s
...
15/01/26 09:24:42 INFO SyncProducer: Connected to rev27:9092 for producing
5000 * 18 seconds left...
10000 * 17 seconds left...
...
115000 * 3 seconds left...
120000 * 2 seconds left...
130000 * 1 seconds left...
Produced 134137 msgs in 20s -> 6706.0m/s.
If you were running your local consumer you should see a big ol' pile of alphanumeric randomness stream by.  Just like that you can spew a pretty good chunk of messages to a topic on command. 


What's next?

On to part three where we'll consume messages out of Kafka with Spark Streaming and write them to Cassandra.  See part three here!

25 comments:

  1. I am getting this error while running assembly. Any idea

    [warn] Merging 'com/esotericsoftware/minlog/Log.class' with strategy 'first'
    [trace] Stack trace suppressed: run last *:assembly for the full output.
    [error] (*:assembly) deduplicate: different file contents found in the following:
    [error] /home/xyz/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.3.1.jar:com/google/common/base/Absent.class
    [error] /home/xyz/.ivy2/cache/com.google.guava/guava/bundles/guava-14.0.1.jar:com/google/common/base/Absent.class
    [error] Total time: 46 s, completed May 9, 2016 4:32:42 PM

    ReplyDelete
  2. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.


    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

    ReplyDelete


  3. شركة غسيل خزانات بالمدينة المنورة شركة غسيل خزانات بالمدينة المنورة
    افضل شركة تنظيف منازل بالمدينة المنورة شركة تنظيف منازل بالمدينة المنورة

    ReplyDelete
  4. There are lots of Natural Remedies for Achalasia in market but these are very expensive. A product made by Natural Herbs Clinic is one of the useful and low prices Natural Remedy for Achalasia which works without any risk. It is a low price product and made with herbal ingredients that work without any side effects.

    ReplyDelete
  5. Natural Remedies easy to informed assessment of the utility of natural treatment in the administration of Benign Essential Tremor given that the greater part that have been distributed in Natural Remedies for Benign Essential Tremor and it is beneficial for Tremor disease. Therefore, the current examination intends to evaluate the viability of home grown medication medicines on essential tremor.

    ReplyDelete
  6. We are a Trailers rental companies in Abu Dhabi. Get comfortable trucks, trailers for cheap rental in Abu Dhabi. We offer private and business clients a scope of top-quality trailers for short or long haul employ. We are Providing Best Trailer Rentals in Abu Dhabi.

    ReplyDelete
  7. Are you looking for a Trailers rental in Dubai, UAE? We provide the best trailers to transport heavy construction materials anywhere in the UAE. Get Heavy Trailer Truck for Rent in UAE, Trailer Rental Dubai, Trailer for Rent in Sharjah, Trailer Truck for Rent.

    ReplyDelete
  8. Apartment Painting Dubai is the best painting company in Dubai. We offer you flawless wall painting services in Dubai. We give a perfect finish to the interior and exterior. Our PROFESSIONAL Painters in Dubai offers home and villa painting services for 10+ years. Affordable Wall Painting Services by Expert Painters in Dubai for house painting solution in Dubai, villa painting Dubai,
    apartment painting Dubai.

    ReplyDelete
  9. Trailer for rent in Sharjah . We are providing Transportation services with (40ft) Flatbed Trailers and Trucks throughout the UAE, 24/7 hours. We are a Trailers rental companies in Sharjah. Get comfortable trucks and trailers for cheap rental service.

    ReplyDelete
  10. Noor Ahmed Transports LLC Dubai offer Flatbed Trailer for Rent in Dubai, Trailer Truck Rental Sharjah, Abu Dhabi, Trailer Truck for Rent in UAE . Get Heavy Trailer Truck for Rent in UAE, Trailer Rental Dubai, Trailer for Rent in Sharjah, Trailer Truck for Ren. Trailer Rental Abu Dhabi, Heavy Trailer Truck for Rent in UAE, Trucks & Trailer for Rent in Sharjah, Trailer Truck for Rent

    ReplyDelete
  11. Looking to hire a Sober driver in Dubai? Sober Driver Dubai Most Trusted Driver In Dubai UAE. Sober Driver app is designed to provide you with comforting and fast experience of booking a driver in Dubai and UAE. Professional safe driver Dubai drives you home safely, book reliable & designated safer driver. Get driver in 20 minutes starting from 80 AED only. Sober Driver Dubai offers world-class chauffeur services in Dubai at cheapest prices.

    ReplyDelete
  12. We are the professional furniture packers furniture packers and movers in Dubai. Providing Cheapest Movers Service, Reliable and Hassle-free Service. We are expert and professional in packing and moving your office furniture in Dubai. Our office move action plans are carefully drawn up to minimize the loss or damages of your office furniture.

    ReplyDelete
  13. Villa Movers and Packers Dubai Our team can help you with packing, unpacking, loading, and unloading of your belongings. We are Best Movers and Packers in Dubai, Moving Companies, Villa Movers in Dubai, Local Movers in Dubai. Best Dubai movers and packers for office, house, villa shifting and storage in Dubai. As a leading packers and movers in Dubai, we are a 'one-stop-shop' for all your office relocation needs.

    ReplyDelete
  14. Get driver in 20 minutes starting from 70 AED only.7 Days A Week Support. The safe driver Dubai service promotes the Corporate Social Responsibility by offering designated drivers to you at your door step. We promote safety and safe driving. Drink alcohol and drive is strictly prohibited as per UAE Law. Book your Personal Safe Driver that can take you home safely. The Sober Driver Dubai offers world-class chauffeur services in Dubai at cost-effective prices. Call us at 055 730 3487 to make a booking!

    ReplyDelete
  15. Here Follows the List of the Top 10 Logistics Companies in Dubai · 1. Noor Ahmed Transports LLC, 2. Trailer Rental Dubai, 3. Rent Pickup Dubai, 4. Mumtaz Hussain Technical Services LLC, 5. Mac World Logistics LLC, 6. Verks Global Logistics LLC, 7. Consolidated Shipping Services (CSS Group), 8. Emirates Logistics LLC, 9. National Association of Freight and Logistics (NAFL). 10. CEVA Logistics

    ReplyDelete
  16. Buy online Mango , Chaunsa (چونسہ آم) from FruitOye.pk at the best price. We deliver export quality Mangoes at your door step. Chaunsa mango is the second most sought after mango variety after Sindhri in the Sindh region. It is famous for its blended sweet and sour taste. Buy Premium Export Quality Chaunsa Mango Online in Pakistan at Affordable Prices. It is said that Chaunsa was made popular by Sher Shah Suri, while celebrating his victory over Humayun at Chausa, he named his favourite mango Chaunsa.

    ReplyDelete
  17. Buy online Mango, Chaunsa (چونسہ آم) from FruitOye.pk at the best price. We deliver export quality Mangoes at your door step. Chaunsa mango is the second most sought after mango variety after Sindhri in the Sindh region. It is famous for its blended sweet and sour taste. Buy Premium Export Quality Chaunsa Mango Online in Pakistan at Affordable Prices. It is said that Chaunsa was made popular by Sher Shah Suri, while celebrating his victory over Humayun at Chausa, he named his favourite mango Chaunsa.

    ReplyDelete
  18. Buy online Mango, Chaunsa (چونسہ آم) from FruitOye.pk at the best price. We deliver export quality Mangoes at your door step. Chaunsa mango is the second most sought after mango variety after Sindhri in the Sindh region. It is famous for its blended sweet and sour taste. Buy Premium Export Quality Chaunsa Mango Online in Pakistan at Affordable Prices. It is said that Chaunsa was made popular by Sher Shah Suri, while celebrating his victory over Humayun at Chausa, he named his favourite mango Chaunsa.

    ReplyDelete
  19. Buy online Mango, Chaunsa (چونسہ آم) from FruitOye.pk at the best price. We deliver export quality Mangoes in Lahore at your door step. Chaunsa mango is the second most sought after mango variety after Sindhri in the Sindh region. It is famous for its blended sweet and sour taste. Buy Premium Export Quality Chaunsa Mango Online in Pakistan at Affordable Prices. It is said that Chaunsa was made popular by Sher Shah Suri, while celebrating his victory over Humayun at Chausa, he named his favourite mango Chaunsa.

    ReplyDelete
  20. Designated Driver Dubai is an Sober Driver Dubai Most Trusted Driver In UAE. Sober Driver Dubai from ExpertDrivers.ae Welcome to Awesome Drive, an emerging chauffeur service in all over dubai.
    Book a safe Designated driver in Dubai to get back home safely. A reliable, designated, safe driver service starts @ 70 AED only for 24x7 at any time day or night.

    ReplyDelete
  21. Deliver best assortment of Farm Fresh Fruits & Dry Fruits products in online at wholesale prices from fruitoye.pk . We provide fast Open Box & Taste It Now Delivery at your door step. Best produce delivery services · Best produce box for saving money.

    ReplyDelete