Tuesday, December 16, 2014

Simple Python application on Apache Spark Cluster

Overview


As an exercise, I am working on duplicating my previous examples in Python.  It is clear that Python has, and is gaining, traction in the data world.  So, it makes sense to have a working knowledge of it.

As with my other examples, everything will find it's way to my Github repositories - forking and enhancements welcome.

This effort is about the most simplistic Python submit to Spark Cluster example possible.  But, when you move beyond the REPL, you have to start somewhere right?

Prerequisites:

  • Java 1.7+ (Oracle JDK required)
  • A Spark cluster (how to do it here.)
  • git


Clone the Example

Begin by cloning the example project from github - super-simple-spark-python-app and cd into the project directory.


[bkarels@ahimsa work]$ git clone git@github.com:bradkarels/super-simple-spark-python-app.git
Initialized empty Git repository in /home/bkarels/work/super-simple-spark-python-app/.git/
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 3), reused 6 (delta 1)
Receiving objects: 100% (13/13), done.
Resolving deltas: 100% (3/3), done.
[bkarels@ahimsa work]$ cd super-simple-spark-python-app/


Move the file tenzingyatso.txt to your home directory.
[bkarels@ahimsa super-simple-spark-python-app]$ mv tenzingyatso.txt ~

Modify simple.py path to the sample file (save and close).

file = sc.textFile("/home/bkarels/tenzingyatso.txt")
becomes...
file = sc.textFile("/home/yourUserNameHere/tenzingyatso.txt")
...or some such similar thing.

Spark it up (with python)!

If your local Spark cluster is not up and running, do that now.  If you need to review how to go about that, you can look here.

Make Sparks fly! (i.e. run it)


Since this example does not have a packaged application (e.g. jar, egg, etc.), we can invoke spark-submit with just our simple python file.


[bkarels@ahimsa super-simple-spark-python-app]$ $SPARK_HOME/bin/spark-submit --master spark://127.0.0.1:7077 ./simple.py
Your expected output to the console should be a line count of 7 wrapped in a nice battery of asterisks and the copy from the first line of the example file.  If you see that - this has worked.

62 comments:

  1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training India . Nowadays Java has tons of job opportunities on various vertical industry.

    ReplyDelete
    Replies
    1. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts. Python Projects for Students Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account. Project Center in Chennai

      Delete
  2. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    java training in bangalore

    ReplyDelete
  3. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.


    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

    ReplyDelete
  4. A really good post man, very thankful and hopeful that you will write many more posts like this one.
    appvn

    ReplyDelete
  5. Thank you so much for posting this. I really appreciate your work. Keep it up. Great work!Best software training company with placement in Hyderabad

    ReplyDelete
  6. Hi,Very nice post.Thanks for the information provided.It is very interesting and very informative.I am always impressed with your post and helpful tips.Keep posting tips and relevant content as usual.
    big data training in btm

    ReplyDelete

  7. نجار ابواب بالرياض نجار بالرياض
    تركيب ستائر بالرياض شركة تركيب ستائر بالرياض
    تنظيف مكيفات بالرياض شركة تنظيف مكيفات بالرياض
    شركة تنظيف افران الغاز بالرياض شركة تنظيف افران بالرياض

    ReplyDelete

  8. Best Article buy Pain Pills online Excellent post. I appreciate this site. Stick with it! Because the admin of this web page is working, no doubt very quickly it will be well-known, due to its quality contents.This website was how do you say it? Relevant!! Finally, I’ve found something that helped me.
    Best Article buy Roxicodone online Excellent post
    buy Xanax online
    buy Oxycodone online

    Best Article buy Pain Medications online Excellent post. I appreciate this site. Stick with it! Because the admin of this web page is working, no doubt very quickly it will be well-known, due to its quality contents.This website was how do you say it? Relevant!! Finally, I’ve found something that helped me.

    buy Research Chemicals online

    buy Roxicodone online

    buy Cbd Isolate online

    ReplyDelete
  9. Thanks for sharing your innovative ideas to our vision. I have read your blog and I gathered some new information through your blog. Your blog is really very informative and unique. Keep posting like this. Awaiting for your further update. If you are looking for any Python programming related information, please visit our website python training institute in Bangalore

    ReplyDelete
  10. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job,Keep it up.

    Try Our Psychic Reader In Toronto Canada Services and Get All the benefits of it in your life, we make All your Personal problems solved in just minutes.

    ReplyDelete

  11. Buy Farm Fresh seasonal Mango Fruits online at best price. Send Mangoes to all Pakistan like Karachi, Lahore, Islamabad, Rawalpindi, Peshawar, Multan, Faisalabad, Hyderabad and Jhelum on door step fruitoyepk.

    ReplyDelete
  12. movers and packers in dubai | noorahmedtransports Noor Ahmad Transport is a Mover and Packer service provider in Dubai Marina. Pickup Truck Rental Dubai 1/3 Ton - 7/10 Ton Truck with Tail Lift. Home/Office Shifting, Man with van service in Dubai. Call: 050 651 2943

    ReplyDelete
  13. movers and packers in dubai marina Noor Ahmad Transport is a Mover and Packer service provider in Dubai Marina. Pickup Truck Rental Dubai 1/3 Ton - 7/10 Ton Truck with Tail Lift. Home/Office Shifting, Man with van service in Dubai. Call: 050 651 2943

    ReplyDelete
  14. Pickup Truck rental dubaiPickup Rental Dubai for Home shifting, Office shifting,Man with van, Pickup for rent in Dubai: Contact us 0551375065

    ReplyDelete
  15. Thanks for Sharing such an amazing article. Keep working. Your Site is very nice, and it's very helping us. this post is unique and interesting, thank you for sharing this awesome information.
    We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental in Abu Dhabi

    ReplyDelete
  16. Hi, this is an amazing blog there is more effective information. We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental Dubai

    ReplyDelete
  17. Thanks for Sharing such an amazing article. Keep working. Your Site is very nice, and it's very helping us. this post is unique and interesting, thank you for sharing this awesome information.
    We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental in Abu Dhabi

    ReplyDelete
  18. Thanks for showing that kind of knowledge on your portal. keep it up. Getting so much information after reading your blog. We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental abu-dhabi

    ReplyDelete
  19. Thanks for showing that kind of knowledge on your portal. keep it up. Getting so much information after reading your blog. We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental abu-dhabi

    ReplyDelete
  20. Hi, this is an amazing blog there is more effective information. We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental Dubai

    ReplyDelete
  21. Thanks for Sharing such an amazing article. Keep working. Your Site is very nice, and it's very helping us. this post is unique and interesting, thank you for sharing this awesome information.
    We are providing logistic services for the last 20 years. Noor Ahmed Transports LLC has built a reputation of reliability in the transportation of goods and logistics locally in Dubai due to our professional approach, competitive pricing, and a large network of vehicles.
    trailer rental dubai

    ReplyDelete
  22. We provide home delivery of the freshest fresh fruits and vegetables,
    and “ grocery in Lahore ”grocery in Lahore.

    ReplyDelete
  23. We provide home delivery of the freshest fresh fruits and vegetables,
    and grocery delivery in Lahore .

    ReplyDelete
  24. NukarShop.pkis Best Online Delivery Service in Pakistan. Order your Favourite Food, Grocery, Books, Beauty Products & a range of other Items on NukarShop.pk with Instant Delivery Service. Order via NukarShop app.

    ReplyDelete
  25. NukarShop.pkis Best Online Delivery Service in Lahore. Order your Favourite Food, Grocery, Books, Beauty Products & a range of other Items on NukarShop.pk with Instant Delivery Service. You can Order groceries online in Lahore Order groceries online in Lahore via NukarShop app.

    ReplyDelete
  26. Buy grocery near me . Choose from 1,000+ products from Fresh Fruits, Vegetables, Bakery Products, Meat & Poultry. Snacks, Household Cleaning Items & Much More. Fresh Vegetables & Fruits. Bakery items. Office supplies. Meat One. Neighborhoods: Gulberg, DHA, Jauhar Town, Model Town.

    ReplyDelete
  27. 3 Ton Pickup Truck Rental is the best choice for you in Dubai and all over UAE. You can take our services for various reasons like furniture moving, house moving, office moving, and bundling, moving, a man with a van, business and corporate freight benefits in all over Dubai. We are otherwise called “Pickup rental in Dubai” and our mission is to offer you with the market rates. Peoples in the UAE all-time favor online movers’ organizations when they need to move. Our team is expert in 3-ton pickup for rent , 5-ton pickup for rent, 7-ton pickup for rent and 10-ton pickup for rent.
    Noor Ahmed Transports LLC has Pickup Trucks all over UAE. We offer 3-ton 5-ton 7-ton and 40 Feet Long trailers for rent with drivers in UAE.

    ReplyDelete
  28. office & home shifting services. Pickup for Rent in Arabian Ranches Dubai UAE. Hire 5 Ton, 7 Ton, 10 Ton Pickup for Rent Dubai & Trailer Truck Rental Dubai. pickup for rent in mirdif. pickup for rent in jvc. pickup for rent in al barsha. pickup for rent in sports city. pickup for rent in silicon oasis. pickup for rent in dubai .We provide reliable & cost effective pickup rental services as per your business.

    ReplyDelete
  29. Trailer Rental Dubai - Trailer Truck for Rent in UAE 0558902008
    Get Heavy Trailer Truck for Rent in UAE, Trailer Rental Dubai , Trailer for Rent in Sharjah, Trailer Truck for Rent Call: 055 890 2008. Trailer truck for rent in uae, trailer for rent in sharjah, trailer rental abu dhabi.

    ReplyDelete
  30. Noor Ahmed Transports is the best option for you. You can take our services for different reasons like furniture shifting, house shifting, office shifting, and packaging, moving, man with van, commercial & corporate cargo services in all over Dubai. We are also known as
    Pickup For Rent in Green Community and our motto is to provide you the best services at best market rates. People in the UAE always prefer online movers’ companies when the need to shifting. Our team is experienced in 3-ton pickup for rent, 5-ton pickup for rent, 7-ton pickup for rent and 10-ton pickup for rent.

    ReplyDelete
  31. Noor Ahmed Transports is the best option for you. You can take our services for different reasons like furniture shifting, house shifting, office shifting, and packaging, moving, man with van, commercial & corporate cargo services in all over Dubai. We are also known as
    Pickup For Rent in International City Dubai and our motto is to provide you the best services at best market rates. People in the UAE always prefer online movers’ companies when the need to shifting. Our team is experienced in 3-ton pickup for rent, 5-ton pickup for rent, 7-ton pickup for rent and 10-ton pickup for rent.

    ReplyDelete
  32. Noor Ahmed Transports LLC provides 3 Ton to 10 Ton
    Pickup for Rent in Emirates Hill Dubai . Truck Rental Dubai UAE. Pickup Truck Service and all Logistics Movers and Packers services – Trailer Truck for Rent Dubai, for Commercial Transportation, Home/Office Shifting, Man with van service in Dubai. We are here to serve you. We have big and small moving vans, Pickup Truck for Moving, Pickup Rental Dubai for Home shifting, Office shifting, Man with a van, Pickup for rent in Dubai, Truck Rent in Dubai, Pickup Truck rental Dubai and Dubai Movers & Packers.We provide pickup trucks for rent in Dubai, Abu Dhabi, Ras Al Khaimah from Sharjah. Ton 3 and 7 Ton and 10 Ton Pickup Available for rental. We have a team of experienced drivers available.

    ReplyDelete
  33. Pickup For Rent in Dubai Sports City , Pickup Truck and Trailers for rent or lease on a nationwide basis. If your company needs pickup Trucks or trailers on a daily, weekly, monthly or yearly lease/rental Noor Ahmed Transports can provide those in a timely manner either delivered or your pickup with your drivers. With over 20 years of experience in knowing what fits your needs we have old, new and reconditioned flatbed trailers and pickup Trucks at your disposal.

    ReplyDelete
  34. Noor Ahmed Transports LLC provides 3 Ton to 10 Ton Pickup Truck Rental Dubai. Truck Rental Dubai UAE. Pickup Truck Service and all Logistics Movers and Packers services – Trailer Truck for Rent Dubai, for Commercial Transportation, Home/Office Shifting, Man with van service in Dubai. We are here to serve you. We have big and small moving vans, Pickup Truck for Moving, Pickup Rental Dubai for Home shifting, Office shifting, Man with a van, Pickup for Rent in Elite Residence Dubai . Truck Rent in Dubai, Pickup Truck rental Dubai and Dubai Movers & Packers.

    ReplyDelete
  35. Pickup For Rent in International Financial Center . Hire 5 Ton, 7 Ton, 10 Ton Pickup for Rent Dubai & Trailer Truck Rental Dubai. pickup for rent in mirdif. pickup for rent in jvc. pickup for rent in al barsha. pickup for rent in sports city. pickup for rent in silicon oasis. pickup for rent in dubai . We provide reliable & cost effective pickup rental services as per your business. office & home shifting services.

    ReplyDelete
  36. Pickup For Rent in Internet City Dubai , Pickup Truck and Trailers for rent or lease on a nationwide basis. If your company needs pickup Trucks or trailers on a daily, weekly, monthly or yearly lease/rental Noor Ahmed Transports can provide those in a timely manner either delivered or your pickup with your drivers. With over 20 years of experience in knowing what fits your needs we have old, new and reconditioned flatbed trailers and pickup Trucks at your disposal.

    ReplyDelete
  37. A Pickup For Rent in Dubai Investment Park can provide you with space and strength you’ve been missing for all those tough jobs. Full-size pickups command attention on the road with their size and power. We provide services like10 ton pickup for rent, pickup for moving Dubai,3-ton pickup for rent in Dubai, pickup for rent in bur Dubai, pickup for rent in Sharjah,7 ton pickup for rent in Dubai.

    ReplyDelete
  38. Mumtaz Hussain Technical Services llc is a villa painting services, painting services in dubai marina, best painting company dubai , wall painting services, studio painting in dubai, home painting services, painting walls service, painting services in abu dhabi.

    ReplyDelete
  39. Mumtaz Hussain Technical Services llc is a villa painting services, painting services in dubai marina, best painting company dubai , wall painting services, studio painting in dubai, home painting services, painting walls service, painting services in abu dhabi.

    ReplyDelete
  40. We provide Villa painting services,
    Apartment painting services, office painting services, showrooms painting services and building painting services in dubai.
    Mumtaz Hussain Technical Services llc is a best villa painting services, offering painting services in dubai marina as a best painting company dubai. our services include wall painting services, studio painting in dubai, home painting services, painting walls service, painting services in abu dhabi.

    ReplyDelete
  41. We provide professional Wall Painting services in Dubai. Including designer painting, wall texture designs, 2D, 3D, and other wall art painting works. We also provide Villa painting services, Apartment painting services, office painting services, showroom painting services, and building painting services in Dubai.
    Mumtaz Hussain Technical Services LLC is the best villa painting service, offering painting services in Dubai marina as the best painting company Dubai. our services include wall painting services, studio painting in Dubai, home painting services, painting walls service, painting services in Abu Dhabi.

    ReplyDelete
  42. Best Villa Painting Services. Book professional Villa Painting Services Dubai. Our team of interior and exterior villa expert painters. Get Free Quotation in Minutes. We provide a complete solution for apartment and villa painting services in Dubai. All the experts of Fixito can give your wall a complete new and modern look to get the house of your dreams. We provide you the best plan to achieve your satisfaction by providing paramount painting services in Dubai.

    ReplyDelete
  43. Noor Ahmed Transport is a professional professional
    trailer rental in Dubai , UAE. We have the best performing and bard new vehicles for long-distance and heavy load transportation all over UAE. Our goal is to have a personalized, flexible, and professional service when dealing with our clients for domestic and commercial transportation needs. Noor Ahmed Transports’ modern fleet of transport Trucks and Trailers in wide range and capacity, including everything from 1 Ton Pickup Truck to Flatbed Trailers.

    ReplyDelete
  44. Noor Ahmed Transport is a professional professional
    trailer rental in Dubai , UAE. We have the best performing and bard new vehicles for long-distance and heavy load transportation all over UAE. Our goal is to have a personalized, flexible, and professional service when dealing with our clients for domestic and commercial transportation needs. Noor Ahmed Transports’ modern fleet of transport Trucks and Trailers in wide range and capacity, including everything from 1 Ton Pickup Truck to Flatbed Trailers.

    ReplyDelete
  45. Good painting on your walls help brighten your room in a way that looks fresh and subtle. Interior & Exterior Villa Painting Services in Dubai. Office & Apartment Painting. All Fitout Services. Villa Exterior Painting. Affordable Prices. Best Painters. Experienced & Skilled.Offering wall painting services in dubai,
    painting services in dubai reviews, villa painting services, painting services in dubai marina, studio painting in dubai
    best painting company dubai.

    ReplyDelete
  46. Nukarshop provides a variety of services including renovation, and repair & online maintenance services. We offer the below-mentioned services with a 100% guarantee of customer satisfaction.Plumber Services, Cleaning Services,Painter Services,Electrician Services, Ac services and handyman services.

    ReplyDelete
  47. Order grocery delivery near me Online Get Farm-Fresh Fruits & Vegetables Delivered To Your Door Step with fast delivery. Online Shopping on nukarshop.pk for Best Deals on Household & Personal, Cooking Items & Much More. Register Online. Sign Up For Newsletter. Download Mobile App.

    ReplyDelete
  48. Trucks & Trailer for Rent in UAE.We offer brand new flat Trailers for Cheap Rentals in all over UAE. Need to transport some cargo and you are looking for a truck? We offer the best rental services for flat trailers and other solutions for transport and logistic services. Flat trailer rental in uae, trailer rental dubai, tipper trailer for rent, trailer for rent in sharjah, trailer rental abu dhabi, trailer rental dubai to saudi.

    ReplyDelete
  49. Noor Ahmed Transport is a professional
    Trailer Truck for Rent in UAE. We have the best performing and new vehicles for significant length and weighty burden transportation all over UAE. We will probably have a customized, adaptable, and proficient assistance when managing our customers. Noor Ahmed Transports’ advanced armada of hefty vehicle trucks and trailers in wide reach and limit, including everything from 3 Ton Pickup Truck to Flatbed Trailers.

    ReplyDelete
  50. Safe Driver Dubai is one of the leading chauffeured car service providers in the UAE. When you are looking for a reliable person to comfort drive for you. Professional safe driver Dubai drives you home safely, book reliable & designated safer driver. Get driver in 20 minutes starting from 80 AED only. The Driver offers world-class chauffeur services in Abu Dhabi and Dubai at cost-effective prices. The safe driver Dubai service promotes the Corporate Social Responsibility (CSR) by offering designated drivers to you. Hire a
    Safe Driver Dubai. Safe driver Dubai Most Trusted Private Driver Service in UAE. Required Safe driver Dubai? Call +971 55 7303 487 to hire a driver in Dubai.

    ReplyDelete
  51. We offers a comprehensive range of Technical Services for domestic and commercial properties at a reasonable price in all over UAE. We are trusted and reliable technical services provider who serve customers in the Dubai as well as throughout United Arab Emirates. We have extensive experience of domestic and business Technical Services and no job is too small. Our customers value our professionalism, work ethic and our competitive prices. We offer these services.
    Painting Contracting Services, Building Cleaning Services, Carpentry & Flooring Contracting, Floor and Wall Tiling Works, Storage services , Engraving & Ornamentation Works, Post Construction Cleaning Service, and Pickup Truck Hire/Moving & Shifting Service.

    ReplyDelete
  52. its been long since i saw a post that's so educative and informational. i will make sure to share this my facebook group. you can also view contents on our websites below.

    French Bulldog Puppies For Sale

    French Bulldog Breeders

     French Bulldog Puppies For Sale Near Me

    French Bulldog Puppies For adoption

    French Bulldog Puppies


    Blue French Bulldog Puppies

    ReplyDelete
  53. Safe Driver Dubai is your all-inclusive resource when you are searching for the Safe Driver Service Dubai. We have aptitude in giving you the best wellbeing driving administrations. At the point when you are searching for a dependable individual to comfort drive for you or your friends and family, our best protected Safe Drivers in Dubai will meet your interests. We spend significant time in offering security driver administrations. You can rely on us while chasing for the best protected driver organization in UAE.

    ReplyDelete

  54. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.

    Derivative Calculator
    edublackboards
    How to find the height of a trapezoid
    Area of a Rectangle

    ReplyDelete