Monday, December 2, 2019

Spark on k8s on Raspberry Pi's for Fun and Profit - Intro

...and we're back after a bit of time off from this.  Long version of the short story is that running Spark on Kubernetes (k8s) seems to be a very solid direction to take things in general.  Particularly from the standpoint of small, local experimentation.  But as with most "super simple" things like this there is a goodly pile of moving parts, a foundation of sorts, that can trip people up and turn that "Up and Running in 20 minutes" tutorial into a on-and-off two day frustration fest that ends in giving up or going a different direction - and that sucks.

So what I'm tinkering with now is a multi-node k8s cluster on Raspberry Pi's on which to run k8s on which to run Spark.  You may be thinking to yourself, "But those tiny things only have 1Gb RAM - you do get what Spark is right?!"  Ya, I get it, but if I've learned anything over the past couple years it is that starting with very small datasets is the absolute best way to develop data-centric applications (generally speaking).  What better way to force the issue than to be memory constrained?

 So why not just tip up some GKE instances and call it a day?  I'll leave that to the reader to debate internally.  For me this project is:
  • Neat
  • Fun
  • Looks cool on my desk
  • Gives me direct exposure to a bunch of bits of the stack that are otherwise obfuscated
  • I can also use the cluster for my home automation projects
  • I won't forget to kill a job and get whacked with a bigger than expected Google bill
  • ...and it looks cool on my desk
 Don't get me wrong, a lot of this series will be effectively "notes to self", but with hope these notes will be of value for others playing with a pile of ~$200 in hardware.  Also, some of the Pi bits might just be a missing bit of some other project - I'm learning as I go with the Pi so why not share?

In the end, if I do this right, I will assemble a zero-to-Spark-Job tutorial.  Until then, I'll just be adding bits as I go.  For example, next up is mounting USB drives to the Pi for extra storage.

Here we go...again.  Time to Spark it up some more.

36 comments:

  1. It's very useful blog post with inforamtive and insightful content and i had good experience with this information.I have gone through CRS Info Solutions Home which really nice. Learn more details About Us of CRS info solutions. Here you can see the Courses CRS Info Solutions full list. Find Student Registration page and register now.Find this real time DevOps Training and great teaching. Join now on Selenium Training online course. Upskill career with Tableau training by crs info solutions. Latest trending course is Salesforce Lightning training with excellent jobs.

    ReplyDelete
  2. What an extremely wonderful post this is. Genuinely, perhaps the best post I've at any point seen to find in as long as I can remember. Goodness, simply keep it up.pmp certification in malaysia

    ReplyDelete
  3. This is a great post I saw thanks to sharing. I really want to hope that you will continue to share great posts in the future.
    what is hrdf

    ReplyDelete
  4. I have a strategic I'm seconds ago chipping away at, and I have been at the post for such data
    360DigiTMG data analytics course

    ReplyDelete
  5. I looked at some very important and to maintain the length of the strength you are looking for on your website
    "
    hrdf claimable courses"

    ReplyDelete
  6. Buy Herbal Care Products online. We provide Natural Herbal Remedies, health and skin diseases information. Natural herbal products use men and women without side effects.

    ReplyDelete
  7. Herbs Solutions By Nature offer Herbal Supplement for Health and Skin Diseases. Natural Herbal Treatment really does help you recover from your health condition.

    ReplyDelete
  8. There are lots of Natural Remedies for Achalasia in market but these are very expensive. A product made by Natural Herbs Clinic is one of the useful and low prices Natural Remedy for Achalasia which works without any risk. It is a low price product and made with herbal ingredients that work without any side effects.

    ReplyDelete
  9. Welcome to an incredible gathering here you will pick up every little thing about me
    https://360digitmg.com/masters-in-full-stack-data-scientist-course/

    ReplyDelete
  10. I truly like your style of blogging. I added it to my preferred's blog webpage list and will return soon…
    masters in data science

    ReplyDelete
  11. Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return.
    Best Institute for Data Science in Hyderabad

    ReplyDelete
  12. Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work.data science course in Hyderabad

    ReplyDelete
  13. I'm cheerful I found this blog! Every now and then, understudies need to psychologically the keys of beneficial artistic articles forming. Your information about this great post can turn into a reason for such individuals.
    arttificial intelligence training in aurangabad

    ReplyDelete
  14. Latest Natural Health News around the world. Natural health magazine with new stories. Natural health tips and solutions including analysis and opinion on top health stories with best features of well-being.

    ReplyDelete
  15. The best Natural Treatment of Atrial Fibrillation includes diet, exercise, yoga and herbs. A good nutrition is completely essential for good hearth. If you think that a petrol engine will not run on diesel fuel, it's easy to understand this idea. If you don't worry good food, then your body cannot help you as it is erected to. It lacks the necessary nutrients to function truthfully.

    ReplyDelete
  16. I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
    Data Science courses

    ReplyDelete
  17. Natural Herbal Treatment for Lichen Planus can help reduce the itching and discomfort. Herbal Supplement may help relieve symptoms linked with skin and mouth.

    ReplyDelete

  18. I think I have never seen such blogs before that have completed things with all the details which I want. So kindly update this ever for us.

    Data Science Training in Hyderabad

    ReplyDelete
  19. Natural Remedies for Achalasia dealt with straightforwardly by forceful widening. Natural remedies protect the throat from pain and Dairy products also help with Achalasia disease. Buy Herbal Product for Achalasia provides a supplement to cure Achalasia infection. These are the essential cures that help to improve the throat.

    ReplyDelete
  20. Diet Plan

    for Weight Loss
    . A healthy eating plan gives your body the

    nutrients it needs every day while staying within daily calorie goal

    for weight loss.

    ReplyDelete
  21. A debt of gratitude is in order for sharing the information, keep doing awesome... I truly delighted in investigating your site. great asset...
    data scientist training in hyderabad

    ReplyDelete
  22. If you are seeing any of these signs, make sure if you have Granuloma Annulare. Mild pain, irritation, and itching are some of the signs connected with this disorder. Buy Herbal Product for Granuloma Annulare. can use many patients to obtain relief from this problem. Natural Remedies for Granuloma Annulare. can help to decrease itching and the appearance of skin lesions.

    ReplyDelete
  23. Ooohhh! Now this has some heavy calories dear friends, like real heavy calories! It is a 2000 Calorie Diabetic Diet Plan great for the muscular giants to satisfy their hunger cravings and simultaneously maintain blood sugar levels in their bodies if they are diabetic.

    ReplyDelete
  24. Pickup Rental DubaiPickup Rental Dubai for Home shifting, Office shifting, Man with van, Pickup for rent in Dubai The majority of people always prefer to get 10 ton pickup for rent and 3 ton pickup for rent in Dubai. We aim to make you happy and satisfy with our quality services.
    Renting a Pickup Truck Choose a 3-ton pickup truck for rent available for a very low price in all locations across Dubai, Sharjah, and neighboring emirates. We also offer bigger trailer trucks for rent for better logistics support in moving heavy machinery
    We provide 3 Ton, 5 Ton, 7 Ton, 10 Ton Pickup Truck Rental Dubai with Tail Lift Pickup for Rent in Dubai Available 24/7

    ReplyDelete
  25. Trailer Rental Dubai, UAE Noor Ahmed Transport is a professional trailer rental in Dubai, UAE. We have the best performing and bard new vehicles for long-distance and heavy load transportation all over UAE. Our goal is to have a personalized, flexible, and professional service when dealing with our clients. Get Heavy Trailer Truck for Rent in UAE, Trailer Rental Dubai , Trailer for Rent in Sharjah, Trailer Truck for Rent Call

    ReplyDelete
  26. Pickup Truck Rental Ajman , Al Ajman. Shifting house villas office carpenter labor Packing good service and best price. Transportation Service to All UAE. Pickup Rental Dubai/Pickup Truck Rental Dubai 3-ton pickup Abu Dhabi. 3 Ton Pickup Canter For Rent. 3 Ton pickup for rent in Dubai.

    ReplyDelete
  27. Trucks &Trailers for rent in Sharjah . Get comfortable trucks, trailers for cheap rent. Flatbed trailer available for rent. We are providing Transportation services with (40ft) long Trailer in Sharjah. Trailer Leasing & Rental Service's In Sharjah UAE.

    ReplyDelete