Monday, December 2, 2019

Spark on k8s on Raspberry Pi's for Fun and Profit - Intro

...and we're back after a bit of time off from this.  Long version of the short story is that running Spark on Kubernetes (k8s) seems to be a very solid direction to take things in general.  Particularly from the standpoint of small, local experimentation.  But as with most "super simple" things like this there is a goodly pile of moving parts, a foundation of sorts, that can trip people up and turn that "Up and Running in 20 minutes" tutorial into a on-and-off two day frustration fest that ends in giving up or going a different direction - and that sucks.

So what I'm tinkering with now is a multi-node k8s cluster on Raspberry Pi's on which to run k8s on which to run Spark.  You may be thinking to yourself, "But those tiny things only have 1Gb RAM - you do get what Spark is right?!"  Ya, I get it, but if I've learned anything over the past couple years it is that starting with very small datasets is the absolute best way to develop data-centric applications (generally speaking).  What better way to force the issue than to be memory constrained?

 So why not just tip up some GKE instances and call it a day?  I'll leave that to the reader to debate internally.  For me this project is:
  • Neat
  • Fun
  • Looks cool on my desk
  • Gives me direct exposure to a bunch of bits of the stack that are otherwise obfuscated
  • I can also use the cluster for my home automation projects
  • I won't forget to kill a job and get whacked with a bigger than expected Google bill
  • ...and it looks cool on my desk
 Don't get me wrong, a lot of this series will be effectively "notes to self", but with hope these notes will be of value for others playing with a pile of ~$200 in hardware.  Also, some of the Pi bits might just be a missing bit of some other project - I'm learning as I go with the Pi so why not share?

In the end, if I do this right, I will assemble a zero-to-Spark-Job tutorial.  Until then, I'll just be adding bits as I go.  For example, next up is mounting USB drives to the Pi for extra storage.

Here we go...again.  Time to Spark it up some more.


  1. It's very useful blog post with inforamtive and insightful content and i had good experience with this information.I have gone through CRS Info Solutions Home which really nice. Learn more details About Us of CRS info solutions. Here you can see the Courses CRS Info Solutions full list. Find Student Registration page and register now.Find this real time DevOps Training and great teaching. Join now on Selenium Training online course. Upskill career with Tableau training by crs info solutions. Latest trending course is Salesforce Lightning training with excellent jobs.

  2. What an extremely wonderful post this is. Genuinely, perhaps the best post I've at any point seen to find in as long as I can remember. Goodness, simply keep it up.pmp certification in malaysia

  3. This is a great post I saw thanks to sharing. I really want to hope that you will continue to share great posts in the future.
    what is hrdf

  4. I have a strategic I'm seconds ago chipping away at, and I have been at the post for such data
    360DigiTMG data analytics course

  5. I looked at some very important and to maintain the length of the strength you are looking for on your website
    hrdf claimable courses"

  6. Buy Herbal Care Products online. We provide Natural Herbal Remedies, health and skin diseases information. Natural herbal products use men and women without side effects.

  7. Herbs Solutions By Nature offer Herbal Supplement for Health and Skin Diseases. Natural Herbal Treatment really does help you recover from your health condition.

  8. There are lots of Natural Remedies for Achalasia in market but these are very expensive. A product made by Natural Herbs Clinic is one of the useful and low prices Natural Remedy for Achalasia which works without any risk. It is a low price product and made with herbal ingredients that work without any side effects.

  9. Welcome to an incredible gathering here you will pick up every little thing about me

  10. I truly like your style of blogging. I added it to my preferred's blog webpage list and will return soon…
    masters in data science

  11. Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return.
    Best Institute for Data Science in Hyderabad

  12. Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good science course in Hyderabad

  13. I'm cheerful I found this blog! Every now and then, understudies need to psychologically the keys of beneficial artistic articles forming. Your information about this great post can turn into a reason for such individuals.
    arttificial intelligence training in aurangabad

  14. Latest Natural Health News around the world. Natural health magazine with new stories. Natural health tips and solutions including analysis and opinion on top health stories with best features of well-being.

  15. The best Natural Treatment of Atrial Fibrillation includes diet, exercise, yoga and herbs. A good nutrition is completely essential for good hearth. If you think that a petrol engine will not run on diesel fuel, it's easy to understand this idea. If you don't worry good food, then your body cannot help you as it is erected to. It lacks the necessary nutrients to function truthfully.

  16. I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
    Data Science courses

  17. Natural Herbal Treatment for Lichen Planus can help reduce the itching and discomfort. Herbal Supplement may help relieve symptoms linked with skin and mouth.


  18. I think I have never seen such blogs before that have completed things with all the details which I want. So kindly update this ever for us.

    Data Science Training in Hyderabad