Spark on Kubernetes: why and how to migrate your Spark pipelines to Cloud-Native Apache Spark
In the upcoming version of Spark (3.1), the Spark on Kubernetes integration will officially be declared production ready. A lot of companies have already adopted Spark on Kubernetes to benefit from containerization, reduce their costs, and make their architecture more portable and flexible. In this talk we'll go over the main pros & cons of running Spark on Kubernetes (as opposed to Hadoop YARN or proprietary platforms). The speaker, an ex-Databricks engineer now co-founder of Data Mechanics, a commercial Spark platform deployed on Kubernetes, will give practical tips to make this migration successful.
