Hadoop and Spark Versions
DataProc supports the following Hadoop and Spark versions. Note that this will change with time as new versions come out:
- Spark 1.5.2
- Hadoop 2.7.1
- Pig 0.15.0
- Hive 1.2.1
- GCS connector 1.4.3-hadoop2
- BigQuery connector 0.7.3-hadoop2 (https://github.com/GoogleCloudPlatform/bigdata-interop)
For more information, go to http://cloud.google.com/dataproc-versions.
In the following steps, we will use Google Cloud Console (the user interface used to create a Spark Cluster and submit a job).