Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Минимальный проект на Spark

Код

Code Block
import org.apache.spark.sql.SparkSession

object Minimal {

  def main(args: Array[String]) {
    val spark = SparkSession.builder.appName("Spark Application").getOrCreate()
    spark.stop()
  }
}

В зависимости проекта должны быть включены библиотеки spark-core и spark-sql

Запуск


Code Block
set HADOOP_HOME=<path to bin/winutils.exe>
set SPARK_HOME=<path to spark home>
set MAIN_CLASS=Minimal

%SPARK_HOME%\bin\spark-submit --class %MAIN_CLASS% --master local[2] your_jar.jar


Загрузка, сохранение данных в разных форматах

...