Spark在Windows上调试

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

1. 背景

(1) spark的一般开发与运行流程是在本地Idea或Eclipse中写好对应的spark代码，然后打包部署至驱动节点，然后运行spark-submit。然而，当运行时异常，如空指针或数据库连接等出现问题时，又需要再次修改优化代码，然后再打包....有木有可能只需一次部署？

(2) 当新版本的spark发布时，想立刻马上体验新特性，而当前没有现成的spark集群，或spark集群版本较老，又如何体验新特性呢？

2. 方案

(1) 无需多次打包测试，直接在本地测试或调试通过，然后只需要打包部署一次即可。

spark支持standalone本地模式，初始化SparkConf时，设置master时，仅需指定"local[*]"或"local[1]"

(2) 基于本地模式，即使无现有的spark集群，也可以调试新版本的spark

只需在sbt或maven的配置文件中增加新版本的依赖即可。

(3) 设置spark的日志级别

spark默认打印INFO信息，比如我只想打印take操作后的少许数据，但调用spark时打印日志太多，就得从一大堆日志中进行查找。因此更改spark的默认日志级别。具体配置如下：

# Set everything to be logged to the console

log4j.rootCategory=INFO, console

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose

log4j.logger.org.spark_project.jetty=ERROR

log4j.logger.org.spark_project=ERROR

log4j.logger.org.apache.spark=ERROR

log4j.logger.org.apache.parquet=ERROR

log4j.logger.parquet=ERROR

log4j.logger.io.netty=ERROR

log4j.logger.org.apache.hadoop=FATAL

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support

log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL

# 控制台输出

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p %c{1}:%L - %m%n

(4) 测试代码

import org.apache.spark.{SparkConf, SparkContext}

object Test {

  def main(args: Array[String]): Unit = {

    val sc = new SparkContext(new SparkConf().setMaster("local[1]").setAppName("test"))

    println(sc.version)

    sc.parallelize(List(1,2,3,4)).foreach(println)

    sc.stop()

  }

}

运行结果

log4j: Trying to find [log4j.xml] using context classloader sun.misc.Launcher$AppClassLoader@18b4aac2.

log4j: Trying to find [log4j.xml] using sun.misc.Launcher$AppClassLoader@18b4aac2 class loader.

log4j: Trying to find [log4j.xml] using ClassLoader.getSystemResource().

log4j: Trying to find [log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@18b4aac2.

log4j: Using URL [file:/E:/IntelliJWorkSpace/AIMind-backend/aimind_backend/pipeline-tools/target/classes/log4j.properties] for automatic log4j configuration.

log4j: Reading configuration from URL file:/E:/IntelliJWorkSpace/AIMind-backend/aimind_backend/pipeline-tools/target/classes/log4j.properties

log4j: Parsing for [root] with value=[INFO, console].

log4j: Level token is [INFO].

log4j: Category root set to INFO

log4j: Parsing appender named "console".

log4j: Parsing layout options for "console".

log4j: Setting property [conversionPattern] to [%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n].

log4j: End of parsing for "console".

log4j: Setting property [target] to [System.err].

log4j: Parsed "console" options.

log4j: Parsing for [org.spark_project.jetty] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category org.spark_project.jetty set to ERROR

log4j: Handling log4j.additivity.org.spark_project.jetty=[null]

log4j: Parsing for [org.spark_project] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category org.spark_project set to ERROR

log4j: Handling log4j.additivity.org.spark_project=[null]

log4j: Parsing for [org.apache.spark] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category org.apache.spark set to ERROR

log4j: Handling log4j.additivity.org.apache.spark=[null]

log4j: Parsing for [org.apache.hadoop.hive.metastore.RetryingHMSHandler] with value=[FATAL].

log4j: Level token is [FATAL].

log4j: Category org.apache.hadoop.hive.metastore.RetryingHMSHandler set to FATAL

log4j: Handling log4j.additivity.org.apache.hadoop.hive.metastore.RetryingHMSHandler=[null]

log4j: Parsing for [parquet] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category parquet set to ERROR

log4j: Handling log4j.additivity.parquet=[null]

log4j: Parsing for [io.netty] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category io.netty set to ERROR

log4j: Handling log4j.additivity.io.netty=[null]

log4j: Parsing for [org.apache.hadoop] with value=[FATAL].

log4j: Level token is [FATAL].

log4j: Category org.apache.hadoop set to FATAL

log4j: Handling log4j.additivity.org.apache.hadoop=[null]

log4j: Parsing for [org.apache.parquet] with value=[ERROR].

log4j: Level token is [ERROR].

log4j: Category org.apache.parquet set to ERROR

log4j: Handling log4j.additivity.org.apache.parquet=[null]

log4j: Finished configuring.

2.4.1

1

2

3

4