Redis on Spark:Task not serializable

In Spark, the functions on RDD s (like map here) are serialized and send to the executors for processing. This implies that all elements contained within those operations should be serializable.

The Redis connection here is not serializable as it opens TCP connections to the target DB that are bound to the machine where it's created.

The solution is to create those connections on the executors, in the local execution context. There're few ways to do that. Two that pop to mind are:

• rdd.mapPartitions : lets you process a whole partition at once, and therefore amortize the cost of creating connections)
• Singleton connection managers: Create the connection once per executor

mapPartitions is easier as all it requires is a small change to the program structure:

val perhit = perhitFile.mapPartitions{partition =>

    val r = new RedisClient("192.168.1.101", 6379) // create the connection in the context of the mapPartition operation

    val res = partition.map{ x =>

        ...

        val refStr = r.hmget(...) // use r to process the local data

    }

    r.close // take care of resources

    res

}

A singleton connection manager can be modeled with an object that holds a lazy reference to a connection (note: a mutable ref will also work).

object RedisConnection extends Serializable {

   lazy val conn: RedisClient = new RedisClient("192.168.1.101", 6379)

}

This object can then be used to instantiate 1 connection per worker JVM and is used as a Serializable object in an operation closure.

val perhit = perhitFile.map{x =>

    val param = f(x)

    val refStr = RedisConnection.conn.hmget(...) // use RedisConnection to get a connection to the local data

    }

}

The advantage of using the singleton object is less overhead as connections are created only once by JVM (as opposed to 1 per RDD partition)

There're also some disadvantages:

• cleanup of connections is tricky (shutdown hook/timers)
• one must ensure thread-safety of shared resources

(*) code provided for illustration purposes. Not compiled or tested.

You're trying to serialize the client . You have one RedisClient , r , that you're trying to use inside the map that will be run across different cluster nodes. Either get the data you want out of redis separately before doing a cluster task, or create the client individually for each cluster task inside your map block (perhaps by using mapPartitions rather than map , as creating a new redis client for each individual row is probably a bad idea).