I have a spark streaming application, for every batch, I need to insert it to the hbase which is protected by kerberos. I found a solution, that is in the driver side I create a connection and obtain a token from that conn and then pass it to the executor. In the executor side, I decode it and get the token, in this way I can insert data to hbase successfully. This seems good, but my concern is that will the token expired? If so , how to solve it please?
My code snippet is
val ugi=UserGroupInformation.loginUserFromKeytabAndReturnUGI(principle,keytabfile);
ugi.doAs(new PrivilegedAction[Unit]() {
def run(): Unit = {
conn = ConnectionFactory.createConnection(conf)
val token = TokenUtil.obtainToken(conn)
tokenStr = token.encodeToUrlString()
}
})
in the rdd.foreachpartition,
val token = new Token()
token.decodeFromUrlString(tokenStr)
UserGroupInformation.getCurrentUser.addToken(token)
Although I have searched a lot from Internet about this issue, but I did not found a good solution about this issue. The common answer to this question is
UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
But as my test, inside this method,
public synchronized void checkTGTAndReloginFromKeytab() throws IOException {
if (!isSecurityEnabled()
|| user.getAuthenticationMethod() != AuthenticationMethod.KERBEROS
|| !isKeytab)
return;
KerberosTicket tgt = getTGT();
if (tgt != null && Time.now() < getRefreshTime(tgt)) {
return;
}
reloginFromKeytab();
}
The isKeytab is always false, so it will never execute the following code,I do not understand why this return false. So anybody can help me solve this question? Any help is appreciated!
