4

In HiveThriftServer2 class, what is the difference between calling the startWithContext vs calling the main?

I have a customer UDF jar that I want to register, so that every time when the thrift server boots up, all these are auto configure. Is there a way to do this?

Can I use Hive context to register the UDF jar and functions and call the HiveThriftServer2.startWithContext to start up the server?

Thanks

seamonkeys
  • 123
  • 4
  • 1
    If you add the JAR onto the CLASSPATH, then restart the server, it'll get auto configured – OneCricketeer Sep 02 '17 at 05:16
  • And calling `main` kills all previous SparkContext – OneCricketeer Sep 02 '17 at 05:18
  • @cricket_007 Can you clarify what you mean by adding the JAR onto the CLASSPATH? I tried to start the Thrift Server by **sudo -u $SVC_USER /usr/lib/spark/sbin/start-thriftserver.sh --jars /opt/lib/custom-udfs.jar** but this doesn't register the custom-udfs.jar. Thanks – seamonkeys Sep 02 '17 at 23:28
  • Well, I was thinking `$SPARK_HOME/jars`, or you give `--jars` with the spark executor, not the server, as mentioned here. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_spark-component-guide/content/calling-udfs.html – OneCricketeer Sep 03 '17 at 00:15
  • Possible duplicate of [How to access custom UDFs through Spark Thrift Server?](https://stackoverflow.com/questions/45553037/how-to-access-custom-udfs-through-spark-thrift-server) – OneCricketeer Sep 03 '17 at 00:15
  • And please don't repost your quesitons – OneCricketeer Sep 03 '17 at 00:16

1 Answers1

2

What you are looking for is called hive.aux.jars.path, and it's a Hive property, not Spark specific.

I personally haven't tried it, but I'm thinking something like this

./sbin/start-thriftserver.sh \
  --hiveconf hive.aux.jars.path=file:///opt/lib/custom-udfs.jar

References

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I tried your suggestion doing `./sbin/start-thriftserver.sh \ --hiveconf hive.aux.jars.path=file:///opt/lib/custom-udfs.jar` Then connect to Thrift Server via Beeline `beeline -u jdbc:hive2://localhost:10000/default` . But when I `list jars`, the custom-udfs.jar is still not listed as a resource. Any ideas? – seamonkeys Sep 03 '17 at 17:47
  • Have you tried copying the jar into spark folder or not? – OneCricketeer Sep 03 '17 at 18:43
  • 1
    I got this working by having the custom-udfs.jar in the `/usr/lib/hive/auxlib` directory instead of `/opt/lib/`. Like you suggested, specify `hive.aux.jars.path` pointing to it. This is working because I don't need to do `add jar` once I am connected via Beeline before creating a function. My question is now how come the hive `list jars` command doesn't list the custom-udfs.jar? – seamonkeys Sep 04 '17 at 04:05
  • Not sure. Probably a bug – OneCricketeer Sep 04 '17 at 13:57