0

Hi I am new to Apache Spark and I am querying the hive tables using Apache spark sql in java.

And this is my code

 SparkConf sparkConf = new SparkConf().setAppName("Hive").setMaster("local"); JavaSparkContext ctx = new JavaSparkContext(sparkConf); HiveContext sqlContext = new org.apache.spark.sql.hive.HiveContext(ctx.sc()); org.apache.spark.sql.Row[] results = sqlContext.sql("Select * from Tablename where Column='Value'").collect(); org.apache.spark.sql.Row[] results = sqlContext.sql("Select * from Tablename where Column='Value1'").collect();

Also I tried running two different queries in the same application and I watched it is making connections each time with hive meta store. How to solve this and also tell me how to use persist option efficiently.

Improve this question
2
  • If the queries were unrelated, then it makes sense that the Hive meta store is queried twice. It maybe helps if you post your program containing the queries. Commented Jul 27, 2015 at 6:49
  • Thanks for ur reply...Another query is nothing but querying the same table with different value of the same column Commented Jul 27, 2015 at 6:59

1 Answer 1

Reset to default
1

It might help to call sqlContext.cacheTable("Tablename") before executing the two queries.

According to the docs it does what you're looking for.

Caches the specified table in-memory.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Thanks a lot. Also I have another question regarding this. I am using spark in java here so when I run this for the first time it caches the table it probably take few minutes to get the result but when I run this again for the second time will it be faster then the earlier?
The caching and thus the network I/O will be done every time you run your program. The advantage of caching is that you can reuse intermediate results within the same program. But what might speed up your program a little bit is the JVM warmup of the Spark cluster.
Also I have another doubt... I am using Spark sql in web services in which I am passing the different column values for the above queries and how to use the same hive context for every request
I'm not a web service expert but if you're using JAX-WS, then this might help you: stackoverflow.com/a/11096654/4815083
...I have an issue in running spark in hadoop multi node cluster can u hel help me in this?
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.