Passing jdbc connection to spark read

Question

I have an sql script which creates temp tables valid for only that session. Now after running the script, I am trying to read data from the table through spark and then process it. Below is the code I have code for spark read.

sparkSession.read().format("jdbc").option("url", jdbcURL).option("dbtable", tableOrQuery).option("user", userName).option("password", password) .option("driver", driverName).load();

Now I need to pass the jdbc connection I created so that spark can read data in the same session. Is this possible ?

Incidentally I answered a similar question not so long ago - TL;DR; there can be no such option whatsoever. — user10938362
– user10938362, Commented Feb 12, 2019 at 18:50

Duy Nguyen · Accepted Answer · 2019-02-12 14:12:39Z

1

No, you cannot pass jdbc connection to spark. It will manage JDBC connection by itself.

JdbcRelationProvider Create Connection

JdbcUtils connect

answered Feb 12, 2019 at 14:12

Duy Nguyen

1,0256 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

HariJustForFun Over a year ago

Thanks. Do you have any suggestions what I can do instead ?

Duy Nguyen Over a year ago

You can get connection metadata and build Spark URL according to it, solution is in stackoverflow.com/questions/5718952/…

Collectives™ on Stack Overflow

Passing jdbc connection to spark read

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related