2

I am creating a jdbc object in spark driver and I am using that in executor to access the db. So my concern is that is it the same connection object or executors would get a copy of connection object so there would be separate connection per partition?

Improve this question
1
  • Did answer help? Commented Mar 21, 2022 at 6:17

1 Answer 1

Reset to default
1
  • In Scala - not that you state if pyspark, Java or Scala,
    • you can create an Object for a ConnectionPool as per link below;
      • this will be instantiated for each Executor and shared by Cores comprising that Executor. I.e. not as a Singleton for all foreachPartition.
        • Your concern / question is addressed in the previous bullet, i.e. their own Connections.

See https://medium.com/@ravishankar.nair/implementing-a-connectionpool-in-apache-sparks-foreachpartition-4add46dc8ae2. It's a good reference from medium.com.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.