Skip to main content

Return to Question

edited title
Link
stefanobaghino
stefanobaghino
  • 13.4k
  • 4
  • 44
  • 69

Using predicates in spark jdbcSpark JDBC read method

Source Link
ds_user
ds_user
  • 2.2k
  • 7
  • 42
  • 71

Using predicates in spark jdbc read

I am pulling data from sql server to hdfs. Here is my snippet for that,

val predicates = Array[String]("int_id < 500000", "int_id >= 500000 && int_id < 1000000") val jdbcDF = spark.read.format("jdbc") .option("url", dbUrl) .option("databaseName", "DatabaseName") .option("dbtable", table) .option("user", "***") .option("password", "***") .option("predicates", predicates) .load()

My Intellij IDE keeps saying that

"Type mismatch, expected Boolean or Long or Double or String, Actual : Array[String]"

in predicates. Not sure whats wrong with this. Can anyone see whats wrong with this? Also how do I use fetch size here?

Thanks.