I am attempting to write my transformed data frame into MongoDB using this as a guide
https://docs.mongodb.com/spark-connector/master/scala/streaming/
So far, my reading of data frame from MongoDB works perfectly. As shown below.
val mongoURI = "mongodb://000.000.000.000:27017" val Conf = makeMongoURI(mongoURI,"blog","articles") val readConfigintegra: ReadConfig = ReadConfig(Map("uri" -> Conf)) val sparkSess = SparkSession.builder() .master("local") .appName("MongoSparkConnectorIntro") .config("spark.mongodb.output.uri", "mongodb://000.000.000.000:27017/blog.vectors") .getOrCreate() // Uses the ReadConfig val df3 = sparkSess.sqlContext.loadFromMongoDB(ReadConfig(Map("uri" -> "mongodb://000.000.000.000:27017/blog.articles"))) However, writing this data frame to MongoDB seems to prove more difficult.
//reads data from mongo and does some transformations val data = read_mongo() data.show(20,false) data.write.mode("append").mongo() For the last line, I receive the following error.
Exception in thread "main" java.lang.IllegalArgumentException: Missing database name. Set via the 'spark.mongodb.output.uri' or 'spark.mongodb.output.database' property This seems confusing to me as I set this within my spark Session in the code blocks above.
val sparkSess = SparkSession.builder() .master("local") .appName("MongoSparkConnectorIntro") .config("spark.mongodb.output.uri", "mongodb://000.000.000.000:27017/blog.vectors") .getOrCreate() Can you spot anything I'm doing wrong?
