6

I tried to implement a simple producer consumer example with kafka and I achieved with the following properties:

Properties configProperties = new Properties(); configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:" + portNumber); configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.ByteArraySerializer"); configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer"); // Belirtilen property ayarlarına sahip kafka producer oluşturulur org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties);

However when I try the exact same configs, and everything else the same, in another project which is a plugin for a data visualization software, I got this error:

.... // Here there is some other stuff but I thing the important one is the below one Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Producer at App.MyControlPanel.<init>(MyControlPanel.java:130) at App.CytoVisProject.<init>(CytoVisProject.java:29) ... 96 more Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.producer.Producer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 98 more

In the first one that I said it worked, I was using "mvn clean compile assembly:single", but in the second one I created a jar file for the whole project. Because the visualization software wants a jar file to install the plugin. Since every thing is same (At least I could not find any difference, I used same code) I guess the problem is about the way build the project. What happened here? What is the difference between "mvn clean compile assembly:single" and building a jar file in IntelliJ? Why I got this error and how to fix this? Thanks a lot for help!

As I said in the last comment of the first answer, I have a plugin which has manifest and transform as goal. Here:

<plugin> <groupId>com.springsource.bundlor</groupId> <artifactId>com.springsource.bundlor.maven</artifactId> <version>1.0.0.M2</version> <configuration> <outputManifest>C:\Users\USER\AppData\Local\Temp\archetype2tmp/META-INF/MANIFEST.MF</outputManifest> <failOnWarnings>false</failOnWarnings> <removeNullHeaders>true</removeNullHeaders> <manifestHeaders><![CDATA[Bundle-ManifestVersion: 2 Bundle-Name: CytoVisProject Bundle-SymbolicName: CytoVisProject Spring-DM-Version: ${pom.version} ]]></manifestHeaders> </configuration> <!-- generate the manifest automatically during packaging --> <executions> <execution> <id>bundle-manifest</id> <phase>package</phase> <goals> <goal>manifest</goal> <goal>transform</goal> </goals> </execution> </executions> </plugin>

If I use a shade plugin like below:

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.1.0</version> <configuration> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> </execution> </executions> </plugin>

It does not works, because I need to use manifest and transform as goals in my plugin. How can I add kafka classes to the jar file that IntelliJ creates to solve this problem (I am not sure if this can solve or not)?

Improve this question
6
  • Did you include the dependencies in the JAR? Commented Mar 13, 2018 at 19:53
  • @jrtapsell I added this jar file "kafka-clients-0.10.0.0.jar" from the Libraries tab under the project structure. And it is exist in pom.xml. Is it enough? Commented Mar 13, 2018 at 19:57
  • It needs to be in the classpath (classpath issues are common in plugins), if the classes are in the JAR then they will be found, otherwise, the plugin needs to add the dependency JAR manually. Commented Mar 13, 2018 at 19:59
  • @jrtapsell Yes, Producer and consumer class is in the jar file that IntelliJ creates. I do not know the classpath thing, how can I solve this? Thanks for the answer. Commented Mar 13, 2018 at 20:03
  • You need to provide the kafka-clients-0.10.0.0.jar file in your plugin, at the moment it is available at compile time, so no compile errors, then disappears at runtime, hence your error Commented Mar 13, 2018 at 20:10

3 Answers 3

Reset to default
1

I've found an easier solution. I have changed

kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");

to this

kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

And afterwards my code run, also as part of a pre-compiled plug-in.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

What the problem is

At compile time kafka-clients-0.10.0.0.jar is available, so the code compiles successfully, but at runtime it is missing, hence the error you get.

How to fix it

You have 2 options:

  • Include the kafka JAR inside your JAR
  • Add the kafka JAR to your plugin's classpath
Improve this answer

7 Comments

I added this jar file to the resources file in the maven project, now when I buid and create a jar file for the plugin, kafka-clients jar appears in plugin's jar file. But again I got the same error.
You need to include the classes in the JAR, not the whole JAR, this should help
I did exact same steps, but It did not worked again. Is there any way to add kafka jar to classpath?
Can you confirm that the Kafka classes are mixed in to your JAR file, rather than staying in their own one
No they are not mixed. @jrtapsell
|
0

Also, make sure you don't have a dependency for kafka-clients marked with provided scope.

Improve this answer

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.