Skip to content

Commit 0ace692

Browse files
kaiwaehnerkaiwaehner
committed
Cleaning and correct step-by-step guide, but still missing fixes
1 parent 1d28d4d commit 0ace692

File tree

1 file changed

+27
-135
lines changed

1 file changed

+27
-135
lines changed

‎live-demo-kafka-connect-iot-mqtt-connector.adoc‎

Lines changed: 27 additions & 135 deletions
Original file line numberDiff line numberDiff line change
@@ -3,36 +3,26 @@
33
Kai Waehner <kontakt@kai-waehner.de>
44
04 Sept 2018
55

6+
This script assumes that all components (Kafka, Connect, MQTT Broker) use default values.
67

7-
= TODO UPDATE THIS DOCUMENT
8+
== Configuration
89

9-
== Deployment of UDF
10+
Configure MQTT Connector
1011

1112
[source,bash]
1213
----
13-
mvn clean package
14+
TODO
1415
----
1516

16-
Copy `ksql-udf-deep-learning-mqtt-iot-1.0-jar-with-dependencies.jar` from the `target` folder to the `ext` folder of your KSQL installation (you will need to create the `ext` folder).
17-
18-
So if your Confluent Platform installation is at `/Users/kai.waehner/confluent-5.0.0` then copy the JAR to `/Users/kai.waehner/confluent-5.0.0/etc/ksql/ext`
19-
20-
Set `ksql.extension.dir` in `etc/ksql/ksql-server.properties`:
21-
22-
[source,bash]
23-
----
24-
ksql.extension.dir=/Users/kai.waehner/confluent-5.0.0/etc/ksql/ext
25-
----
26-
27-
== Starting services and configuration
17+
== Starting backend services
2818

2919
_Make sure to have Confluent folder on PATH. Otherwise, go to `$CONFLUENT_INSTALL/bin` to execute commands these commands._
3020

31-
Start the KSQL server and dependencies (Kafka, Zookeeper, Schema Registry):
21+
Start Kafka Connect and dependencies (Kafka, Zookeeper, Schema Registry):
3222

3323
[source,bash]
3424
----
35-
confluent start ksql-server
25+
confluent start kafka-connect
3626
----
3727

3828
Create a topic for sensor reading:
@@ -42,80 +32,25 @@ Create a topic for sensor reading:
4232
kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic temperature
4333
----
4434

45-
Now configure the MQTT proxy, by creating a file called `kafka-mqtt-quickstart.properties`:
46-
47-
[source,bash]
48-
----
49-
topic.regex.list=temperature:.*temperature
50-
listeners=0.0.0.0:1883
51-
bootstrap.servers=PLAINTEXT://localhost:9092
52-
confluent.topic.replication.factor=1
53-
----
54-
55-
Launch the MQTT proxy:
56-
57-
[source,bash]
58-
----
59-
kafka-mqtt-start kafka-mqtt-quickstart.properties
60-
----
61-
62-
63-
You don't need to start the MQTT Broker (in this example the Mosquitto server)! You just use Mosquitto later to produce MQTT messages via `mosquitto_pub` command. If you (want to) start the Mosquitto server, you might get a port issue with MQTT Proxy, in that case change the default port of Mosquitto server.
64-
65-
To run `mosquitto_pub` on the Mac, install it first with Homebrew: `brew install mosquitto`. You'll need this if you want to use the `sensor_generator.sh` script later on.
66-
67-
== KSQL and Predictions
68-
69-
Start KSQL CLI:
35+
Start the MQTT Broker:
7036

7137
[source,bash]
7238
----
73-
ksql http://localhost:8088
74-
----
75-
76-
Confirm that the UDF has been successfully registered (check the KSQL server log if not):
77-
78-
[source,sql]
79-
----
80-
ksql> LIST FUNCTIONS;
81-
82-
Function Name | Type
83-
-------------------------------------
84-
ABS | SCALAR
85-
ANOMALY | SCALAR
86-
[...]
87-
----
88-
89-
Register the sensor topic's schema with KSQL:
90-
91-
[source,sql]
92-
----
93-
CREATE STREAM carsensor (eventid integer, sensorinput varchar) WITH (kafka_topic='temperature', value_format='DELIMITED');
39+
todo
9440
----
9541

96-
Set a continuous query running in KSQL:
97-
98-
[source,sql]
99-
----
100-
SELECT EVENTID, ANOMALY(SENSORINPUT) FROM CARSENSOR;
101-
----
102-
103-
Send a sample message to the Kafka topic, from a separate shell prompt (i.e. not KSQL):
104-
105-
[source,bash]
106-
----
107-
echo -e "99999,2.10# 2.13# 2.19# 2.28# 2.44# 2.62# 2.80# 3.04# 3.36# 3.69# 3.97# 4.24# 4.53#4.80# 5.02# 5.21# 5.40# 5.57# 5.71# 5.79# 5.86# 5.92# 5.98# 6.02# 6.06# 6.08# 6.14# 6.18# 6.22# 6.27#6.32# 6.35# 6.38# 6.45# 6.49# 6.53# 6.57# 6.64# 6.70# 6.73# 6.78# 6.83# 6.88# 6.92# 6.94# 6.98# 7.01#7.03# 7.05# 7.06# 7.07# 7.08# 7.06# 7.04# 7.03# 6.99# 6.94# 6.88# 6.83# 6.77# 6.69# 6.60# 6.53# 6.45#6.36# 6.27# 6.19# 6.11# 6.03# 5.94# 5.88# 5.81# 5.75# 5.68# 5.62# 5.61# 5.54# 5.49# 5.45# 5.42# 5.38#5.34# 5.31# 5.30# 5.29# 5.26# 5.23# 5.23# 5.22# 5.20# 5.19# 5.18# 5.19# 5.17# 5.15# 5.14# 5.17# 5.16#5.15# 5.15# 5.15# 5.14# 5.14# 5.14# 5.15# 5.14# 5.14# 5.13# 5.15# 5.15# 5.15# 5.14# 5.16# 5.15# 5.15#5.14# 5.14# 5.15# 5.15# 5.14# 5.13# 5.14# 5.14# 5.11# 5.12# 5.12# 5.12# 5.09# 5.09# 5.09# 5.10# 5.08# 5.08# 5.08# 5.08# 5.06# 5.05# 5.06# 5.07# 5.05# 5.03# 5.03# 5.04# 5.03# 5.01# 5.01# 5.02# 5.01# 5.01#5.00# 5.00# 5.02# 5.01# 4.98# 5.00# 5.00# 5.00# 4.99# 5.00# 5.01# 5.02# 5.01# 5.03# 5.03# 5.02# 5.02#5.04# 5.04# 5.04# 5.02# 5.02# 5.01# 4.99# 4.98# 4.96# 4.96# 4.96# 4.94# 4.93# 4.93# 4.93# 4.93# 4.93# 5.02# 5.27# 5.80# 5.94# 5.58# 5.39# 5.32# 5.25# 5.21# 5.13# 4.97# 4.71# 4.39# 4.05# 3.69# 3.32# 3.05#2.99# 2.74# 2.61# 2.47# 2.35# 2.26# 2.20# 2.15# 2.10# 2.08" | kafkacat -b localhost:9092 -P -t temperature
108-
----
42+
TODO Use other Confluent-CLI commands like
43+
confluent log connect
44+
confluent current
45+
cd ‘temp log directory’
46+
less connect/connect.stderr
47+
confluent list connectors
48+
ls /Users/kai.waehner/confluent-5.0.0/etc
10949

110-
In KSQL you should see the message displayed with the UDF output:
11150

112-
[source,sql]
113-
----
114-
ksql> SELECT EVENTID, ANOMALY(SENSORINPUT) FROM CARSENSOR;
115-
99999 | 1.2104138026620321
116-
----
51+
== Starting Client services
11752

118-
Send a message via MQTT, using `mosquitto_pub` and the MQTT proxy:
53+
Send a MQTT message to the MQTT Broker, using `mosquitto_pub`:
11954

12055
[source,bash]
12156
----
@@ -124,58 +59,7 @@ mosquitto_pub -h 0.0.0.0 -p 1883 -t car/engine/temperature -q 2 -m "99999,2.10#
12459

12560
You should see KSQL show another result from this new message.
12661

127-
Now run a script to generate a stream of MQTT messages:
128-
129-
[source,bash]
130-
----
131-
./sensor_generator.sh
132-
----
133-
134-
You should see in KSQL more messages being printed to the console:
135-
136-
[source,sql]
137-
----
138-
[...]
139-
99999 | 4.03436391020442
140-
99999 | 4.821044621825
141-
99999 | 7.466964581447454
142-
99999 | 4.47345489707657
143-
99999 | 4.388292923118983
144-
[...]
145-
----
146-
147-
Now persist the results of the UDF applied to the data, into a new Kafka topic:
148-
149-
[source,sql]
150-
----
151-
CREATE STREAM ANOMALYDETECTION AS \
152-
SELECT EVENTID, CAST (ANOMALY(SENSORINPUT) AS DOUBLE) AS ANOMALY_VAL \
153-
FROM CARSENSOR;
154-
----
155-
156-
From this new stream, create a derived stream that will include only events breaching a given threshold:
157-
158-
[source,sql]
159-
----
160-
CREATE STREAM ANOMALYDETECTIONWITHFILTER AS \
161-
SELECT EVENTID, ANOMALY_VAL \
162-
FROM ANOMALYDETECTION \
163-
WHERE ANOMALY_VAL > 4;
164-
----
165-
166-
Now you have a KSQL Stream showing breaches, but also a Kafka topic, of which you can see the contents using KSQL's `PRINT` command:
167-
168-
[source,sql]
169-
----
170-
ksql> PRINT ANOMALYDETECTIONWITHFILTER FROM BEGINNING;
171-
Format:STRING
172-
21/08/18 12:36:09 BST , car/engine/temperature , 99999,4.193955593608823
173-
21/08/18 12:36:12 BST , car/engine/temperature , 99999,5.363750640274894
174-
21/08/18 12:36:20 BST , car/engine/temperature , 99999,7.292092517069437
175-
21/08/18 12:36:23 BST , car/engine/temperature , 99999,5.230135737069109
176-
----
177-
178-
or using a CLI tool such as `kafka-console-consumer` from the shell prompt:
62+
Use a CLI tool such as `kafka-console-consumer` from the shell prompt:
17963

18064
[source,bash]
18165
----
@@ -186,9 +70,17 @@ $ kafka-console-consumer --bootstrap-server localhost:9092 --topic ANOMALYDETECT
18670
[...]
18771
----
18872

73+
Now run a script to generate a stream of MQTT messages:
74+
75+
[source,bash]
76+
----
77+
./sensor_generator.sh
78+
----
18979

19080
== Stop services and destroy test data
19181

82+
Stop the sensor_generator script with Control-C. Then stop the backend services:
83+
19284
[source,bash]
19385
----
19486
brew services stop mosquitto

0 commit comments

Comments
 (0)