Patrick McFadin, profile picture
Uploaded byPatrick McFadin
19,632 views

Analyzing Time Series Data with Apache Spark and Cassandra

The document discusses the use of Apache Cassandra and Spark for analyzing time series data, specifically using a weather station example to demonstrate Cassandra's storage capabilities and Spark's processing speed. It outlines table structures for raw weather data and daily aggregates, along with various query capabilities supported by Cassandra and Spark's seamless integration. Additionally, it highlights Spark's advantages in terms of speed, flexibility, and fault tolerance in handling big data analysis.

Embed presentation

Downloaded 625 times
Analyzing Time Series Data with Apache Spark and Cassandra 1 Patrick McFadin
 Chief Evangelist for Apache Cassandra, DataStax @PatrickMcFadin
Apache Cassandra
Cassandra is… • Shared nothing • Masterless peer-to-peer • Great scaling story • Resilient to failure
Cassandra for Applications APACHE CASSANDRA
Example: Weather Station • Weather station collects data • Cassandra stores in sequence • Application reads in sequence
Queries supported CREATE TABLE raw_weather_data (
 wsid text,
 year int,
 month int,
 day int,
 hour int,
 temperature double,
 dewpoint double,
 pressure double,
 wind_direction int,
 wind_speed double,
 sky_condition int,
 sky_condition_text text,
 one_hour_precip double,
 six_hour_precip double,
 PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); Get weather data given •Weather Station ID •Weather Station ID and Time •Weather Station ID and Range of Time
Aggregation Queries CREATE TABLE daily_aggregate_temperature (
 wsid text,
 year int,
 month int,
 day int,
 high double,
 low double,
 mean double,
 variance double,
 stdev double,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); Get temperature stats given •Weather Station ID •Weather Station ID and Time •Weather Station ID and Range of Time Windsor California July 1, 2014 High: 73.4 Low : 51.4
Apache Spark
Map Reduce Input Data Map Reduce Intermediate Data Output Data
Apache Spark • 10x faster on disk,100x faster in memory than Hadoop MR • Works out of the box on EMR • Fault Tolerant Distributed Datasets • Batch, iterative and streaming analysis • In Memory Storage and Disk • Integrates with Most File and Storage Options Up to 100× faster (2-10× on disk) 2-5× less code
Spark Components Spark Core Spark SQL structured Spark Streaming real-time MLlib machine learning GraphX graph
The DAG Step 1 Step 2 Step 3 Step 4 Step 5
org.apache.spark.rdd.RDD Resilient Distributed Dataset (RDD) •Created through transformations on data (map,filter..) or other RDDs •Immutable •Partitioned •Reusable
RDD Operations •Transformations - Similar to scala collections API •Produce new RDDs •filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract •Actions •Require materialization of the records to generate a value •collect: Array[T], count, fold, reduce..
Analytic Analytic Search Transformation Action RDD Operations
Cassandra and Spark
Great combo Store a ton of data Analyze a ton of data
Great combo Spark Streaming Near Real-time SparkSQL Structured Data MLLib Machine Learning GraphX Graph Analysis
Great combo Spark Streaming Near Real-time SparkSQL Structured Data MLLib Machine Learning GraphX Graph Analysis CREATE TABLE raw_weather_data ( wsid text, year int, month int, day int, hour int, temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, sky_condition int, sky_condition_text text, one_hour_precip double, six_hour_precip double, PRIMARY KEY ((wsid), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); Spark Connector
Executer Master Worker Executer Executer Server
Master Worker Worker Worker Worker 0-24Token Ranges 0-100 25-49 50-74 75-99 I will only analyze 25% of the data.
Master Worker Worker Worker Worker 0-24 25-49 50-74 75-9975-99 0-24 25-49 50-74 AnalyticsTransactional
Executer Master Worker Executer Executer 75-99 SELECT * FROM keyspace.table WHERE token(pk) > 75 AND token(pk) <= 99 Spark RDD Spark Partition Spark Partition Spark Partition Spark Connector
Executer Master Worker Executer Executer 75-99 Spark RDD Spark Partition Spark Partition Spark Partition
Spark Reads on Cassandra Awesome animation by DataStax’s own Russell Spitzer
Spark RDDs Represent a Large Amount of Data Partitioned into Chunks RDD 1 2 3 4 5 6 7 8 9Node 2 Node 1 Node 3 Node 4
Node 2 Node 1 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks RDD 2 346 7 8 9 Node 3 Node 4 1 5
Node 2 Node 1 RDD 2 346 7 8 9 Node 3 Node 4 1 5 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks
Cassandra Data is Distributed By Token Range
Cassandra Data is Distributed By Token Range 0 500
Cassandra Data is Distributed By Token Range 0 500 999
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 Without vnodes
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 With vnodes
Node 1 120-220 300-500 780-830 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions
Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 1 780-830
1 Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-400 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500 3
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
421 Node 1 spark.cassandra.input.split.size 50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 3
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50780-830 Node 1
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
Spark Connector Cassandra Cassandra + Spark Joins and Unions No Yes Transformations Limited Yes Outside Data Integration No Yes Aggregations Limited Yes
Type mapping CQL Type Scala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
Attaching to Spark and Cassandra // Import Cassandra-specific functions on SparkContext and RDD objects import org.apache.spark.{SparkContext, SparkConf}
 import com.datastax.spark.connector._ /** The setMaster("local") lets us run & test the job right in our IDE */
 val conf = new SparkConf(true) .set("spark.cassandra.connection.host", "127.0.0.1") .setMaster(“local[*]") .setAppName(getClass.getName) // Optionally
 .set("cassandra.username", "cassandra")
 .set("cassandra.password", “cassandra") 
 val sc = new SparkContext(conf)
Weather station example CREATE TABLE raw_weather_data (
 wsid text, 
 year int, 
 month int, 
 day int, 
 hour int, 
 temperature double, 
 dewpoint double, 
 pressure double, 
 wind_direction int, 
 wind_speed double, 
 sky_condition int, 
 sky_condition_text text, 
 one_hour_precip double, 
 six_hour_precip double, 
 PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);
Simple example /** keyspace & table */
 val tableRDD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 
 
 /** get a simple count of all the rows in the raw_weather_data table */
 val rowCount = tableRDD.count()
 
 
 println(s"Total Rows in Raw Weather Table: $rowCount")
 sc.stop()
Simple example /** keyspace & table */
 val tableRDD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 
 
 /** get a simple count of all the rows in the raw_weather_data table */
 val rowCount = tableRDD.count()
 
 
 println(s"Total Rows in Raw Weather Table: $rowCount")
 sc.stop() Executer SELECT * FROM isd_weather_data.raw_weather_data Spark RDD Spark Partition Spark Connector
Using CQL SELECT temperature
 FROM raw_weather_data
 WHERE wsid = '724940:23234'
 AND year = 2008
 AND month = 12
 AND day = 1; val cqlRRD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 .select("temperature")
 .where("wsid = ? AND year = ? AND month = ? AND DAY = ?",
 "724940:23234", "2008", "12", “1")
Using SQL SELECT wsid, year, month, day, max(temperature) high
 FROM raw_weather_data
 GROUP BY wsid, year, month, day; Wait wut? SELECT w.name, w.lat, w.long
 FROM raw_weather_data r,
 JOIN weather_station w
 ON w.id = r.wsid
 GROUP BY r.wsid;
Python! from pyspark_cassandra import CassandraSparkContext, Row
 from pyspark import SparkContext, SparkConf
 from pyspark.sql import SQLContext # needed for toDF()
 
 conf = SparkConf() 
 .setAppName("Weather App") 
 .setMaster("spark://127.0.0.1:7077") 
 .set("spark.cassandra.connection.host", "127.0.0.1")
 
 sc = CassandraSparkContext(conf=conf)
 sql = SQLContext(sc)
 // Count unique weather stations
 temps = sc.cassandraTable("isd_weather_data", "raw_weather_data").toDF()
 food_count = temps.select("temperature").groupBy("wsid").count()
Weather Station Analysis • Weather station collects data • Cassandra stores in sequence • Spark rolls up data into new tables Windsor California July 1, 2014 High: 73.4 Low : 51.4
Roll-up table(SparkSQL example) • Weather Station Id and Date are unique • High and low temp for each day spark-sql> INSERT INTO TABLE > daily_aggregate_temperature > SELECT wsid, year, month, day, max(temperature) high, min(temperature) low > FROM raw_weather_data > GROUP BY wsid, year, month, day; OK Time taken: 2.345 seconds aggregations CREATE TABLE daily_aggregate_temperature (
 wsid text,
 year int,
 month int,
 day int,
 high double,
 low double,
 mean double,
 variance double,
 stdev double,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC);
What just happened • Data is read from temperature table • Transformed • Inserted into the daily_high_low table Table: temperature Table: daily_high_low Read data from table Transform Insert data into table
Spark Streaming
zillions of bytes gigabytes per second Spark Versus Spark Streaming
Analytic Analytic Search Spark Streaming Kinesis,'S3'
DStream - Micro Batches μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD) Processing of DStream = Processing of μBatches, RDDs DStream • Continuous sequence of micro batches • More complex processing models are possible with less effort • Streaming computations as a series of deterministic batch computations on small time intervals
Now what? Cassandra Only DC Cassandra + Spark DC Spark Jobs Spark Streaming
You can do this at home! https://github.com/killrweather/killrweather
Thank you! Bring the questions Follow me on twitter @PatrickMcFadin

More Related Content

PPTX
Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
byStampedeCon
 
PPTX
Real time data pipeline with spark streaming and cassandra with mesos
byRahul Kumar
 
PDF
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
byDataStax Academy
 
PDF
Apache Cassandra and Python for Analyzing Streaming Big Data
byprajods
 
PDF
Cassandra spark connector
byDuyhai Doan
 
PDF
Apache cassandra & apache spark for time series data
byPatrick McFadin
 
PDF
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
PDF
OLAP with Cassandra and Spark
byEvan Chan
 
Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
byStampedeCon
 
Real time data pipeline with spark streaming and cassandra with mesos
byRahul Kumar
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
byDataStax Academy
 
Apache Cassandra and Python for Analyzing Streaming Big Data
byprajods
 
Cassandra spark connector
byDuyhai Doan
 
Apache cassandra & apache spark for time series data
byPatrick McFadin
 
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
OLAP with Cassandra and Spark
byEvan Chan
 

What's hot

PDF
Analytics with Cassandra & Spark
byMatthias Niehoff
 
PDF
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
byHelena Edelson
 
PDF
Spark with Cassandra by Christopher Batey
bySpark Summit
 
PPTX
Alpine academy apache spark series #1 introduction to cluster computing wit...
byHolden Karau
 
PDF
Streaming Analytics with Spark, Kafka, Cassandra and Akka
byHelena Edelson
 
PPTX
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
PDF
SMACK Stack 1.1
byJoe Stein
 
PDF
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...
bySpark Summit
 
PPTX
Kafka Lambda architecture with mirroring
byAnant Rustagi
 
PDF
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
bySpark Summit
 
PDF
Lambda architecture
bySzilveszter Molnár
 
PDF
Spark And Cassandra: 2 Fast, 2 Furious
byJen Aman
 
PDF
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
byNatalino Busa
 
PDF
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
byLegacy Typesafe (now Lightbend)
 
PDF
Cassandra & Spark for IoT
byMatthias Niehoff
 
PDF
Kafka spark cassandra webinar feb 16 2016
byHiromitsu Komatsu
 
PDF
Spark Streaming with Cassandra
byJacek Lewandowski
 
PDF
Akka in Production - ScalaDays 2015
byEvan Chan
 
PDF
Feeding Cassandra with Spark-Streaming and Kafka
byDataStax Academy
 
PDF
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
byEvan Chan
 
Analytics with Cassandra & Spark
byMatthias Niehoff
 
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
byHelena Edelson
 
Spark with Cassandra by Christopher Batey
bySpark Summit
 
Alpine academy apache spark series #1 introduction to cluster computing wit...
byHolden Karau
 
Streaming Analytics with Spark, Kafka, Cassandra and Akka
byHelena Edelson
 
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
SMACK Stack 1.1
byJoe Stein
 
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...
bySpark Summit
 
Kafka Lambda architecture with mirroring
byAnant Rustagi
 
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
bySpark Summit
 
Lambda architecture
bySzilveszter Molnár
 
Spark And Cassandra: 2 Fast, 2 Furious
byJen Aman
 
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
byNatalino Busa
 
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
byLegacy Typesafe (now Lightbend)
 
Cassandra & Spark for IoT
byMatthias Niehoff
 
Kafka spark cassandra webinar feb 16 2016
byHiromitsu Komatsu
 
Spark Streaming with Cassandra
byJacek Lewandowski
 
Akka in Production - ScalaDays 2015
byEvan Chan
 
Feeding Cassandra with Spark-Streaming and Kafka
byDataStax Academy
 
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
byEvan Chan
 

Similar to Analyzing Time Series Data with Apache Spark and Cassandra

PDF
Cassandra and Spark
bydatastaxjp
 
PDF
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
PDF
Nike Tech Talk: Double Down on Apache Cassandra and Spark
byPatrick McFadin
 
PDF
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
PDF
Spark Cassandra Connector Dataframes
byRussell Spitzer
 
PPTX
5 Ways to Use Spark to Enrich your Cassandra Environment
byJim Hatcher
 
PDF
Cassandra and Spark, closing the gap between no sql and analytics codemotio...
byDuyhai Doan
 
PPTX
Lightning fast analytics with Cassandra and Spark
byVictor Coustenoble
 
PPTX
Spark + Cassandra = Real Time Analytics on Operational Data
byVictor Coustenoble
 
PPTX
BI, Reporting and Analytics on Apache Cassandra
byVictor Coustenoble
 
PDF
Spark and Cassandra 2 Fast 2 Furious
byRussell Spitzer
 
PDF
Cassandra London - C* Spark Connector
byChristopher Batey
 
PDF
Apache cassandra and spark. you got the the lighter, let's start the fire
byPatrick McFadin
 
PDF
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
PDF
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
PDF
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
PDF
Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...
byData Con LA
 
PDF
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
PPTX
Lightning Fast Analytics with Cassandra and Spark
byTim Vincent
 
PDF
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
byQbeast
 
Cassandra and Spark
bydatastaxjp
 
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
Nike Tech Talk: Double Down on Apache Cassandra and Spark
byPatrick McFadin
 
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
Spark Cassandra Connector Dataframes
byRussell Spitzer
 
5 Ways to Use Spark to Enrich your Cassandra Environment
byJim Hatcher
 
Cassandra and Spark, closing the gap between no sql and analytics codemotio...
byDuyhai Doan
 
Lightning fast analytics with Cassandra and Spark
byVictor Coustenoble
 
Spark + Cassandra = Real Time Analytics on Operational Data
byVictor Coustenoble
 
BI, Reporting and Analytics on Apache Cassandra
byVictor Coustenoble
 
Spark and Cassandra 2 Fast 2 Furious
byRussell Spitzer
 
Cassandra London - C* Spark Connector
byChristopher Batey
 
Apache cassandra and spark. you got the the lighter, let's start the fire
byPatrick McFadin
 
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...
byData Con LA
 
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
Lightning Fast Analytics with Cassandra and Spark
byTim Vincent
 
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
byQbeast
 

More from Patrick McFadin

PDF
Advanced data modeling with apache cassandra
byPatrick McFadin
 
PDF
Cassandra 3.0 advanced preview
byPatrick McFadin
 
PDF
Building Antifragile Applications with Apache Cassandra
byPatrick McFadin
 
PDF
Storing time series data with Apache Cassandra
byPatrick McFadin
 
PDF
Time series with Apache Cassandra - Long version
byPatrick McFadin
 
PDF
Laying down the smack on your data pipelines
byPatrick McFadin
 
PDF
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
PDF
Successful Architectures for Fast Data
byPatrick McFadin
 
PDF
Introduction to cassandra 2014
byPatrick McFadin
 
PDF
Time series with apache cassandra strata
byPatrick McFadin
 
PDF
Cassandra 2.0 and timeseries
byPatrick McFadin
 
PDF
An Introduction to time series with Team Apache
byPatrick McFadin
 
PDF
Real data models of silicon valley
byPatrick McFadin
 
PDF
The world's next top data model
byPatrick McFadin
 
PDF
Cassandra EU - Data model on fire
byPatrick McFadin
 
PDF
Cassandra 2.0 better, faster, stronger
byPatrick McFadin
 
PDF
Cassandra at scale
byPatrick McFadin
 
PDF
Making money with open source and not losing your soul: A practical guide
byPatrick McFadin
 
PDF
Help! I want to contribute to an Open Source project but my boss says no.
byPatrick McFadin
 
PDF
Open source or proprietary, choose wisely!
byPatrick McFadin
 
Advanced data modeling with apache cassandra
byPatrick McFadin
 
Cassandra 3.0 advanced preview
byPatrick McFadin
 
Building Antifragile Applications with Apache Cassandra
byPatrick McFadin
 
Storing time series data with Apache Cassandra
byPatrick McFadin
 
Time series with Apache Cassandra - Long version
byPatrick McFadin
 
Laying down the smack on your data pipelines
byPatrick McFadin
 
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
Successful Architectures for Fast Data
byPatrick McFadin
 
Introduction to cassandra 2014
byPatrick McFadin
 
Time series with apache cassandra strata
byPatrick McFadin
 
Cassandra 2.0 and timeseries
byPatrick McFadin
 
An Introduction to time series with Team Apache
byPatrick McFadin
 
Real data models of silicon valley
byPatrick McFadin
 
The world's next top data model
byPatrick McFadin
 
Cassandra EU - Data model on fire
byPatrick McFadin
 
Cassandra 2.0 better, faster, stronger
byPatrick McFadin
 
Cassandra at scale
byPatrick McFadin
 
Making money with open source and not losing your soul: A practical guide
byPatrick McFadin
 
Help! I want to contribute to an Open Source project but my boss says no.
byPatrick McFadin
 
Open source or proprietary, choose wisely!
byPatrick McFadin
 

Recently uploaded

PDF
Cloud Infrastructure monitoring with Grafana Cloud
byImma Valls Bernaus
 
PDF
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
PDF
Parallel Computing BCS702 Module notes of the vtu college 7th sem 4.pdf
byMatheshVishnu
 
PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ２「AI Computing Broker: Make your GPUs ...
byPC Cluster Consortium
 
PDF
Agentic Intro and Hands-on: Build your first Coded Agent
byUiPathCommunity
 
PPTX
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ３「FUJITSU-MONAKA series: Arm-based pro...
byPC Cluster Consortium
 
PDF
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
PPTX
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 
PDF
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
PDF
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
PDF
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
PDF
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
PDF
Lets Build a Serverless Function with Kiro
byChris Bingham
 
PDF
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 
PDF
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
PDF
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
PDF
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
PDF
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 
PDF
Cybersecurity Prevention and Detection: Unit 2
byVICTOR MAESTRE RAMIREZ
 
Cloud Infrastructure monitoring with Grafana Cloud
byImma Valls Bernaus
 
[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...
bywintari3
 
Parallel Computing BCS702 Module notes of the vtu college 7th sem 4.pdf
byMatheshVishnu
 
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ２「AI Computing Broker: Make your GPUs ...
byPC Cluster Consortium
 
Agentic Intro and Hands-on: Build your first Coded Agent
byUiPathCommunity
 
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ３「FUJITSU-MONAKA series: Arm-based pro...
byPC Cluster Consortium
 
[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...
byWintari Yasmin
 
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
Lets Build a Serverless Function with Kiro
byChris Bingham
 
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 
[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...
byWintari Yasmin
 
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 
Cybersecurity Prevention and Detection: Unit 2
byVICTOR MAESTRE RAMIREZ
 

Analyzing Time Series Data with Apache Spark and Cassandra

  • 1.
    Analyzing Time SeriesData with Apache Spark and Cassandra 1 Patrick McFadin
 Chief Evangelist for Apache Cassandra, DataStax @PatrickMcFadin
  • 2.
  • 3.
    Cassandra is… • Sharednothing • Masterless peer-to-peer • Great scaling story • Resilient to failure
  • 4.
  • 5.
    Example: Weather Station •Weather station collects data • Cassandra stores in sequence • Application reads in sequence
  • 6.
    Queries supported CREATE TABLEraw_weather_data (
 wsid text,
 year int,
 month int,
 day int,
 hour int,
 temperature double,
 dewpoint double,
 pressure double,
 wind_direction int,
 wind_speed double,
 sky_condition int,
 sky_condition_text text,
 one_hour_precip double,
 six_hour_precip double,
 PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); Get weather data given •Weather Station ID •Weather Station ID and Time •Weather Station ID and Range of Time
  • 7.
    Aggregation Queries CREATE TABLEdaily_aggregate_temperature (
 wsid text,
 year int,
 month int,
 day int,
 high double,
 low double,
 mean double,
 variance double,
 stdev double,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); Get temperature stats given •Weather Station ID •Weather Station ID and Time •Weather Station ID and Range of Time Windsor California July 1, 2014 High: 73.4 Low : 51.4
  • 8.
  • 9.
  • 10.
    Apache Spark • 10xfaster on disk,100x faster in memory than Hadoop MR • Works out of the box on EMR • Fault Tolerant Distributed Datasets • Batch, iterative and streaming analysis • In Memory Storage and Disk • Integrates with Most File and Storage Options Up to 100× faster (2-10× on disk) 2-5× less code
  • 11.
    Spark Components Spark Core SparkSQL structured Spark Streaming real-time MLlib machine learning GraphX graph
  • 13.
    The DAG Step 1 Step2 Step 3 Step 4 Step 5
  • 14.
    org.apache.spark.rdd.RDD Resilient Distributed Dataset(RDD) •Created through transformations on data (map,filter..) or other RDDs •Immutable •Partitioned •Reusable
  • 15.
    RDD Operations •Transformations -Similar to scala collections API •Produce new RDDs •filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract •Actions •Require materialization of the records to generate a value •collect: Array[T], count, fold, reduce..
  • 16.
  • 17.
  • 18.
    Great combo Store aton of data Analyze a ton of data
  • 19.
    Great combo Spark Streaming NearReal-time SparkSQL Structured Data MLLib Machine Learning GraphX Graph Analysis
  • 20.
    Great combo Spark Streaming NearReal-time SparkSQL Structured Data MLLib Machine Learning GraphX Graph Analysis CREATE TABLE raw_weather_data ( wsid text, year int, month int, day int, hour int, temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, sky_condition int, sky_condition_text text, one_hour_precip double, six_hour_precip double, PRIMARY KEY ((wsid), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); Spark Connector
  • 21.
  • 22.
  • 23.
  • 24.
    Executer Master Worker Executer Executer 75-99 SELECT * FROM keyspace.table WHEREtoken(pk) > 75 AND token(pk) <= 99 Spark RDD Spark Partition Spark Partition Spark Partition Spark Connector
  • 25.
  • 26.
    Spark Reads onCassandra Awesome animation by DataStax’s own Russell Spitzer
  • 27.
    Spark RDDs Represent aLarge Amount of Data Partitioned into Chunks RDD 1 2 3 4 5 6 7 8 9Node 2 Node 1 Node 3 Node 4
  • 28.
    Node 2 Node 1 SparkRDDs Represent a Large Amount of Data Partitioned into Chunks RDD 2 346 7 8 9 Node 3 Node 4 1 5
  • 29.
    Node 2 Node 1 RDD 2 346 78 9 Node 3 Node 4 1 5 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks
  • 30.
    Cassandra Data isDistributed By Token Range
  • 31.
    Cassandra Data isDistributed By Token Range 0 500
  • 32.
    Cassandra Data isDistributed By Token Range 0 500 999
  • 33.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4
  • 34.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 Without vnodes
  • 35.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 With vnodes
  • 36.
    Node 1 120-220 300-500 780-830 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions
  • 37.
    Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 1 780-830
  • 38.
    1 Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 39.
    2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 40.
    2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size50 Reported  density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 41.
    2 1 Node 1 300-400 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
  • 42.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
  • 43.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500 3
  • 44.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3 400-500
  • 45.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 46.
    4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 47.
    4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 48.
    421 Node 1 spark.cassandra.input.split.size 50 Reported density  is  0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 3
  • 49.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50780-830 Node 1
  • 50.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
  • 51.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
  • 52.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
  • 53.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
  • 54.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows 50 CQL Rows
  • 55.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows
  • 56.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
  • 57.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
  • 58.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 59.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 60.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 61.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 62.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 63.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 64.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 65.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 66.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 67.
    Spark Connector Cassandra Cassandra + Spark Joinsand Unions No Yes Transformations Limited Yes Outside Data Integration No Yes Aggregations Limited Yes
  • 68.
    Type mapping CQL TypeScala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
  • 69.
    Attaching to Sparkand Cassandra // Import Cassandra-specific functions on SparkContext and RDD objects import org.apache.spark.{SparkContext, SparkConf}
 import com.datastax.spark.connector._ /** The setMaster("local") lets us run & test the job right in our IDE */
 val conf = new SparkConf(true) .set("spark.cassandra.connection.host", "127.0.0.1") .setMaster(“local[*]") .setAppName(getClass.getName) // Optionally
 .set("cassandra.username", "cassandra")
 .set("cassandra.password", “cassandra") 
 val sc = new SparkContext(conf)
  • 70.
    Weather station example CREATETABLE raw_weather_data (
 wsid text, 
 year int, 
 month int, 
 day int, 
 hour int, 
 temperature double, 
 dewpoint double, 
 pressure double, 
 wind_direction int, 
 wind_speed double, 
 sky_condition int, 
 sky_condition_text text, 
 one_hour_precip double, 
 six_hour_precip double, 
 PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);
  • 71.
    Simple example /** keyspace& table */
 val tableRDD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 
 
 /** get a simple count of all the rows in the raw_weather_data table */
 val rowCount = tableRDD.count()
 
 
 println(s"Total Rows in Raw Weather Table: $rowCount")
 sc.stop()
  • 72.
    Simple example /** keyspace& table */
 val tableRDD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 
 
 /** get a simple count of all the rows in the raw_weather_data table */
 val rowCount = tableRDD.count()
 
 
 println(s"Total Rows in Raw Weather Table: $rowCount")
 sc.stop() Executer SELECT * FROM isd_weather_data.raw_weather_data Spark RDD Spark Partition Spark Connector
  • 73.
    Using CQL SELECT temperature
 FROMraw_weather_data
 WHERE wsid = '724940:23234'
 AND year = 2008
 AND month = 12
 AND day = 1; val cqlRRD = sc.cassandraTable("isd_weather_data", "raw_weather_data")
 .select("temperature")
 .where("wsid = ? AND year = ? AND month = ? AND DAY = ?",
 "724940:23234", "2008", "12", “1")
  • 74.
    Using SQL SELECT wsid,year, month, day, max(temperature) high
 FROM raw_weather_data
 GROUP BY wsid, year, month, day; Wait wut? SELECT w.name, w.lat, w.long
 FROM raw_weather_data r,
 JOIN weather_station w
 ON w.id = r.wsid
 GROUP BY r.wsid;
  • 75.
    Python! from pyspark_cassandra importCassandraSparkContext, Row
 from pyspark import SparkContext, SparkConf
 from pyspark.sql import SQLContext # needed for toDF()
 
 conf = SparkConf() 
 .setAppName("Weather App") 
 .setMaster("spark://127.0.0.1:7077") 
 .set("spark.cassandra.connection.host", "127.0.0.1")
 
 sc = CassandraSparkContext(conf=conf)
 sql = SQLContext(sc)
 // Count unique weather stations
 temps = sc.cassandraTable("isd_weather_data", "raw_weather_data").toDF()
 food_count = temps.select("temperature").groupBy("wsid").count()
  • 76.
    Weather Station Analysis •Weather station collects data • Cassandra stores in sequence • Spark rolls up data into new tables Windsor California July 1, 2014 High: 73.4 Low : 51.4
  • 77.
    Roll-up table(SparkSQL example) •Weather Station Id and Date are unique • High and low temp for each day spark-sql> INSERT INTO TABLE > daily_aggregate_temperature > SELECT wsid, year, month, day, max(temperature) high, min(temperature) low > FROM raw_weather_data > GROUP BY wsid, year, month, day; OK Time taken: 2.345 seconds aggregations CREATE TABLE daily_aggregate_temperature (
 wsid text,
 year int,
 month int,
 day int,
 high double,
 low double,
 mean double,
 variance double,
 stdev double,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC);
  • 78.
    What just happened •Data is read from temperature table • Transformed • Inserted into the daily_high_low table Table: temperature Table: daily_high_low Read data from table Transform Insert data into table
  • 79.
  • 80.
    zillions of bytesgigabytes per second Spark Versus Spark Streaming
  • 81.
  • 82.
    DStream - MicroBatches μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD) Processing of DStream = Processing of μBatches, RDDs DStream • Continuous sequence of micro batches • More complex processing models are possible with less effort • Streaming computations as a series of deterministic batch computations on small time intervals
  • 83.
    Now what? Cassandra Only DC Cassandra +Spark DC Spark Jobs Spark Streaming
  • 84.
    You can dothis at home! https://github.com/killrweather/killrweather
  • 86.
    Thank you! Bring thequestions Follow me on twitter @PatrickMcFadin