Edureka!, profile picture
Uploaded byEdureka!
2,601 views

Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka

The document provides an extensive overview of Spark SQL, including its advantages over Apache Hive, features, architecture, and libraries. It discusses how Spark SQL improves performance through real-time querying and integration with various data formats, while offering practical use cases like Twitter sentiment analysis and stock market data analysis. Additionally, it outlines the steps for initializing a Spark shell, creating datasets, and executing SQL operations within Spark.

Technology◦
Related topics:
Apache Spark•
In this document
Powered by AI

Introduction to Spark Certification Training including best practices in DevOps culture and an overview of what to expect.

Discusses the limitations of Apache Hive and reasons for Spark SQL's development to facilitate improved performance and capabilities.

Highlights the advantages of Spark SQL over Hive, including faster execution and real-time querying, with use cases like Twitter sentiment analysis.

Describes key features of Spark SQL, including integration with Spark, support for various data formats, and defining user functions.

Outlines the architecture of Spark SQL, including DataFrame DSL and APIs for data processing.

Details the libraries available in Spark SQL, including Data Source API and DataFrame API, and their functionalities.

Demonstrates the process of starting Spark shell, creating datasets, and using JSON files for structured data.

Shows how to add schemas to RDDs, perform transformations, and create DataFrames using structured data.

Explains how to load JSON data, create DataFrames, and manipulate the schema for data analysis.

Illustrates the process of creating and interacting with Hive tables in Spark SQL, including SQL operations.

Introduces a use case for stock market analysis using Spark SQL to process large datasets effectively.Details stock data used for analysis, including information about the companies involved.

Presents techniques for advanced stock analysis including joining stock data, transformations, and calculating averages.

Celebrates the completion of training, demonstrates Spark SQL's power in real-time analytics.

Encourages questions, queries, and feedback from participants.

Embed presentation

www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING 5 Best Practices in DevOps Culture
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING What to expect? Why Spark SQL 1 Use Case 5 Hands-On Examples 4 Spark SQL Features 3 Spark SQL Libraries 2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Why Spark SQL?
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Why Do We Need Spark SQL? Spark SQL was built to overcome the limitations of Apache Hive running on top of Spark. Limitations of Apache Hive Hive uses MapReduce which lags in performance with medium and small sized datasets ( <200 GB) No resume capability Hive cannot drop encrypted databases
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Advantages Over Hive Spark SQL uses the metastore services of Hive to query the data stored and managed by Hive. Advantages How? Faster execution 600 secs 50 secs 1 No migration hurdles 2 Real time querying 3 Batch
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Success Story
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Success Story Twitter Sentiment Analysis With Spark SQL Trending Topics can be used to create campaigns and attract larger audience Sentiment helps in crisis management, service adjusting and target marketing NYSE: Real Time Analysis of Stock Market Data Banking: Credit Card Fraud Detection Genomic Sequencing
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Features SQL Integration With Spark Uniform Data Access Seamless Support Transformations Performance Standard Connectivity User Defined Functions
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Features Spark SQL is used for the structured/semi structured data analysis in Spark. Spark SQL integrates relational processing with Spark’s functional programming.1 2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Features SQL queries can be converted into RDDs for transformations Support for various data formats3 4 RDD 1 RDD 2 Shuffle transform Drop split point Invoking RDD 2 computes all partitions of RDD 1
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING 5 Performance And Scalability Spark SQL Overview
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Features Standard JDBC/ODBC Connectivity6 7 User Defined Functions lets users define new Column-based functions to extend the Spark vocabulary User
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING UDF Example Creating a UDF ‘toUpperCase’ to convert a string to upper case Registering our UDF in the list of functions
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Architecture
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Architecture Architecture Of Spark SQL DataFrame DSL Spark SQL & HQL DataFrame API Data Source API CSV JDBCJSON
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Libraries
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Libraries Spark SQL has the following libraries: 1 Data Source API DataFrame API Interpreter & Optimizer SQL Service 2 3 4
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Data Source API DataFrame API Interpreter & Optimizer SQL Service Data Source API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Data Source API Data Source API is used to read and store structured and semi- structured data into Spark SQL Features:  Structured/ Semi-structured data  Multiple formats  3rd party integration Data Source API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Data Source API DataFrame API Interpreter & Optimizer SQL Service DataFrame API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING DataFrame API DataFrame API converts the data that is read through Data Source API into tabular columns to help perform SQL operations Features:  Distributed collection of data organized into named columns  Equivalent to a relational table in SQL  Lazily evaluated DataFrame API Named Columns Data Source API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Data Source API DataFrame API Interpreter & Optimizer SQL Service SQL Interpreter & Optimizer
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING SQL Interpreter & Optimizer SQL Interpreter & Optimizer handles the functional programming part of Spark SQL. It transforms the DataFrames RDDs to get the required results in the required formats. Features:  Functional programming  Transforming trees  Faster than RDDs  Processes all size data e.g. Catalyst: A modular library for distinct optimization Interpreter & Optimizer Resilient Distributed Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Data Source API DataFrame API Interpreter & Optimizer SQL Service SQL Service
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING SQL Service Spark SQL Service Interpreter & Optimizer Resilient Distributed Dataset  SQL Service is the entry point for working along structured data in Spark  SQL is used to fetch the result from the interpreted & optimized data We have thus used all the four libraries in sequence. This completes a Spark SQL process
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables Querying Using Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables Starting Up Spark Shell
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell - Intialization //We first import a Spark Session into Apache Spark. import org.apache.spark.sql.SparkSession //Creating a Spark Session ‘spark’ using the ‘builder()’ function. val spark = SparkSession.builder().appName("Spark SQL basic example").config("spark.some.config.option", "some-value").getOrCreate() //Importing the Implicts class into our ‘spark’ Session. import spark.implicits._ //We now create a DataFrame ‘df’ and import data from the ’employee.json’ file. val df = spark.read.json("examples/src/main/resources/employee.json") //Displaying the DataFrame ‘df’. The result is a table of ages and names from our ’employee.json’ file. df.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell – Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables Creating Datasets
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Creating Dataset - Case Class & Dataset After understanding DataFrames, let us now move on to Dataset API. The below code creates a Dataset class in SparkSQL. //Creating a class ‘Employee’ to store name and age of an employee. case class Employee(name: String, age: Long) //Assigning a Dataset ‘caseClassDS’ to store the record of Andrew. val caseClassDS = Seq(Employee("Andrew", 55)).toDS() //Displaying the Dataset ‘caseClassDS’. caseClassDS.show() //Creating a primitive Dataset to demonstrate mapping of DataFrames into Datasets. val primitiveDS = Seq(1, 2, 3).toDS() //Assigning the above sequence into an array. primitiveDS.map(_ + 1).collect()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Creating Dataset - Case Class & Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Creating Dataset – Reading File //Setting the path to our JSON file ’employee.json’. val path = "examples/src/main/resources/employee.json" //Creating a Dataset and from the file. val employeeDS = spark.read.json(path).as[Employee] //Displaying the contents of ’employeeDS’ Dataset. employeeDS.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Creating Dataset – Reading File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables Adding Schema To RDDs
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema To RDDs – Initialization //Importing Expression Encoder for RDDs, Encoder library and Implicts class into the shell. import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder import org.apache.spark.sql.Encoder import spark.implicits._ //Creating an ’employeeDF’ DataFrame from ’employee.txt’ and mapping the columns based on delimiter comma ‘,’ into a temporary view ’employee’. val employeeDF = spark.sparkContext.textFile("examples/src/main/resources/employee.txt").map(_.split(",")).ma p(attributes => Employee(attributes(0), attributes(1).trim.toInt)).toDF() //Creating the temporary view ’employee’. employeeDF.createOrReplaceTempView("employee") //Defining a DataFrame ‘youngstersDF’ which will contain all the employees between the ages of 18 and 30. val youngstersDF = spark.sql("SELECT name, age FROM employee WHERE age BETWEEN 18 AND 30") //Mapping the names from the RDD into ‘youngstersDF’ to display the names of youngsters. youngstersDF.map(youngster => "Name: " + youngster(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema To RDDs – Initialization
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema To RDDs - Transformation //Converting the mapped names into string for transformations. youngstersDF.map(youngster => "Name: " + youngster.getAs[String]("name")).show() //Using the mapEncoder from Implicits class to map the names to the ages. implicit val mapEncoder = org.apache.spark.sql.Encoders.kryo[Map[String, Any]] //Mapping the names to the ages of our ‘youngstersDF’ DataFrame. The result is an array with names mapped to their respective ages. youngstersDF.map(youngster => youngster.getValuesMap[Any](List("name", "age"))).collect()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema To RDDs - Transformation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema – Reading File & Adding Schema //Importing the ‘types’ class into the Spark Shell. import org.apache.spark.sql.types._ //Importing ‘Row’ class into the Spark Shell. Row is used in mapping RDD Schema. import org.apache.spark.sql.Row //Creating a RDD ’employeeRDD’ from the text file ’employee.txt’. val employeeRDD = spark.sparkContext.textFile("examples/src/main/resources/employee.txt") //Defining the schema as “name age”. This is used to map the columns of the RDD. val schemaString = "name age" //Defining ‘fields’ RDD which will be the output after mapping the ’employeeRDD’ to the schema ‘schemaString’. val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, nullable = true)) //Obtaining the type of ‘fields’ RDD into ‘schema’. val schema = StructType(fields)
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema – Reading File & Adding Schema
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema – Transformation Result //We now create a RDD called ‘rowRDD’ and transform the ’employeeRDD’ using the ‘map’ function into ‘rowRDD’. val rowRDD = employeeRDD.map(_.split(",")).map(attributes => Row(attributes(0), attributes(1).trim)) //We define a DataFrame ’employeeDF’ and store the RDD schema into it. val employeeDF = spark.createDataFrame(rowRDD, schema) //Creating a temporary view of ’employeeDF’ into ’employee’. employeeDF.createOrReplaceTempView("employee") //Performing the SQL operation on ’employee’ to display the contents of employee. val results = spark.sql("SELECT name FROM employee") //Displaying the names of the previous operation from the ’employee’ view. results.map(attributes => "Name: " + attributes(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Adding Schema – Transformation Result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables JSON Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Data – Loading File //Importing Implicits class into the shell. import spark.implicits._ //Creating an ’employeeDF’ DataFrame from our ’employee.json’ file. val employeeDF = spark.read.json("examples/src/main/resources/employee.json")
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Data – Loading File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Data – Parquet File //Creating a ‘parquetFile’ temporary view of our DataFrame. employeeDF.write.parquet("employee.parquet") val parquetFileDF = spark.read.parquet("employee.parquet") parquetFileDF.createOrReplaceTempView("parquetFile") //Selecting the names of people between the ages of 18 and 30 from our Parquet file. val namesDF = spark.sql("SELECT name FROM parquetFile WHERE age BETWEEN 18 AND 30") //Displaying the result of the Spark SQL operation. namesDF.map(attributes => "Name: " + attributes(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Data – Parquet File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Dataset – Creating DataFrame //Setting to path to our ’employee.json’ file. val path = "examples/src/main/resources/employee.json" //Creating a DataFrame ’employeeDF’ from our JSON file. val employeeDF = spark.read.json(path) //Printing the schema of ’employeeDF’. employeeDF.printSchema() //Creating a temporary view of the DataFrame into ’employee’. employeeDF.createOrReplaceTempView("employee") //Defining a DataFrame ‘youngsterNamesDF’ which stores the names of all the employees between the ages of 18 and 30 present in ’employee’. val youngsterNamesDF = spark.sql("SELECT name FROM employee WHERE age BETWEEN 18 AND 30") //Displaying the contents of our DataFrame. youngsterNamesDF.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Dataset – Creating DataFrame
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Dataset – RDD Operation //Creating a RDD ‘otherEmployeeRDD’ which will store the content of employee George from New Delhi, Delhi. val otherEmployeeRDD = spark.sparkContext.makeRDD("""{"name":"George","address":{" city":"New Delhi","state":"Delhi"}}""" :: Nil) //Assigning the contents of ‘otherEmployeeRDD’ into ‘otherEmployee’. val otherEmployee = spark.read.json(otherEmployeeRDD) //Displaying the contents of ‘otherEmployee’. otherEmployee.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING JSON Dataset – RDD Operation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Starting Up Spark Shell Creating Dataset Adding Schema To RDD JSON Dataset Hive Tables Hive Tables
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – Case Class & Spark Session //Importing ‘Row’ class and Spark Session into the Spark Shell. import org.apache.spark.sql.Row import org.apache.spark.sql.SparkSession //Creating a class ‘Record’ with attributes Int and String. case class Record(key: Int, value: String) //Setting the location of ‘warehouseLocation’ to Spark warehouse. val warehouseLocation = "spark-warehouse" //We now build a Spark Session ‘spark’ to demonstrate Hive example in Spark SQL. val spark = SparkSession.builder().appName("Spark Hive Example").config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate() //Importing Implicits class and SQL library into the shell. import spark.implicits._ import spark.sql //Creating a table ‘src’ with columns to store key and value. sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – Case Class & Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – SQL Operation //We now load the data from the examples present in Spark directory into our table ‘src’. sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src") //The contents of ‘src’ is displayed below. sql("SELECT * FROM src").show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – SQL Operation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – SQL & DataFrame Transformation //We perform the ‘count’ operation to select the number of keys in ‘src’ table. sql("SELECT COUNT(*) FROM src").show() //We now select all the records with ‘key’ value less than 10 and store it in the ‘sqlDF’ DataFrame. val sqlDF = sql("SELECT key, value FROM src WHERE key < 10 ORDER BY key") //Creating a Dataset ‘stringDS’ from ‘sqlDF’. val stringsDS = sqlDF.map {case Row(key: Int, value: String) => s"Key: $key, Value: $value"} //Displaying the contents of ‘stringDS’ Dataset. stringsDS.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables – SQL & DataFrame Transformation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables - Result //We create a DataFrame ‘recordsDF’ and store all the records with key values 1 to 100. val recordsDF = spark.createDataFrame((1 to 100).map(i => Record(i, s"val_$i"))) //Create a temporary view ‘records’ of ‘recordsDF’ DataFrame. recordsDF.createOrReplaceTempView("records") //Displaying the contents of the join of tables ‘records’ and ‘src’ with ‘key’ as the primary key. sql("SELECT * FROM records r JOIN src s ON r.key = s.key").show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Hive Tables - Result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Stock Market Analysis With Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Problem Statement Computations to be done:  Compute the average closing price  List the companies with highest closing prices  Compute average closing price per month  List the number of big price rises and falls  Compute Statistical correlation We will use Spark SQL to retrieve trends in the stock market data and thus establish a financial strategy to avoid risky investment Stock Market trading generates huge real time data. Analysis of this data is the key to winning over losing. This real time data is often present in multiple formats. We need to compute the analysis with ease.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Requirements Process huge data Process data in real-time Easy to use and not very complex Requirements: Handle input from multiple sources
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Why Spark SQL? Process huge data Process data in real-time Easy to use and not very complex Requirements: Handle input from multiple sources
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Stock Market Analysis We will use stock data from yahoo finance for the following stocks: AAON Inc., AAON ABAXIS Inc., ABAX Fastenal Company, FAST F5 Networks, FFIV Gilead Sciences, GILD Microsoft Corporation, MSFT O'Reilly Automotive, ORLY PACCAR Inc., PCAR A. Schulman, SHLM Wynn Resorts Limited, WYNN Our Dataset has data from 10 companies trading in NASDAQ
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Dataset The Microsoft stocks MSFT.csv file has the following format :
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Implementing Stock Analysis Using Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Flow Diagram Huge amount of real time stock data 1 DataFrame API for Relational Processing 2 RDD for Functional Programming 3 Calculate Company with Highest Closing Price / Year Calculate Average Closing Price / Year Calculate Statistical Correlation between Companies Calculate Dates with Deviation in Stock Price 5 Spark SQL Query Spark SQL Query 4 4 Query 3 Query 4 Query 1 Query 2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Spark SQL Query Use Case: Flow Example Calculate Average Closing Price / Year 4 Real Time Stock Market Data 1 AAON Company DataFrame 2 JoinClose RDD 3 Result Table 5
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Starting Spark Shell Initialization of Spark SQL in Spark Shell Starting a Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Creating Case Class 1. Creating Case Class 2. Defining parseStock schema 3. Defining parseRDD 4. Reading AAON.csv into stocksAAONDF DataFrame
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Display DataFrame Displaying DataFrame stocksAAONDF Similarly we create DataFrames for every other company
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Average Monthly Closing Display the Average of Adjacent Closing Price for AAON for every month
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Steep Change In Graph When did the closing price for Microsoft go up or down by more than 2 dollars in a day? 1. Create ‘result’ to select days when the difference was greater than 2 2. Displaying the result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Join AAON, ABAX & FAST Stocks We now join AAON, ABAX & FAST stocks in order to compare closing prices 1. Create a Union of AAON, ABAX & FAST stocks as joinclose 2. Display joinclose
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Storing joinclose As Parquet 1. Store joinclose as a Parquet file joinstock.parquet 2. We can then create a DataFrame to work on the table 3. Display the DataFrame ‘df’.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Average Closing Per Year 1. Create newTables containing the Average Closing Prices of AAON, ABAX and FAST per year 2. Display ‘newTables’ 3. Register ‘newTable’ as a temporary table
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Transformation Transformation of ‘newTables’ with Year and corresponding 3 companies’ data into CompanyAll table 1. Create transformed table ‘CompanyAll’ 2. Display ‘CompanyAll’
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Best Of Average Closing 1. Create ‘BestCompany’ containing the Best Average Closing Prices of AAON, ABAX and FAST per year 2. Display ‘BestCompany’ 3. Register ‘BestCompany’ as a temporary table
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Best Performing Company Per Year Here, we find the company with the best Closing Price Average per year 1. Create ‘FinalTable’ from the join of BestCompanyYear and CompanyAll 2. Displaying FinalTable 3. Register ‘FinalTable’
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Use Case: Correlation We use Statistics library to find the correlation between AAON and ABAX companies closing prices. Correlation, in the finance and investment industries, is a statistic that measures the degree to which two securities move in relation to each other. The closer the correlation is to 1, the graph of the stocks follow a similar trend.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Conclusion Congrats! We have hence demonstrated the power of Spark SQL in Real Time Data Analytics for Stock Market. The hands-on examples will give you the required confidence to work on any future projects you encounter in Spark SQL.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING Thank You … Questions/Queries/Feedback

More Related Content

PDF
Introduction to apache spark
byAakashdata
 
PPTX
Programming in Spark using PySpark
byMostafa
 
PPTX
Apache Spark Core
byGirish Khanzode
 
PPTX
Apache Spark Architecture
byAlexey Grishchenko
 
PDF
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
byDatabricks
 
PDF
Spark SQL
byJoud Khattab
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
Introduction to apache spark
byAakashdata
 
Programming in Spark using PySpark
byMostafa
 
Apache Spark Core
byGirish Khanzode
 
Apache Spark Architecture
byAlexey Grishchenko
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
byDatabricks
 
Spark SQL
byJoud Khattab
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 

What's hot

PPTX
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
bySimplilearn
 
PDF
Understanding Query Plans and Spark UIs
byDatabricks
 
PDF
Apache Spark Introduction
bysudhakara st
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
PPTX
Introduction to Apache Spark
byRahul Jain
 
PPTX
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
bySimplilearn
 
PPTX
Spark
byHeena Madan
 
PDF
A Thorough Comparison of Delta Lake, Iceberg and Hudi
byDatabricks
 
PDF
Apache Spark Overview
byVadim Y. Bichutskiy
 
PDF
Introduction to Apache Spark
byAnastasios Skarlatidis
 
PPTX
Spark architecture
byGauravBiswas9
 
PPTX
PySpark dataframe
byJaemun Jung
 
PDF
Introduction to Spark Internals
byPietro Michiardi
 
PPTX
Apache Spark Fundamentals
byZahra Eskandari
 
PDF
Introduction to PySpark
byRussell Jurney
 
PDF
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
PDF
Apache Spark Core – Practical Optimization
byDatabricks
 
PDF
Big Data Architecture
byGuido Schmutz
 
PDF
Memory Management in Apache Spark
byDatabricks
 
PPTX
Tuning and Debugging in Apache Spark
byPatrick Wendell
 
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
bySimplilearn
 
Understanding Query Plans and Spark UIs
byDatabricks
 
Apache Spark Introduction
bysudhakara st
 
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
Introduction to Apache Spark
byRahul Jain
 
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
bySimplilearn
 
Spark
byHeena Madan
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
byDatabricks
 
Apache Spark Overview
byVadim Y. Bichutskiy
 
Introduction to Apache Spark
byAnastasios Skarlatidis
 
Spark architecture
byGauravBiswas9
 
PySpark dataframe
byJaemun Jung
 
Introduction to Spark Internals
byPietro Michiardi
 
Apache Spark Fundamentals
byZahra Eskandari
 
Introduction to PySpark
byRussell Jurney
 
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
Apache Spark Core – Practical Optimization
byDatabricks
 
Big Data Architecture
byGuido Schmutz
 
Memory Management in Apache Spark
byDatabricks
 
Tuning and Debugging in Apache Spark
byPatrick Wendell
 

Similar to Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka

PDF
What is Apache Spark | Apache Spark Tutorial For Beginners | Apache Spark Tra...
byEdureka!
 
PDF
Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...
byEdureka!
 
PPTX
Spark sql
byZahra Eskandari
 
PDF
Big Data Processing with Spark and Scala
byEdureka!
 
PDF
Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...
byEdureka!
 
PPTX
Spark for big data analytics
byEdureka!
 
PDF
Spark Hadoop Tutorial | Spark Hadoop Example on NBA | Apache Spark Training |...
byEdureka!
 
PDF
Big Data Processing With Spark
byEdureka!
 
PPTX
Apache Spark & Scala
byEdureka!
 
PPTX
5 things one must know about spark!
byEdureka!
 
PDF
Spark SQL | Apache Spark
byEdureka!
 
PDF
Spark For Faster Batch Processing
byEdureka!
 
PDF
Spark Streaming
byEdureka!
 
PDF
Spark is going to replace Apache Hadoop! Know Why?
byEdureka!
 
PPTX
Big data Processing with Apache Spark & Scala
byEdureka!
 
PPTX
5 reasons why spark is in demand!
byEdureka!
 
PDF
5 things one must know about spark!
byEdureka!
 
PDF
5 Reasons why Spark is in demand!
byEdureka!
 
PDF
Spark Will Replace Hadoop ! Know Why
byEdureka!
 
PPTX
Learning spark ch09 - Spark SQL
byphanleson
 
What is Apache Spark | Apache Spark Tutorial For Beginners | Apache Spark Tra...
byEdureka!
 
Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...
byEdureka!
 
Spark sql
byZahra Eskandari
 
Big Data Processing with Spark and Scala
byEdureka!
 
Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...
byEdureka!
 
Spark for big data analytics
byEdureka!
 
Spark Hadoop Tutorial | Spark Hadoop Example on NBA | Apache Spark Training |...
byEdureka!
 
Big Data Processing With Spark
byEdureka!
 
Apache Spark & Scala
byEdureka!
 
5 things one must know about spark!
byEdureka!
 
Spark SQL | Apache Spark
byEdureka!
 
Spark For Faster Batch Processing
byEdureka!
 
Spark Streaming
byEdureka!
 
Spark is going to replace Apache Hadoop! Know Why?
byEdureka!
 
Big data Processing with Apache Spark & Scala
byEdureka!
 
5 reasons why spark is in demand!
byEdureka!
 
5 things one must know about spark!
byEdureka!
 
5 Reasons why Spark is in demand!
byEdureka!
 
Spark Will Replace Hadoop ! Know Why
byEdureka!
 
Learning spark ch09 - Spark SQL
byphanleson
 

More from Edureka!

PDF
Top 5 Trending Business Intelligence Tools | Edureka
byEdureka!
 
PDF
AWS Cloud Practitioner Tutorial | Edureka
byEdureka!
 
PDF
Introduction to DevOps | Edureka
byEdureka!
 
PDF
Kubernetes Installation on Ubuntu | Edureka
byEdureka!
 
PDF
Top 10 Dying Programming Languages in 2020 | Edureka
byEdureka!
 
PDF
Python Programming Tutorial | Edureka
byEdureka!
 
PDF
Tableau Tutorial for Data Science | Edureka
byEdureka!
 
PDF
How to Deploy Java Web App in AWS| Edureka
byEdureka!
 
PDF
Top Maven Interview Questions in 2020 | Edureka
byEdureka!
 
PDF
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
byEdureka!
 
PDF
Cognitive AI Tutorial | Edureka
byEdureka!
 
PDF
Blue Prism Top Interview Questions | Edureka
byEdureka!
 
PDF
RPA in 2020 | Edureka
byEdureka!
 
PDF
EA Algorithm in Machine Learning | Edureka
byEdureka!
 
PDF
Importance of Digital Marketing | Edureka
byEdureka!
 
PDF
Linux Mint Tutorial | Edureka
byEdureka!
 
PDF
Big Data on AWS Tutorial | Edureka
byEdureka!
 
PDF
Email Notifications in Jenkins | Edureka
byEdureka!
 
PDF
Top 5 PMP Certifications | Edureka
byEdureka!
 
PDF
What to learn during the 21 days Lockdown | Edureka
byEdureka!
 
Top 5 Trending Business Intelligence Tools | Edureka
byEdureka!
 
AWS Cloud Practitioner Tutorial | Edureka
byEdureka!
 
Introduction to DevOps | Edureka
byEdureka!
 
Kubernetes Installation on Ubuntu | Edureka
byEdureka!
 
Top 10 Dying Programming Languages in 2020 | Edureka
byEdureka!
 
Python Programming Tutorial | Edureka
byEdureka!
 
Tableau Tutorial for Data Science | Edureka
byEdureka!
 
How to Deploy Java Web App in AWS| Edureka
byEdureka!
 
Top Maven Interview Questions in 2020 | Edureka
byEdureka!
 
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
byEdureka!
 
Cognitive AI Tutorial | Edureka
byEdureka!
 
Blue Prism Top Interview Questions | Edureka
byEdureka!
 
RPA in 2020 | Edureka
byEdureka!
 
EA Algorithm in Machine Learning | Edureka
byEdureka!
 
Importance of Digital Marketing | Edureka
byEdureka!
 
Linux Mint Tutorial | Edureka
byEdureka!
 
Big Data on AWS Tutorial | Edureka
byEdureka!
 
Email Notifications in Jenkins | Edureka
byEdureka!
 
Top 5 PMP Certifications | Edureka
byEdureka!
 
What to learn during the 21 days Lockdown | Edureka
byEdureka!
 

Recently uploaded

PDF
Supercharging Android Apps with On-Device AI: Gemini Nano
bydinoykraj
 
PDF
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
PDF
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
bywesley chun
 
PDF
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
PDF
OutSystsems User Group Brabant November 2025
bymail496323
 
PPTX
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
PDF
IDSECCONF2025 - Faisal Ilham - Semi Automating Vulnerability Scanner and Expl...
byidsecconf
 
PPTX
Streamlining Business Processes with UiPath Apps
byTracy Dixon
 
PDF
Linux Foundation Certified System Administrator (LFCS) Exam.pdf
byLinuxCert Guru
 
PDF
UiPath DevConnect 2025: UiPath ScreenPlay - The Future of Human-Like Automation
byDianaGray10
 
PDF
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
PPTX
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
PPTX
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 
PDF
IDSECCONF2025 - Budi Rahardjo - Cyber Security and AI.pdf
byidsecconf
 
PPTX
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
PDF
Mastering Agentic Orchestration with UiPath Maestro | Hands on Workshop
byUiPathCommunity
 
PDF
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
PDF
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
PDF
Top Crypto Supers 15th Report November 2025
byStephen Perrenod
 
PPTX
Understanding the digital experience of FE teaching staff in 2024/25
byJisc
 
Supercharging Android Apps with On-Device AI: Gemini Nano
bydinoykraj
 
[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...
byWintari Yasmin
 
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
bywesley chun
 
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
OutSystsems User Group Brabant November 2025
bymail496323
 
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
IDSECCONF2025 - Faisal Ilham - Semi Automating Vulnerability Scanner and Expl...
byidsecconf
 
Streamlining Business Processes with UiPath Apps
byTracy Dixon
 
Linux Foundation Certified System Administrator (LFCS) Exam.pdf
byLinuxCert Guru
 
UiPath DevConnect 2025: UiPath ScreenPlay - The Future of Human-Like Automation
byDianaGray10
 
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 
IDSECCONF2025 - Budi Rahardjo - Cyber Security and AI.pdf
byidsecconf
 
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
Mastering Agentic Orchestration with UiPath Maestro | Hands on Workshop
byUiPathCommunity
 
AI in Food Production (Foodwell) - AI Solutions Supporting Operations
bybyteLAKE
 
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
Top Crypto Supers 15th Report November 2025
byStephen Perrenod
 
Understanding the digital experience of FE teaching staff in 2024/25
byJisc
 

Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka