BigDL Deep Learning in Apache Spark - AWS re:invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Large Scale Deep Learning with BigDL T i m F o x | B i g D a t a a n d M a c h i n e L e a r n i n g C o n s u l t a n t | E l e p h a n t S c a l e

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ABOUT ME Ti m F o x , P r i n c i p a l @ E l e p h a n t S c a l e P r a c t i t i o n e r a n d Tr a i n e r i n D a t a E n g i n e e r i n g a n d D a t a S c i e n c e Author of “Data Science in Python” on LinkedIn Learning t i m @ e l e p h a n t s c a l e . c o m L i n k e d i n : t i m - f o x - 0 0 6 3 5 4 1

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ABOUT Elephant Scale • Tr a i n i n g i n B i g D a t a a n d A I t e c h n o l o g i e s • B i g D a t a : S p a r k , H a d o o p , C l o u d , N o S Q L , S t r e a m i n g • A I : M a c h i n e L e a r n i n g , D e e p L e a r n i n g , B i g D L , Te n s o r f l o w • B i g D L t r a i n i n g a v a i l a b l e ! • P u b l i c a n d P r i v a t e t r a i n i n g s a v a i l a b l e • B i g D L S a n d b o x : e l e p h a n t s c a l e . c o m / s a n d b o x E l e p h a n t s c a l e . c o m i n f o @ e l e p h a n t s c a l e . c o m

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AI / MACHINE LEARNING / DEEP LEARNING Artificial Intelligence (AI): Broader concept of machines being able to carry out 'smart' tasks Machine Learning: A type of AI that allows software to learn from data without explicitly programmed Deep Learning: Using Neural Networks to solve some hard problems A r t i f i c i a l I n t e l l i g e n c e M a c h i n e L e a r n i n g D e e p L e a r n i n g

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DEEP LEARNING APPLICATIONS S e l f D r i v i n g C a r s • ML system using image recognition • Where the edge of the road / road sign / car in front F a c e r e c o g n i t i o n • Facebook images • System learns from images manually tagged and then automatically detects faces in uploaded photos

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DEEP LEARNING HISTORY E a r l y a t t e m p t s a t D e e p L e a r n i n g d i d n o t s u c c e e d . • Compute Power was insufficient for the time. • Training Datasets were insufficiently sized for good results. • We lacked the ability to parallelize our work. I n t h e m o d e r n e r a , D e e p L e a r n i n g h a s b e e n s u c c e s s f u l . • 'Big Data' – now we have so much data to train our models • 'Big Data ecosystem' – excellent big data platforms (Hadoop, Spark, NoSQL) are available as open source • 'Big Compute' - cloud platforms significantly lowered the barrier to massive compute power • $1 buys you 16 core + 128 G + 10 Gigabit machine for 1 hr on AWS! • So running a 100 node cluster for 5 hrs  $500

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AI SOFTWARE ECO SYSTEM M a c h i n e L e a r n i n g D e e p L e a r n i n g Java - Weka - Mahout - DeepLearning4J Python - SciKit - Tensorflow - Theano - Caffe R - Many libraries - Deepnet - Darch Distributed - H20 - Spark - H20 - Spark - BigDL Cloud - AWS - AWS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MACHINE LEARNING AND BIG DATA Until recently most of the machine learning is done on “single computer” (with lots of memory–100s of GBs) Most R/Python/Java libraries are “single node based” Now Big Data tools make it possible to run machine learning algorithms at massive scale – distributed across a cluster

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TOOLS FOR SCALABLE MACHINE LEARNING A p a c h e S p a r k M L • Runs on top of popular Spark framework • Massively scalable • Can use memory (caching) effectively for iterative algorithms • Language support: Scala, Java, Python, R B i g D L • Built for Apache Spark and Optimized for Intel Xeon • Language Support: Scala, Java, Python Te n s o r F l o w • Based on “data flow graphs” • Language support: Python, C++ • https://www.tensorflow.org/

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TOOLS FOR SCALABLE CLOUD MACHINE LEARNING A m a z o n M a c h i n e L e a r n i n g • Ready to go algorithms • Visualization tools • Wizards to guide • Scalable on Amazon Cloud

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WHAT IS BIGDL A d i s t r i b u t e d d e e p l e a r n i n g l i b r a r y f o r A p a c h e S p a r k F e a t u r e p a r i t y wi t h p o p u l a r d e e p l e a r n i n g f r a m e wo r k s • Caffe, Torch, Tensorflow H i g h P e r f o r m a n c e • Powered by Intel Math Kernel Library (MKL) and multi threaded programming C a n s c a l e t o h u g e d a t a s e t s • Using Apache Spark for scale O p e n s o u r c e ! ( D e c 2 0 1 6 ) A c t i v e D e v e l o p m e n t

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PRODUCTION ML/DL SYSTEMS ARE COMPLEX! A c t u a l M L / D L i s o n l y s m a l l p o r t i o n o f m a s s i v e p r o d u c t i o n s y s t e m B i g D L r u n n i n g o n a s c a l a b l e p l a t f o r m l i k e S p a r k h e l p s s i m p l i f y t h e c o m p l e x i t y

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BIGDL FILLS THE 'GAP' IN BIG DATA + DEEP LEARNING F o l l o ws p r o v e n d e s i g n p a t t e r n s f o r d e a l i n g wi t h B i g D a t a S e n d s ' c o m p u t e t o d a t a ' r a t h e r t h a n r e a d i n g m a s s i v e d a t a o v e r n e t wo r k . U s e s ' d a t a l o c a l i t y ' o f H D F S ( H a d o o p F i l e S y s t e m U t i l i z e s ' c l u s t e r m a n a g e r s ' l i k e YA R N / M E S O S • Automatically handles hardware/software failures • Elasticity and resource sharing in a cluster

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BIGDL & SPARK R u n B i g D L a p p l i c a t i o n s a s S p a r k a p p l i c a t i o n s S c a l a , J a v a , a n d P y t h o n s u p p o r t U s e o t h e r S p a r k ' s f e a t u r e s • In memory compute • Integrate with Spark ML and Streaming E a s y d e v e l o p m e n t wi t h J u p y t e r N o t e b o o k

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BIGDL VS TENSORFLOW B i g D L Te n s o r f l o w Runtime Scala Engine with Python front-end C++ Engine with Python front- end Hadoop compatibility Can run natively on Spark & Hadoop Accesses Hadoop data as a client only Distributed Operation Scalable with Apache Spark for massive scale out of the box Does not support massive distribution out of the box Runs Tensorflow Models Yes Yes Acceleration CPU w/MKL CPU/GPU Summary Excellent for distributing deep-learning models to massive scale on big-data. Great TCO value. Excellent library for small- medium scale data, although GPU hardware costs can be significant.

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BIGDL: BIG COMPUTE PLUS BIG DATA B i g D L h e l p s u s i n b a l a n c i n g o u r n e e d s • Big Compute: Fast Linear Algebra, Intel MKL library • Optimized for Intel Xeon • Big Data: I/O parallelized to run on many CPUs B i g D L A l l o ws M a s s i v e S c a l a b i l i t y • Natively Designed to run on Spark • Works with Hadoop eco system (via Spark) Hadoop is THE Big Data platform for on-premise deployments P l a y s n i c e l y wi t h o t h e r B i g D L f r a m e wo r k s • Use existing Tensorflow or Caffe at scale in BigDL • Train new models based on existing TF / Caffe models

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BIGDL USE CASES F r a u d d e t e c t i o n S e n t i m e n t a n a l y s i s I m a g e r e c o g n i t i o n Find more at: https://github.com/intel-analytics/analytics-zoo/

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GPUS (GRAPHICS PROCESSING UNITS) G P U s h a v e a d d r e s s e d p a s t i s s u e s i n t r a i n i n g p e r f o r m a n c e • Example: Tensorflow - optimized to run well on GPUs. C P U i n p a s t n o t v e c t o r i z e d f o r p a r a l l e l c o m p u t e • Meant that GPUs were much faster for deep learning M o d e r n I n t e l X e o n C P U s h a v e v e c t o r i z e d l i n e a r a l g e b r a • Properly optimized, approaches speed of GPUs • CPUs are now a credible alternative to running on GPUs • Cost Advantage and Scalability

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. INTEL MATH KERNEL LIBRARY (MKL) F e a t u r e s h i g h l y o p t i m i z e d , t h r e a d e d , a n d v e c t o r i z e d m a t h f u n c t i o n s t h a t m a x i m i z e p e r f o r m a n c e o n e a c h p r o c e s s o r f a m i l y . U t i l i z e s i n d u s t r y - s t a n d a r d C a n d F o r t r a n A P I s f o r c o m p a t i b i l i t y w i t h p o p u l a r B L A S , L A PA C K , a n d F F T W f u n c t i o n s — n o c o d e c h a n g e s r e q u i r e d . D i s p a t c h e s o p t i m i z e d c o d e f o r e a c h p r o c e s s o r a u t o m a t i c a l l y w i t h o u t t h e n e e d t o b r a n c h c o d e . P r o v i d e s p r i o r i t y s u p p o r t , c o n n e c t i n g y o u d i r e c t l y t o I n t e l e n g i n e e r s f o r c o n f i d e n t i a l a n s w e r s t o t e c h n i c a l q u e s t i o n s

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CPU VERSUS GPU FOR BIG DATA C P U o f f e r s h i g h e r s c a l a b i l i t y a t l o w e r c o s t v e r s u s G P U O p t i m i z e d S o f t w a r e a n d l i b r a r i e s o n C P U a l l o w s i n g l e - n o d e p e r f o r m a n c e t o a p p r o a c h G P U p e r f o r m a n c e . G P U p l u s C P U a r c h i t e c t u r e s c a n b e e f f e c t i v e f o r s m a l l e r n u m b e r o f n o d e s , w h e n c o s t i s n o t a c o n c e r n . ” B i g C o m p u t e ” v e r s u s “ B i g D a t a ”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RUNNING BIGDL D e ve l o p i n g : U s e t h e f o l l o wi n g t o d e v e l o p y o u r B i g D L a p p s e ff o r t l e s s l y • Docker • VM Sandbox D e p l o y i n g : C l o u d r e a d y d e p l o y m e n t • Amazon AMI

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BigDL Summary B i g D L o f f e r s o u t s t a n d i n g s c a l a b i l i t y a n d p e r f o r m a n c e B i g D L o p t i m i z e s T C O b y r u n n i n g b e i n g t u n e d a n d o p t i m i z e d f o r I n t e l X e o n P r o c e s s o r s B i g D L b r i n g s d e e p l e a r n i n g t o S p a r k C l u s t e r s a n d H a d o o p D a t a s e t s B i g D L c a n b e u s e d t o d e p l o y Te n s o r f l o w a n d C a f f e m o d e l s t o b i g d a t a .

ABOUT ME A l e x K a l i n i n V P, A I / M a c h i n e L e a r n i n g | S i z m e k a l e x . k a l i n i n @ s i z m e k . c o m L i n k e d i n : l i n k e d i n . c o m / i n / a l e x k a l i n i n /

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://www.linkedin.com/in/alexkalinin/ alex.kalinin@sizmek.com 1 0 0 , 0 0 0 , 0 0 0 , 0 0 0 r e q u e s t s p e r d a y P B s o f t r a i n i n g d a t a AI-POWERED MARKETING AND OPTIMIZATION 7 0 , 0 0 0 , 0 0 0 / m i n u t e 1 , 2 0 0 , 0 0 0 / s e c

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. INVENTION OF CONVOLUTIONAL NEURAL NETWORKS • L e N e t - 5 n e t wo r k d e v e l o p e d i n 1 9 9 8 b y Ya n n L e C u n • To r s t e n H u b e l a n d D a v i d W i e s e l

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CONVOLUTIONAL NETWORK 0 0 0 0 1 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 Convolution Pooling Convolution PoolingInput FC FC Source: https://www.clarifai.com /technology

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ABOUT ME S u j e e M a n i y a m F o u n d e r / P r i n c i p a l @ E l e p h a n t S c a l e P r a c t i t i o n e r a n d Tr a i n e r i n D a t a E n g i n e e r i n g a n d D a t a S c i e n c e Author - "Hadoop and Spark" video training on O'Reilly Media - "HBase Design Patterns" - "Hadoop illuminated" s u j e e @ e l e p h a n t s c a l e . c o m L i n k e d i n : l i n k e d i n . c o m / i n / s u j e e m a n i y a m

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RUNNING BIGDL D e ve l o p i n g : U s e t h e f o l l o wi n g t o d e v e l o p y o u r B i g D L a p p s e ff o r t l e s s l y • Docker • VM Sandbox • Amazon AMI D e p l o y i n g : C l o u d r e a d y d e p l o y m e n t • Amazon AMI

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker Step 1 : Install Docker on your laptop Step 2 : get docker image docker pull elephantscale/bigdl-sandbox Step 3 : download tutorials git clone https://github.com/elephantscale/bigdl-tutorials Step 4 : Launch docker cd bigdl-tutorials ./run-bigdl-docker.sh elephantscale/bigdl-sandbox Step 5 : Go to Jupyter notebook

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VM-Sandbox Step 1 : Install VMware Player or VirtualBox on your laptop Step 2 : Download BigDL-Sandbox image http://elephantscale.com/sandbox/ Step 3 : (In Sandbox) download tutorials git clone https://github.com/elephantscale/bigdl-tutorials Step 4 : (In Sandbox) Run BigDL natively cd bigdl-tutorials ./run-bigdl-native.sh Step 5 : (In Sandbox) Go to Jupyter notebook

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker on AWS Step 1 : Spin up an AMI (Ubuntu recommended) Step 2 : Install Docker on the instance Step 3 : get docker image docker pull elephantscale/bigdl-sandbox Step 4 : download tutorials git clone https://github.com/elephantscale/bigdl-tutorials Step 5 : Launch docker cd bigdl-tutorials ./run-bigdl-docker.sh elephantscale/bigdl-sandbox Step 6 : Go to Jupyter notebook

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AMI on AWS Step 1 : Spin up BigDL AMI Step 2 : download tutorials git clone https://github.com/elephantscale/bigdl-tutorials Step 3 : Run BigDL cd bigdl-tutorials ./run-bigdl-native.sh Step 4 : Go to Jupyter notebook

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. QUESTIONS G i t H u b : h t t p s : / / g i t h u b . c o m / a l e x - k a l i n i n / l e n e t - b i g d l L i n k e d I n : h t t p s : / / ww w. l i n k e d i n . c o m / i n / a l e x k a l i n i n /

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Notebooks and Resources BigDL: software.intel.com/bigdl Tutorials: github.com/dnielsen/bigdl-resources Sandbox: elephantscale.com/sandbox BigDL AMI: aws.amazon.com/marketplace/ Training: elephantscale.com Slides: slideshare.net/dcnielsen/ Tim Fox Sujee Maniyam Alex Kalinin Dave Nielsen Elephant Scale Elephant Scale Sizmek Intel Software

BigDL Deep Learning in Apache Spark - AWS re:invent 2017

More Related Content

Similar to BigDL Deep Learning in Apache Spark - AWS re:invent 2017

More from Dave Nielsen

Recently uploaded

BigDL Deep Learning in Apache Spark - AWS re:invent 2017

Editor's Notes