Download as KEY, PPTX
![GenericUserPreferenceArray item id - long - 8 bytes preference value - float - 4 bytes ] x millions - one user id - long - 8 bytes](https://image.slidesharecdn.com/machinelearningwithapachemahout-120216160514-phpapp02/75/Machine-Learning-with-Apache-Mahout-40-2048.jpg)
![K-Means Guess at initial placement of the centers (centroids) ]- Expectation - assign the nearest Wash, points to each centroid rinse, Maximization - reposition the centroid repeat](https://image.slidesharecdn.com/machinelearningwithapachemahout-120216160514-phpapp02/75/Machine-Learning-with-Apache-Mahout-56-2048.jpg)
Uploaded byDaniel Glauser
Machine Learning with Apache Mahout
This document provides an introduction to machine learning with Apache Mahout. It defines machine learning as a branch of artificial intelligence that uses statistics and large datasets to make smart decisions. Common applications include spam filtering, credit card fraud detection, medical diagnostics, and search engines. Apache Mahout is a platform for machine learning algorithms that allows users to build their own algorithms or use existing functionality like recommender engines, classification, and clustering.
Machine Learning with Apache Mahout
- 1. Machine Learning with Apache Mahout
- 2. http://twitter.com/danielglauser http://www.linkedin.com/in/danglauser danglauser@gmail.com
- 3. What is Machine Learning?
- 4. What is Machine Learning? A branch of Artificial Intelligence
- 5. What is Machine Learning? A branch of Artificial Intelligence Creative use of statistics
- 6. What is Machine Learning? A branch of Artificial Intelligence Creative use of statistics Smart decisions from large data sets
- 7. What is Machine Learning? A branch of Artificial Intelligence Creative use of statistics Smart decisions from large data sets All of the above
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15. Math Alert If youwant to go big with Machine Learning math is necessary What math?
- 17. Statistics Discrete Math Linear Algebra Probability
- 21. Apache Mahout A platformfor Machine Learning Roll your own algorithm, use the platform Easy integration with Hadoop
- 22. History • 2005 TheTaste framework • 2008 Services built on Lucene
- 24. Mahout is composed of... Recommender Engines Classification Clustering Frequent itemsets
- 25. A brief introto: Recommender Engines Classification Clustering
- 26. Recommendations For a givenset of input, make a recommendation
- 27. Recommendations Rank the bestout of many possibilities
- 28. Recommenders are typically User based or Item based
- 29. Neighborhood Nearest N Users Threshold
- 30.
- 31. PearsonCorrelationSimilarity Produces a value between 1 and -1 Tendency of two series to move together
- 32. PearsonCorrelationSimilarity 1 - the two series are similar 0 - no similarity -1 - opposite similarity
- 33. PearsonCorrelationSimilarity Problems Doesn’t take into account how many items overlap between users Cannot find similarity between two users if they only have one item in common Undefined if two users have identical preferences
- 34.
- 36.
- 37. How big isa Java Object?
- 38. GenericPreference user id -long - 8 bytes item id - long - 8 bytes preference value - float - 4 bytes
- 39. PreferenceArray Why not justuse an array or an ArrayList? A little overhead x millions of items = a *lot* of overhead
- 40. GenericUserPreferenceArray item id - long - 8 bytes preference value - float - 4 bytes ] x millions - one user id - long - 8 bytes
- 41.
- 42.
- 43.
- 44. Clustering Surface naturally occurringgroups of data A notion of similarity (and dissimilarity)
- 45. Clustering Algorithms do notrequire training Stopping condition - iterate until close enough
- 46. Common Clustering Algorithms K-Means Fuzzy K-Means Meanshift Centroid generation Direchlet clustering
- 47.
- 48. Feature Selection Figure outwhat features of your data are interesting
- 49. Vectorization Represent the interestingfeatures in an n- dimensional space
- 50. N-Dimensional Space Every wordin a group of documents Size, shape, color of an object
- 51. N-Dimensional Space Every wordin a group of documents Size, shape, color of an object
- 52.
- 53. Representing Vectors DenseVector Random Seek RandomAccessSparseVector SequentialAccessSparseVector
- 54. Hadoop SequenceFiles Input vectors SequenceFile(s) Initial SequenceFile(s) Centoids
- 55. K-Means 50+ years old,in commonly used for 25 years Set the number of clusters - k Works well even if you don’t pick a good - k
- 56. K-Means Guess at initialplacement of the centers (centroids) ]- Expectation - assign the nearest Wash, points to each centroid rinse, Maximization - reposition the centroid repeat
- 57. C1 C2 C3
- 58. C1 C2 C3
- 59. C1 C1 C2 C3 C3 C2
- 60. C1 C3 C2
- 61. C1 C3 C2
- 62. C1 C1 C3 C3 C2 C2
- 63. C1 C3 C2
- 64. C1 C3 C2
- 65. C1 C1 C3 C3 C2 C2
- 66. C1 C3 C2
- 67. C1 C3 C2
- 68. C1 C1 C3 C3 C2 C2
- 69. Stop! C1 C3 C2
- 70.
- 71.
- 72.
- 73.
- 74.
- 75. Classification BFF39D 577335 B3E631 D0F5B0 90B073 AFCF3C
- 76. Classification BFF39D 577335 B3E631 D0F5B0 90B073 AFCF3C
- 77. Classification BFF39D 577335 B3E631 D0F5B0 90B073 AFCF3C Green
- 78. Attributes of Classification Algorithms Require training (supervised) Make a single decision with a very limited set of outcomes
- 79. Classification Typical answers naturallyfit into categories
- 80. Examples of Classification Credit card fraud prediction Customer attrition Diabetes detector Search Engine
- 81. Training - learnedprocess that produces a model Model - output of the training algorithm
- 82. Predictor variable -input for classification model Target variable - what we are trying to predict
- 83.
- 84. Common Algorithms Stochastic GradientDecent (SGD) Support Vector Machine (SVM) Naive Bayes Complementary Naive Bayes Random Forrest
- 85.
- 86. Overhead Parallel processing requiresmanagement overhead Especially when spread over multiple machines
- 88. Vector SequenceFile Keys Values Implements Implements WritableComparable Writeable
- 89. Java WritableComparable Comparable Writeable Serializable
- 90.
- 91.
- 92.
- 93. Classification Train me tothink like you
- 94. Integration with Hadoop Through SequenceFiles and Map/Reduce jobs
- 95.
- 96.
- 101. n-dimensional space http://en.wikipedia.org/wiki/File:Coord_system_CA_0.svg Batmanhttp://www.flickr.com/photos/farukahmet/3005752670/sizes/l/in/photostream/ duke http://kenai.com/projects/duke/pages/Home mahout logo http://mahout.apache.org/ scalability diagram http://manning.com/owen/ Thanks! classification diagram http://manning.com/owen/ phew http://www.flickr.com/photos/iain/1022210850/ clouds http://www.flickr.com/photos/spazzo_1493/3682989696/ spam http://www.flickr.com/photos/johotravels/4334224546/ credit card http://www.flickr.com/photos/thetruthabout/4542026865/ medical diagnostics http://www.flickr.com/photos/adrianclarkmbbs/3063516728/ search engines http://www.flickr.com/photos/enda/144377951/ angry http://www.flickr.com/photos/jmgasalla/3467458535/ crystal ball http://www.flickr.com/photos/mache/142561526/ glasses http://www.flickr.com/photos/nickwheeleroz/2220008689/ coffee http://www.flickr.com/photos/mr_t_in_dc/2818254382/
- 102. http://twitter.com/danielglauser http://www.linkedin.com/in/danglauser danglauser@gmail.com
Editor's Notes
- #2 \n
- #3 \n
- #4 \n
- #5 \n
- #6 \n
- #7 \n
- #8 \n
- #9 \n
- #10 \n
- #11 \n
- #12 \n
- #13 \n
- #14 \n
- #15 \n
- #16 \n
- #17 \n
- #18 \n
- #19 \n
- #20 \n
- #21 \n
- #22 \n
- #23 \n
- #24 \n
- #25 \n
- #26 \n
- #27 \n
- #28 \n
- #29 \n
- #30 \n
- #31 \n
- #32 \n
- #33 \n
- #34 \n
- #35 \n
- #36 \n
- #37 \n
- #38 \n
- #39 \n
- #40 \n
- #41 \n
- #42 \n
- #43 \n
- #44 \n
- #45 \n
- #46 \n
- #47 \n
- #48 \n
- #49 \n
- #50 \n
- #51 \n
- #52 \n
- #53 \n
- #54 \n
- #55 \n
- #56 \n
- #57 \n
- #58 \n
- #59 \n
- #60 \n
- #61 \n
- #62 \n
- #63 \n
- #64 \n
- #65 \n
- #66 \n
- #67 \n
- #68 \n
- #69 \n
- #70 \n
- #71 \n
- #72 \n
- #73 \n
- #74 \n
- #75 \n
- #76 \n
- #77 \n
- #78 \n
- #79 \n
- #80 \n
- #81 \n
- #82 \n
- #83 \n
- #84 \n
- #85 \n
- #86 \n
- #87 \n
- #88 \n
- #89 \n
- #90 \n
- #91 \n
- #92 \n
- #93 \n
- #94 \n
- #95 \n
- #96 \n
- #97 \n
- #98 \n
- #99 \n
- #100 \n
- #101 \n
- #102 \n
- #103 \n
- #104 \n
- #105 \n