coverpage
Title Page
Copyright
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
Getting Up and Running with Spark
Installing and setting up Spark locally
Spark clusters
The Spark programming model
SparkContext and SparkConf
SparkSession
The Spark shell
Resilient Distributed Datasets
Creating RDDs
Spark operations
Caching RDDs
Broadcast variables and accumulators
SchemaRDD
Spark data frame
The first step to a Spark program in Scala
The first step to a Spark program in Java
The first step to a Spark program in Python
The first step to a Spark program in R
SparkR DataFrames
Getting Spark running on Amazon EC2
Launching an EC2 Spark cluster
Configuring and running Spark on Amazon Elastic Map Reduce
UI in Spark
Supported machine learning algorithms by Spark
Benefits of using Spark ML as compared to existing libraries
Spark Cluster on Google Compute Engine - DataProc
Hadoop and Spark Versions
Creating a Cluster
Submitting a Job
Summary
Math for Machine Learning
Linear algebra
Setting up the Scala environment in Intellij
Setting up the Scala environment on the Command Line
Fields
Real numbers
Complex numbers
Vectors
Vector spaces
Vector types
Vectors in Breeze
Vectors in Spark
Vector operations
Hyperplanes
Vectors in machine learning
Matrix
Types of matrices
Matrix in Spark
Distributed matrix in Spark
Matrix operations
Determinant
Eigenvalues and eigenvectors
Singular value decomposition
Matrices in machine learning
Functions
Function types
Functional composition
Hypothesis
Gradient descent
Prior likelihood and posterior
Calculus
Differential calculus
Integral calculus
Lagranges multipliers
Plotting
Summary
Designing a Machine Learning System
What is Machine Learning?
Introducing MovieStream
Business use cases for a machine learning system
Personalization
Targeted marketing and customer segmentation
Predictive modeling and analytics
Types of machine learning models
The components of a data-driven machine learning system
Data ingestion and storage
Data cleansing and transformation
Model training and testing loop
Model deployment and integration
Model monitoring and feedback
Batch versus real time
Data Pipeline in Apache Spark
An architecture for a machine learning system
Spark MLlib
Performance improvements in Spark ML over Spark MLlib
Comparing algorithms supported by MLlib
Classification
Clustering
Regression
MLlib supported methods and developer APIs
Spark Integration
MLlib vision
MLlib versions compared
Spark 1.6 to 2.0
Summary
Obtaining Processing and Preparing Data with Spark
Accessing publicly available datasets
The MovieLens 100k dataset
Exploring and visualizing your data
Exploring the user dataset
Count by occupation
Movie dataset
Exploring the rating dataset
Rating count bar chart
Distribution of number ratings
Processing and transforming your data
Filling in bad or missing data
Extracting useful features from your data
Numerical features
Categorical features
Derived features
Transforming timestamps into categorical features
Extract time of day
Text features
Simple text feature extraction
Sparse Vectors from Titles
Normalizing features
Using ML for feature normalization
Using packages for feature extraction
TFID
IDF
Word2Vector
Skip-gram model
Standard scalar
Summary
Building a Recommendation Engine with Spark
Types of recommendation models
Content-based filtering
Collaborative filtering
Matrix factorization
Explicit matrix factorization
Implicit Matrix Factorization
Basic model for Matrix Factorization
Alternating least squares
Extracting the right features from your data
Extracting features from the MovieLens 100k dataset
Training the recommendation model
Training a model on the MovieLens 100k dataset
Training a model using Implicit feedback data
Using the recommendation model
ALS Model recommendations
User recommendations
Generating movie recommendations from the MovieLens 100k dataset
Inspecting the recommendations
Item recommendations
Generating similar movies for the MovieLens 100k dataset
Inspecting the similar items
Evaluating the performance of recommendation models
ALS Model Evaluation
Mean Squared Error
Mean Average Precision at K
Using MLlib's built-in evaluation functions
RMSE and MSE
MAP
FP-Growth algorithm
FP-Growth Basic Sample
FP-Growth Applied to Movie Lens Data
Summary
Building a Classification Model with Spark
Types of classification models
Linear models
Logistic regression
Multinomial logistic regression
Visualizing the StumbleUpon dataset
Extracting features from the Kaggle/StumbleUpon evergreen classification dataset
StumbleUponExecutor
Linear support vector machines
The naive Bayes model
Decision trees
Ensembles of trees
Random Forests
Gradient-Boosted Trees
Multilayer perceptron classifier
Extracting the right features from your data
Training classification models
Training a classification model on the Kaggle/StumbleUpon evergreen classification dataset
Using classification models
Generating predictions for the Kaggle/StumbleUpon evergreen classification dataset
Evaluating the performance of classification models
Accuracy and prediction error
Precision and recall
ROC curve and AUC
Improving model performance and tuning parameters
Feature standardization
Additional features
Using the correct form of data
Tuning model parameters
Linear models
Iterations
Step size
Regularization
Decision trees
Tuning tree depth and impurity
The naive Bayes model
Cross-validation
Summary
Building a Regression Model with Spark
Types of regression models
Least squares regression
Decision trees for regression
Evaluating the performance of regression models
Mean Squared Error and Root Mean Squared Error
Mean Absolute Error
Root Mean Squared Log Error
The R-squared coefficient
Extracting the right features from your data
Extracting features from the bike sharing dataset
Training and using regression models
BikeSharingExecutor
Training a regression model on the bike sharing dataset
Generalized linear regression
Decision tree regression
Ensembles of trees
Random forest regression
Gradient boosted tree regression
Improving model performance and tuning parameters
Transforming the target variable
Impact of training on log-transformed targets
Tuning model parameters
Creating training and testing sets to evaluate parameters
Splitting data for Decision tree
The impact of parameter settings for linear models
Iterations
Step size
L2 regularization
L1 regularization
Intercept
The impact of parameter settings for the decision tree
Tree depth
Maximum bins
The impact of parameter settings for the Gradient Boosted Trees
Iterations
MaxBins
Summary
Building a Clustering Model with Spark
Types of clustering models
k-means clustering
Initialization methods
Mixture models
Hierarchical clustering
Extracting the right features from your data
Extracting features from the MovieLens dataset
K-means - training a clustering model
Training a clustering model on the MovieLens dataset
K-means - interpreting cluster predictions on the MovieLens dataset
Interpreting the movie clusters
Interpreting the movie clusters
K-means - evaluating the performance of clustering models
Internal evaluation metrics
External evaluation metrics
Computing performance metrics on the MovieLens dataset
Effect of iterations on WSSSE
Bisecting KMeans
Bisecting K-means - training a clustering model
WSSSE and iterations
Gaussian Mixture Model
Clustering using GMM
Plotting the user and item data with GMM clustering
GMM - effect of iterations on cluster boundaries
Summary
Dimensionality Reduction with Spark
Types of dimensionality reduction
Principal components analysis
Singular value decomposition
Relationship with matrix factorization
Clustering as dimensionality reduction
Extracting the right features from your data
Extracting features from the LFW dataset
Exploring the face data
Visualizing the face data
Extracting facial images as vectors
Loading images
Converting to grayscale and resizing the images
Extracting feature vectors
Normalization
Training a dimensionality reduction model
Running PCA on the LFW dataset
Visualizing the Eigenfaces
Interpreting the Eigenfaces
Using a dimensionality reduction model
Projecting data using PCA on the LFW dataset
The relationship between PCA and SVD
Evaluating dimensionality reduction models
Evaluating k for SVD on the LFW dataset
Singular values
Summary
Advanced Text Processing with Spark
What's so special about text data?
Extracting the right features from your data
Term weighting schemes
Feature hashing
Extracting the tf-idf features from the 20 Newsgroups dataset
Exploring the 20 Newsgroups data
Applying basic tokenization
Improving our tokenization
Removing stop words
Excluding terms based on frequency
A note about stemming
Feature Hashing
Building a tf-idf model
Analyzing the tf-idf weightings
Using a tf-idf model
Document similarity with the 20 Newsgroups dataset and tf-idf features
Training a text classifier on the 20 Newsgroups dataset using tf-idf
Evaluating the impact of text processing
Comparing raw features with processed tf-idf features on the 20 Newsgroups dataset
Text classification with Spark 2.0
Word2Vec models
Word2Vec with Spark MLlib on the 20 Newsgroups dataset
Word2Vec with Spark ML on the 20 Newsgroups dataset
Summary
Real-Time Machine Learning with Spark Streaming
Online learning
Stream processing
An introduction to Spark Streaming
Input sources
Transformations
Keeping track of state
General transformations
Actions
Window operators
Caching and fault tolerance with Spark Streaming
Creating a basic streaming application
The producer application
Creating a basic streaming application
Streaming analytics
Stateful streaming
Online learning with Spark Streaming
Streaming regression
A simple streaming regression program
Creating a streaming data producer
Creating a streaming regression model
Streaming K-means
Online model evaluation
Comparing model performance with Spark Streaming
Structured Streaming
Summary
Pipeline APIs for Spark ML
Introduction to pipelines
DataFrames
Pipeline components
Transformers
Estimators
How pipelines work
Machine learning pipeline with an example
StumbleUponExecutor
Summary
更新时间：2021-07-09 21:08:48
完结共367章
倒序