Introduction to Spark Programming

This Introduction to Spark Programming Courseware for License training class introduces the Apache Spark distributed computing engine and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner.

The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g., RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g., Spark SQL and DataFrames). It also covers more advanced capabilities such as the use of Spark Streaming to process streaming data and provides an overview of Spark GraphX (graph processing) and Spark MLlib (machine learning). Finally, the course explores possible performance issues and strategies for optimization.

The course is very hands-on, with many labs. Participants will interact with Spark through the Spark shell (for interactive, ad hoc processing) as well as through programs using the Spark API. Labs currently support Scala; support of Python/Java is available.

The Apache Spark distributed computing engine is rapidly becoming a primary tool in the processing and analyzing of large-scale data sets. It has many advantages over existing engines, such as Hadoop, including runtime speeds that are 10-100x faster, as well as a much simpler programming model. After taking this course, you will be ready to work with Spark in an informed and productive manner.

Goals
  1. Understand the need for Spark in data processing
  2. Understand the Spark architecture and how it distributes computations to cluster nodes
  3. Become familiar with basic installation / setup / layout of Spark
  4. Use the Spark shell for interactive and ad-hoc operations
  5. Understand RDDs (Resilient Distributed Datasets), and data partitioning, pipelining, and computations
  6. Understand/use RDD ops such as map(), filter(), reduce(), groupByKey(), join(), etc.
  7. Understand Spark's data caching and its usage
  8. Write/run standalone Spark programs with the Spark API
  9. Use Spark SQL / DataFrames to efficiently process structured data
  10. Use Spark Streaming to process streaming (real-time) data
  11. Understand performance implications and optimizations when using Spark
  12. Become familiar with Spark GraphX and MLlib
Outline
  1. Scala Ramp Up
    1. Scala Introduction, Variables, Data Types, Control Flow
    2. The Scala Interpreter
    3. Collections and their Standard Methods (e.g. map())
    4. Functions, Methods, Function Literals
    5. Class, Object, Trait
  2. Introduction to Spark
    1. Overview, Motivations, Spark Systems
    2. Spark Ecosystem
    3. Spark vs. Hadoop
    4. Acquiring and Installing Spark
    5. The Spark Shell
  3. RDDs and Spark Architecture
    1. RDD Concepts, Lifecycle, Lazy Evaluation
    2. RDD Partitioning and Transformations
    3. Working with RDDs - Creating and Transforming (map, filter, etc.)
    4. Key-Value Pairs - Definition, Creation, and Operations
    5. Caching - Concepts, Storage Type, Guidelines
  4. Spark API
    1. Overview, Basic Driver Code, SparkConf
    2. Creating and Using a SparkContext
    3. RDD API
    4. Building and Running Applications
    5. Application Lifecycle
    6. Cluster Managers
    7. Logging and Debugging
  5. Spark SQL
    1. Introduction and Usage
    2. DataFrames and SQLContext
    3. Working with JSON
    4. Querying - The DataFrame DSL, and SQL
    5. Data Formats
  6. Spark Streaming
    1. Overview and Streaming Basics
    2. DStreams (Discretized Steams)
    3. Architecture, Stateless, Stateful, and Windowed Transformations
    4. Spark Streaming API
    5. Programming and Transformations
  7. Performance Characteristics and Tuning
    1. The Spark UI
    2. Narrow vs. Wide Dependencies
    3. Minimizing Data Processing and Shuffling
    4. Using Caching
    5. Using Broadcast Variables and Accumulators
  8. Spark GraphX Overview (Optional)
    1. Introduction
    2. Constructing Simple Graphs
    3. GraphX API
    4. Shortest Path Example
  9. MLLib Overview (Optional)
    1. Introduction
    2. Feature Vectors
    3. Clustering / Grouping, K-Means
    4. Recommendations
    5. Classifications
Class Materials

Each student in our Live Online and our Onsite classes receives a comprehensive set of materials, including course notes and all the class examples.

Class Prerequisites

Experience in the following is required for this Spark class:

  • Reasonable programming experience. An overview of Scala is provided for those who don't know it.
Preparing for Class

Training for your Team

Length: 3 Days
  • Private Class for your Team
  • Online or On-location
  • Customizable
  • Expert Instructors

What people say about our training

This is an excellent class for Project Management newbies as well as those who are experienced. Lots of great resources and information and the instructor was very knowledgeable and helpful.
Shane Walker
Peterson AFB
Taking this course definitely moved my ability to a new level. I can't wait to create some great presentations. Instructor was great.
Karen Dunnigan
URS Corporation
What a great way to learn FrameMaker! I would highly recommend this class to anyone needing a first instruction to FrameMaker or who needs a review. The instructor gave many best practices throughout the course which were as valuable as the basic instruction itself.
Davee Kaplan
C3 Networx
I would definitely take another course from Webucator. The instructor was awesome, and the class size (three students) was great!
Jennifer Porter
Metafore

No cancelation for low enrollment

Certified Microsoft Partner

Registered Education Provider (R.E.P.)

GSA schedule pricing

60,864

Students who have taken Instructor-led Training

11,703

Organizations who trust Webucator for their Instructor-led training needs

100%

Satisfaction guarantee and retake option

9.29

Students rated our trainers 9.29 out of 10 based on 28,812 reviews

Contact Us or call 1-877-932-8228