Story Line
Introduction:
Alice is a support engineer working in TS foundation, which is a software developing company. One of its feature is enabling single sign-on for its applications.
ALICES’ DAY TO DAY CHALLENGES:
Her task is to help the customers, and troubleshoot issues when needed. Whenever there’s a ticket for an issue, the first place she checks in, is the logs in the designated servers. She keeps searching and searching for related words or keyword match. Meanwhile there is change in logs every minute, and this is making her search, more and more hectic.
How can we help her?
SOLUTION:
Well this is where ELK stack comes into the picture
ELK comes with elastic search, logstash and kibana stacked altogether to give her a full analytics system.
Elastic Search enables her to search logs easily and get to know the issue and resolve it in a faster manner; not only that she can get proactive by analyzing the logs, and see if any of those customers are facing any issues or failures.
Now she can log into Kibana and search for relevant keywords easily. She can even limit the research by using timestamp filter. Monitoring single sign-on activities can be easily done by using different visualization graphs on the dashboards
Introduction to ELK stack
Learning Objectives: Let’s help Alice by introducing ELK stack to her, and helping her in understanding the core concepts and the technology behind it. This will help her in learning ELK architecture and various implementation of ELK stack in companies. Upon completing this lesson, you should be able to:
- Introduce ELK stack
- Learn about Architecture of ELK stack
- Understand various ELK terminology
- Learn the basics of Elastic Search, Logstash and Kibana
- Understand ELK stack use case
Topics:
- Introduction to ELK stack
- Why ELK?
- Architecture of ELK
- High level overview of
- Elastic Search
- Logstash
- Kibana
Parsing with Logstash
Learning Objectives: Alice has learnt to the basic concepts of ELK stack. Now what if she has to work with new sets of inputs, let’s help her with the another component of ELK stack, logstash. This module will give her a basic introduction to Logstash and guide through the process of installing Logstash and verifying that everything is running properly. After learning how to stash your first event, you can go on to create a more advanced pipeline that takes Apache web logs as input, parses the logs, and writes the parsed data to an Elasticsearch cluster. Then you learn how to stitch together multiple input and output plugins to unify data from a variety of disparate sources. At the end of this lesson, you should be able to:
- Install and verify running of Logstash on your machine
- Learn to stash first event
- Create a more advanced pipeline that takes Apache web logs as input, parses the logs, and writes the parsed data to an Elasticsearch cluster.
- Learn how to stitch together multiple input and output plugins to unify data from a variety of disparate sources
Topics:
- Introduction to Logstash
- Installing Logstash
- Configuring a log file
- Stashing your First Event
- Parsing Logs with Logstash
- Stitching together Multiple Input and Output
- Plugins
- Execution Model
Hands On:
- Step by step guide to install Logstash on your machine
- Configure the log file
- Stash your first event in Logstash
- Parsing Logs with Logstash
- Installing FileBeats and configuring it to work with Logstash
- Configuring Grok Plugin
Searching with Elasticsearch
Learning Objectives: Alice got the overview of the ELK stack, now she wants a deep understanding of each component of the stack. Let’s help her in getting started, with a brief introduction to Elastic Search with a use-case. At the end of this module, you should be able to:
- Enable Multi Value tags, Numbers, and Full text
- Retrieve the full details of any employee
- Perform Structured search
- Learn about full-text search
- Return highlighted search snippets
Topics:
- Elastic Search Overview
- Installing and running Elastic Search
- Indexing Documents
- Retrieving a Document
- Searching a Document
Hands On:
- Installing and running Elastic Search
- Indexing Documents
- Retrieving Full Document
- Retrieving a part of Document
- Checking Document Existence
- Updating a Document
- Deleting a Document
- Searching a Document (Overview)
Searching in Depth
Learning Objectives: Alice seemed excited and she is curious about learning searching in depth. She wants to explore more about Elastic Search. She understood its not just enough to use the match query. She needs to understand the data and run search query through it. This module explains her, how to index and query your data to allow her to take advantage of word proximity, partial matching, fuzzy matching, and language awareness. At the end of this module, you should be able to:
- Perform Structured Search using Elastic search
- Deploy and understand full text search query
- Know your data with multifield search
- Find associated words
- Understand partial matching query
Topics:
- Structured Search
- Full text Search
- Complicated Search
- Phrase Search
- Highlighting our Search
- Multi-field Search
- Proximity Matching
- Partial Matching
Hands On:
- Above all topics are hands-on intensive
Dealing with Human Language
Learning Objectives: Alice learned and performed various searching queries and was satisfied with it, when she suddenly realized a problem. Her query was not able to remove distinction between singular and plural words, or between tenses. She even faced problem with typos and various other problem. Let’s help Alice in solving her issues by training her on how to deal with human language for improving performance. At the end of this lesson, you should be able to:
- Remove diacritics like ´, ^, and ¨ so that a search for role will also match role, and vice versa using Normalizing Tokens.
- Remove the distinction between singular and plural—fox versus foxes—or between tenses—jumping versus jumped versus jumps—by stemming each word to its root form in Reducing Words to Their Root Form.
- Remove commonly used words or stopwords like the, and, and or to improve search performance in Stopwords: Performance Versus Precision.
- Including synonyms so that a query for quick could also match fast, or UK could match United Kingdom with the help of Synonyms.
- Check for misspellings or alternate spellings, or match on homophones—words that sound the same, like their versus there, meat versus meet versus mete using Typos and Misspellings.
Topics:
- Getting Started with languages
- Identifying Words
- Normalizing Tokens
- Reducing Words to their Root Form
- Stopwords: Performance versus Precision
- Synonyms
- Typos and Misspellings
Hands On:
- Above all topics are hands-on intensive
Data Aggregation
Learning Objectives: Alice learned all about how to search through her data, now once data is searched she needs to get a higher-level overview of the dataset and perform queries on it to get her answers in near-real time. This has made her task very tedious and tiring. Let’s ease her problem by training her with aggregation. Aggregations will allow her to ask sophisticated questions of her data in near real time. With search, we have a query and we want to find a subset of documents that match the query. We are looking for the needle(s) in the haystack. With aggregations, we zoom out to get an overview of our data. Instead of looking for individual documents, we want to analyze and summarize our complete set of data. At the end of this module you will be able to:
- Understand the concepts of buckets and metrics
- Build bar chart with buckets
- Look at the time using Date Histogram
- Filter queries and aggregation
- Sort multivalue bucket
Topics:
- High Level Concepts
- Getting started with Aggregation
- Time Analysis
- Filtering Queries and Aggregations
- Sorting Multivalue Buckets
- Approximate Aggregation
- Doc Values and Field Data
Hands On:
- Above all topics are hands-on intensive
Data Modeling
Learning Objectives: Alice was well-versed in working with SQL she thought that for handling relationships, the golden rule of relational database- normalize your data will be applicable in Elastic Search too. But as a matter of fact, this golden rule does not apply to Elastic Search. Joining entities at query time is expensive—the more joins that are required, the more expensive the query. Performing joins between entities that live on different hardware is so expensive that it is just not practical. In this module let’s discover how data is modelled in Elastic Search. At the end of this lesson, you should be able to:
- Compare Elasticsearch with RDBMS
- Get the best search result by learning Denormalizing Data
- Perform action with Nested Objects
- Understand Parent-Child Relationship
- Finally conclude the module with concept of shards and replicas
Topics:
- Elastic Search vs RDBMS
- Handling Relationships
- Nested Objects
- Parent-Child Relationship
- Designing for Scale
Hands On:
- Above all topics are hands-on intensive
Geo-location
Learning Objectives: The beauty of Elasticsearch is that it allows you to combine geolocation with full-text search, structured search, and analytics. For instance: show me restaurants that mention PIZZA, BURGER, and are within a 5-minute walk, and are open at 11 p.m., and then rank them by a combination of user rating, distance, and price. At the end of this module you will be able to:
- Understand the concepts of Geo Points
- Aggregate Geo Distance
- Understand Geohash and aggregate geohash grid
- Learn about different Geo Shapes
Topics:
- Geo Points
- Geohashes
- Geo Aggregations
- Geo Shapes
Hands On:
- Above all topics are hands-on intensive
Visualization with Kibana
Learning Objectives: Learn to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps. At the end of this lesson, you should be able to:
- Install and Verify Kibana
- Ingest .json files into Elasticsearch
- Create different visualization
- Pie Chart
- Bar Chart
- Coordinate Map
- Summarize the Dashboard
Topics:
- Introduction to Kibana
- Installing Kibana
- Loading Sample Data
- Discovering your Data
- Visualizing your Data
- Working with Dashboard
Hands On:
- Using Kibana to create a dashboard
Implementing ELK stack
Learning Objectives: Learn to interact and explore your data from the Discover page. You have access to every document in every index that matches the selected index pattern. You can submit search queries, filter the search results, and view document data. You can also see the number of documents that match the search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page. At the end of this lesson, you should be able to:
- Search your Data in Kibana
- Understanding Metric Beat
- Comparing CPU time spent in user space with offset by an hour
- Understand Timeline and Analyzing Earthquake with ELK
- Extend what is possible with Elastic search using X-Pack
- Detect Fraud with X-Pack
Topics:
- Setting the Time Filter
- Searching your Data
- Filtering by Field
- Viewing Document Data
- Viewing Document Context
- Viewing Field Statistics
- Data Visualization
- Dashboard
- Analyzing live data with ELK stack
Hands On:
- Time Filter
- Document Context
- Creating a Dashboard
