Data Mining Fundamentals and KDD Process
1. Data Mining Definition and Applications
Data Mining is the process of automatically discovering meaningful patterns, trends, and relationships from large datasets using statistical, machine learning, and database techniques.
Applications:
- Market Basket Analysis
- Fraud Detection
(Other examples: customer segmentation, medical diagnosis, recommendation systems)
2. Knowledge Discovery in Databases (KDD)
KDD is the overall process of extracting useful knowledge from data. It involves several steps such as data cleaning, data integration, data selection, data transformation, data mining, pattern evaluation, and knowledge presentation.
3. Pattern Evaluation in Data Mining
Pattern evaluation is the step in which discovered patterns are assessed to determine whether they are interesting, useful, novel, or valid based on certain measures (like support, confidence, accuracy, etc.).
4. Data Warehousing vs. Data Mining
| Data Warehousing | Data Mining |
|---|---|
| Stores and manages large volumes of historical data. | Extracts patterns and knowledge from stored data. |
| Focuses on data organization, integration, and retrieval. | Focuses on prediction, classification, and pattern discovery. |
5. Data Cleaning and Examples
Data cleaning is the process of detecting and correcting errors, inconsistencies, and missing values to improve data quality.
Example: Replacing a missing age value with the average age of the dataset.
6. Outliers and Handling Methods
Outliers are data points that deviate significantly from the rest of the data and do not follow the overall pattern.
Methods to handle outliers:
- Removal of outliers
- Transformation (e.g., log transformation)
(Other methods: capping, imputation, using robust models)
7. Naive Bayes Classifier
The Naive Bayes classifier is a probabilistic machine learning algorithm based on Bayes’ Theorem, which assumes features are independent of each other. It is widely used for text classification, spam filtering, and sentiment analysis.
8. k-Nearest Neighbors (k-NN) Algorithm
k-Nearest Neighbors (k-NN) is a simple, instance-based learning algorithm where classification or prediction is done based on the k closest data points in the feature space. It uses distance measures such as Euclidean distance and works well for classification and regression tasks.
9. Clustering Techniques
Clustering is an unsupervised learning technique that groups similar data points into clusters such that points in the same cluster are more similar to each other than to points in other clusters.
10. Dendrograms in Hierarchical Clustering
A dendrogram is a tree-like diagram used to show the hierarchical structure of clusters. It visually represents how data points are merged or split during hierarchical clustering.
11. Support and Confidence in Association Rules
Support:
The proportion of transactions in a dataset that contain a particular itemset. It measures the frequency of an itemset.
Confidence:
The probability that item Y appears in a transaction given that X already appears. It measures the strength of the rule X → Y.
12. Market Basket Analysis
Market Basket Analysis is a data mining technique used to find associations between items purchased together in transactions. It helps understand customer buying behavior (e.g., customers who buy bread often buy butter).
13. Confusion Matrix for Model Evaluation
A confusion matrix is a table used to evaluate the performance of a classification model by showing correct and incorrect predictions in the form of TP (True Positive), FP (False Positive), TN (True Negative), and FN (False Negative).
14. Cross-Validation Techniques
Cross-validation is a model evaluation technique where the dataset is divided into training and testing parts multiple times to ensure the model performs well on unseen data. The most common type is k-fold cross-validation.
15. Precision in Classification
Precision is the ratio of correctly predicted positive instances to the total predicted positives.
Precision = TP / (TP + FP)
16. Recall in Classification
Recall is the ratio of correctly predicted positive instances to all actual positives.
Recall = TP / (TP + FN)
Additional resources such as one-page exam-ready PDFs, important 5-mark answers, or expected university questions are also available for preparation.