Usability Inspection, Data Gathering, and Analysis Methods
————— LECTURE 4: —————— INSPECTION:
Experts evaluate interface without involving users. Goal: Identify usability problems early in design.
Advantages: Cheap, fast, early design feedback. Disadvantages: Depends on expert judgment, may miss real user issues.
———– TYPES OF INSPECTION METHODS:
1.) Heuristic Evaluation, 2.) Cognitive Walkthrough, 3.) Pluralistic Walkthrough, 4.) Guidelines Review, 5.) Consistency Inspection.
———– HEURISTIC EVALUATION:
Developed by Jakob Nielsen. Experts evaluate interface using usability heuristics (rules.). Each expert identifies problems and rates severity.
——— SEVERITY RATING SCALE:
Used in heuristic evaluation.
0 = Not a problem, 1 = cosmetic issue, 2 = minor usability problem, 3 = major usability problem, 4 = usability catastrophe.
———- COGNITIVE WALKTHROUGH:
Evaluation method focused on how easy it is for new users to learn the interface. Experts simulate user performing tasks. Questions asked at each step: Will the user try correct action, will the user notice the correct control, will the user associate control with goal, will feedback show progress. Purpose: Evaluate learnability.
———- PLURALISTIC WALKTHROUGH:
Evaluation with multiple stakeholders together. Participants include usability experts, developers, users. Process: Walk thru tasks step by step, each participant writes expected action, discuss differences. Advantage: Multiple perspectives.
———- GUIDELINES REVIEW:
Experts compare interface against design guidelines or standards. EX: platform UI guidelines, company design rules. Goal: Ensure interface follows best practices.
———– CONSISTENCY INSPECTION:
Checks if interface elements behave consistently. EX: menu naming, button placement, icons, terminology. Inconsistent interfaces confuse users.
———– ANALYTICS:
Evaluation using automatically collected usage data.
Data collected from real users. EX: website traffic, click patterns, session duration.
———— ANALYTICS DATA EXAMPLES:
Metrics: Page views, bounce rate, time on page, click paths, conversation rate. Purpose: Understand how users actually use the system.
———– A/B TESTING:
Experiment comparing two versions of interface. EX: Version A -> Current Design. Version B -> New Design. Users are randomly assigned. Metrics compared would be clicks, conversions, time. Goal: Find better design.
———– FUNNEL ANALYSIS:
Tracks user progress through steps. EX: 1.) Homepage, 2.) Product page, 3,) Checkout, 4.) Payment. Used to find where users drop off.
———- HEATMAPS:
Visual representation of where users click or look. Types: Click heatmaps, scroll heatmaps, Eye-tracking heatmaps. Shows user attention areas.
———– ADVANTAGES OF ANALYTICS:
Large data sets, real users, real usage.
———– DISADVANTAGES OF ANALYTICS:
Lacks context, cannot explain why users behave certain way, and privacy concerns.
———– 3 PREDICTIVE MODELS:
PREDICTIVE MODELS:
Mathematical models used to predict user performance. Goal: Estimate usability without testing with users. Used in interface design and human performance prediction.
——— GOMS MODEL:
G -> GOALS, O-> OPERATORS, M-> METHODS, S-> SELECTION RULES. Used to analyze task execution time. EX: Goal is to save a document. Operators might include, move mouse, click button, press key. Method: Sequence of operators.
——– KEYSTROKE-LEVEL MODEL (KLM):
Extension of GOMS. Predicts task completion time and uses standard times for actions. Common Operators: K -> keystroke, P-> Pointing with mouse, H -> Hand movement, M -> Mental preparation. R -> System Response. Total time = sum of operator times.
———- ADVANTAGES OF PREDICTIVE MODELS:
No users needed, fast analysis, and useful for early design.
——– DISADVANTAGES OF PREDICTIVE MODELS:
Simplified assumptions, May not reflect real behavior.
————- LECTURE 5 ————— DATA GATHERING:
Process of collecting data about users, tasks, and environments. Purpose is to understand users, understand tasks, evaluate systems and inform design decisions.
————- CLEAR GOALS:
All data gathering sessions must have clear goals. EX: Understanding user behavior, evaluating usability, identifying problems.
———– PILOT STUDY:
A small test run of the study. Purpose: Test procedures, find problems, refine questions.
———- TRIANGULATION:
Investigating a phenomenon using multiple techniques or perspectives. EX: Using interviews, observation, questionnaires. Different methods reveal different insights.
——— DATA RECORDING METHODS:
Data can be recorded using handwritten notes, audio recording, photos, system logs, video recording.
——– TYPES OF DATA: QUANTITATIVE DATA:
Numeric Data. EX: Time to complete task, number of errors, clicks, success rate. Used for statistical analysis.
——— QUALITATIVE DATA:
Descriptive data, EX: Opinions, user comments, behaviors, observations. Used to understand why users behave certain ways.
——— DATA GATHERING TECHNIQUES:
1.) Interviews, 2.) Questionnaires, 3.) Observation, 4.) Logs/Indirect Observation, 5.) Focus groups.
———- TYPES OF INTERVIEWS: STRUCTURED INTERVIEW:
Fixed questions, same order for all participants. Advantages: Consistent and easy to compare responses. Disadvantages: less flexible.
———- SEMI-STRUCTURED INTERVIEW:
Prepared questions, interviewer may ask follow-up questions. Most commonly used and finds a balance between structure and flexibility.
———- UNSTRUCTURED INTERVIEW:
Very open discussion. Advantages: deep insights. Disadvantages: difficult to analyze.
———– PLANNING AN INTERVIEW:
1.) Develop list of questions or topics. 2.) Prepare documentation (consent form, project description), 3.) Check recording equipment, 4.) Decide interview structure, 5.) Choose time and place.
——— DEVELOPING INTERVIEW QUESTIONS:
Avoid compound questions. EX: How do u like this app compared to previous ones. Better: How do u like this app? Have u used similar app. Avoid jargon and use simple language. Keep questions neutral.
——— RUNNING AN INTERVIEW:
1.) Introduction -> explain the purpose of the interview, confidentiality, and recording permission. 2.) Warm-up Session: Easy questions first, ex: demographics, general experience. 3.) Main Session – Main questions asked, and more probing questions towards the end. 4.) Cool-Off Period: Final questions, and additional comments/suggestions.
———- REMOTE INTERVIEWS:
Advantages: Participants are comfortable, no travel required, anonymity possible. Challenges: Internet problems and less personal interaction.
———- ENRICHING INTERVIEWS:
Interviews can be improved using props. EX: prototypes, screenshots, storyboards, work artifacts. Purpose: Provide context and improve responses.
———- FOCUS GROUPS:
Group discussion with several participants. Used to gather opinions and experiences. Moderator leads discussion and typically 4-8 participants.
————- QUESTIONNAIRES:
Set of written questions answered by participants. Used for collecting demographic data, opinions, and feedback.
———— ADVANTAGES OF QUESTIONNAIRES:
Cheap, fast, can reach many people, and have standardized responses.
———— DISADVANTAGES OF QUESTIONNAIRES:
shallow responses, misunderstanding questions, lower response rates.
———– QUESTIONNAIRE STRUCTURE:
1.) Typical order would be demographic questions, background experience, main research questions.
———– QUESTIONNAIRE DESIGN ADVICE:
Checklist: Consider question order, provide clear instructions, avoid confusing wording, keep questionnaire concise, balance whitespace and readability.
——— QUESTION TYPES:
User selects from fixed options. EX: Rate Usability. 1-very bad. 5- Very good. Advantages: Easy to analyze, and gives quantitative data.
———- OPEN QUESTIONS:
User writes their own answer: EX: What did u dislike about the interface? Advantages: Detailed responses. Disadvantages: Harder to analyze.
——— RATING SCALES:
1.) Likert scale: Used to measure opinions and attitudes. EX: Strongly disagree, agree, neutral, etc.. Common in usability studies.
——— SEMANTIC DIFFERENTIAL SCALE:
Uses pairs of opposite adjectives. EX: Easy ——— Difficult. User selects position between extremes. Scores summed across questions.
——— ADMINISTERING QUESTIONNAIRES
Two major issues. 1.) Obtaining representative sample. 2.) Achieving high response rate.
——– WEB-BASED QUESTIONNAIRES:
Advantages: Interactive and easy distribution. Problems: Usually convenience sampling and results are harder to generalize.
——— DEPLOYING A WEB QUESTIONNAIRE:
STEPS -> 1.) Design error-free questionnaire. 2.) Capture respondent identity safely. 3.) Pilot test questionnaire. Pilot testing may occur in several stages.
——– OBSERVATION:
Watching users perform tasks. Purpose: Understand real behavior and context.
——— DIRECT OBSERVATION IN THE FIELD:
Occurs in natural environment. Ex: Observing students using an app on campus. Benefits: Reveals real behavior, and provides context. Challenges: Difficult to conduct and produces large amounts of data.
——- OBSERVATION FRAMEWORK:
Look for Person (Who is using the technology?), Place (Where is it being used?), Thing (What are they doing?)
——- PASSIVE OBSERVER:
Observer does not participate, used in lab studies.
——- PARTICIPANT OBSERVER:
Observer becomes part of the group, used in ethnographic studies.
——- THINK-ALOUD TECHNIQUE:
Participants say their thoughts aloud while performing tasks. Purpose: Understand user reasoning. EX: User explains what they are thinking while navigating interface.
——- INDIRECT OBSERVATION:
Analyzing behavior without directly watching users. EX: Usage logs, screen records. Purpose: Find patterns in behavior.
CHOOSING DATA GATHERING TECHNIQUES:
1.) Study Goals. If exploring early design ideas -> interviews and observation. If measuring opinions -> questionnaires. 2.) Participants: Technique depends on users, EX: Children vs adults. 3.) Available Resources: Available Resources. Consider: time, money, equipment, etc.
———- CHAPTER 6 ——— FIRST STEPS IN ANALYZING DATA:
First step after collecting data is called initial processing. Interviews -> Raw Data (Process by transcribing recordings, expanding notes, extract relevant sections). Questionnaires: Written responses, online surveys (Process by cleaning up data, removing invalid entries, filter data into categories.) Observation: Photos, think-aloud data, videos, observer notes, (Process by expanding notes, transcribe recordings)
—— NEVER RELY ON INITIAL IMPRESSIONS BC THEY BRING BIAS
—–
TYPES OF AVERAGES:
Mean: Standard average. Median: Middle value when nums are ordered. Mode: Most common value.
——– QUESTION DESIGN:
Matters bc it affects how data is analyzed & what conclusions you draw.
——— THREE COMMON QUALITATIVE ANALYSIS METHODS:
1.) Identifying recurring themes. Affinity Diagram -> Purpose: Organize ideas into groups of related themes. Process -> 1.) Write observations on notes, 2.) Group similar notes together, 3.) Create categories based on patterns. Categories emerge from data, not predefined. Process is inductive.
——– CATEGORIZING DATA:
1.) Break data into elements, 2.) Assign categories, 3.) Analyze patterns.
——- CATEGORIZATION SCHEME:
Goal: Organize the data for analysis.
——— CHALLENGES IN CATEGORIZATION:
Categories should be orthogonal meaning they don’t overlap.
——— CHOOSING GRANULARITY:
Granularity = level of detail. EX: word level, paragraph level. ——–
INTER-RATER RELIABILITY:
Definition: percentage of agreement between researchers. Formula: # of agreements / total items. High Percentage = more reliable categorization.
TOOLS FOR DATA ANALYSIS: SPSS
Statistical software, capabilities: frequency distributions, rank correlations, regression analysis, cluster analysis.
—— MINITAB:
Statistics software used for normality tests, statistics, regression.
——– EXCEL:
Common tool for simple analysis. Capabilities: Frequency counts, distributions, variance, percentiles.
——- R:
programming language for stats. Capabilities: Classification, clustering, graphics.
——- USING THEORETICAL FRAMEWORKS:
Qualitative analysis often uses theoretical frameworks. Frameworks guide how data is interpreted.
——- GROUNDED THEORY:
Method that develops theory from the data itself. Theory is grounded in empircal data meaning it emerges from observation. Bottom-up approach. Goal: Develop a theory that explains the collected data.
——– GROUNDED THEORY PROCESS:
Collect data, analyze data, identify categories, collect more data, refine categories. Continue until no new insights appear.
——- DATA CODING IN GROUNDED THEORY, 3 MAIN CODING STAGES. 1.) OPEN CODING:
Identify: categories, properties, dimensions. Breaks data into pieces.
—— AXIAL CODING:
Connect categories with subcategories. Establish relationships.
—— SELECTIVE CODING:
Integrate categories into a central theory. Organize around a core concept.
——– DISTRIBUTED COGNITION:
Theory that analyzes how cognitive processes are distributed across people, tools, environments. Focus: How info moves through the system.
—— PURPOSE OF MICRO-LEVEL ANALYSIS:
Detailed Analysis reveals hidden practices, coordination patterns, system problems.
—— PRESENTING FINDINGS:
After analysis, results must be presented clearly. EX: Stories, summaries, rigorous notation.
—– RIGOROUS NOTATIONS:
EX: UML, Advantages: Precise representation, structured analysis. Disadvantages: Hard to understand for noobs, may ignore some aspects.
—- 3 WAYS STORIES ARE USED:
1.) Stories told by participants, about participants, and stories built from patterns in data. Help create a representative narrative of system use.