Quantitative Data Collection Methods in Second Language Research
Quantitative Data Collection (Dörnyei)
Sample, Population, and Representativeness
Sample: A group of participants that the researcher examines in an investigation.
Population: The group of people whom the study is about.
Representative: A sample (subset) that accurately reflects the characteristics of the whole population.
Sampling Procedures
Probability Samples
Random Sampling: Selecting members of the population to be included in the sample on a random basis.
Stratified Random Sampling: Dividing the population into groups or ‘strata’ and selecting a random sample from each group.
Systematic Sampling: Selecting every nth member of the target group.
Cluster Sampling: Randomly selecting larger units of the population (e.g., schools) and examining all members within those units.
Non-Probability Samples
Quota Sampling and Dimensional Sampling: Starting with a sampling frame and determining the proportions of subgroups based on specific parameters.
Snowball Sampling: Asking participants who meet the study criteria to identify other potential members of the population.
Convenience or Opportunity Sampling: Selecting members of the target population who are readily available and meet the study criteria.
Sample Size Considerations
Guidelines
Rules of Thumb:
- More scientific sampling procedures allow for smaller sample sizes.
- Correlational research: 30 participants
- Comparative and experimental procedures: 15 participants in each group
- Factor analysis: 100 participants
Statistical Considerations: The sample should have a normal distribution and include 30 or more people.
Sample Composition: Identify subgroups within the sample that may behave differently.
Safety Margin: Leave a margin for unforeseen circumstances when determining the final sample size.
Reverse Approach: Determine the sample size needed to achieve statistically significant results based on the expected impact of the findings.
Problem of Respondent Self-Selection
This occurs when:
- Researchers invite volunteers to participate in the study.
- The design allows for a high dropout rate.
- Participants can choose whether or not to participate.
Questionnaire Surveys
Survey Studies: Aim to describe the characteristics of a population by examining a sample of that group.
Questionnaire: The main data collection method in surveys.
Types of Questionnaire Items
Factual Questions: Gather information about respondents’ background, such as location or marital status.
Behavioral Questions: Inquire about respondents’ past or current actions and experiences.
Attitudinal Questions: Explore respondents’ thoughts, opinions, interests, and values.
Multi-Item Scales: Use a cluster of items with different wording to measure the same construct.
Writing Questionnaire Items
Likert Scales: Respondents indicate their level of agreement or disagreement with a statement.
Semantic Differential Scales: Respondents mark a point on a continuum between two opposite adjectives.
Numerical Rating Scales: Respondents assign a number to describe a feature.
Other Closed-Ended Item Types
True-False Items: Assess whether a feature is present or true.
Multiple Choice Items: Often used in language proficiency testing.
Rank Order Items: Respondents order items according to their preferences.
Rules for Item Wording
- Keep items short and simple.
- Use clear and natural language.
- Avoid ambiguity and loaded words.
- Avoid negative constructions and double-barreled questions.
- Include both positively and negatively worded items.
Questionnaire Format
- Title and introduction
- Specific instructions
- Questionnaire items
- Additional information
- Final thank you
Questionnaire Layout and Item Sequence
- Booklet format with appropriate density
- Clear sequence marking
- Mixing up scales and question types
- Starting with opening and factual questions
Developing and Piloting the Questionnaire
- Drawing Up an Item Pool: Create potential items for each scale, drawing on qualitative data or existing research.
- Initial Piloting: Ask colleagues or experts to review the questions.
- Final Piloting (Dress Rehearsal): Administer the questionnaire to a group similar to the target population.
- Item Analysis: Analyze the pilot data to assess internal consistency, response patterns, and other aspects.
- Post Hoc Item Analysis: Conduct a final item analysis after administering the questionnaire to the full sample.
Administering the Questionnaire
Strategies to Encourage Participation
- Advance notice and clear communication
- Positive attitudes from teachers, parents, and authority figures
- Respectable sponsorship
- Professional behavior of the survey administrator
- Explaining the purpose and significance of the survey
Strengths and Weaknesses of Questionnaires
Strengths:
- Efficient in terms of time, effort, and resources
- Versatile and applicable to various populations and topics
Weaknesses:
- Potential for unreliable or invalid data due to poor questionnaire design
- Risk of superficial data due to the need for simple and straightforward items
- Possible literacy problems and social desirability bias among respondents
Experimental and Quasi-Experimental Studies
- Quantitative data collection designs that can establish cause-effect relationships.
- Experimental Design: Involves manipulating a variable (treatment) and observing its effects on another variable while controlling for other factors.
- Intervention Study: A type of experimental design with an experimental group (receives the treatment) and a control group (baseline for comparison).
- Pre-tests and Post-tests: Measure participants’ progress before and after the intervention.
Experimental Design: Strengths and Weaknesses
Strengths:
- Best method for establishing cause-effect relationships
- Controls for threats to internal validity
Weaknesses:
- May lack external validity (generalizability) due to artificial settings
- Potential for Hawthorne effect (participants changing behavior due to awareness of being observed)
Quasi-Experimental Design: Strengths and Weaknesses
Strengths:
- Takes place in authentic learning environments
Weaknesses:
- Threats to internal validity due to non-random assignment
- Selection bias (differences in pre-existing characteristics of groups)
- Less effective in eliminating alternative explanations compared to true experiments
Collecting Data via the Internet
Advantages
- Reduced costs and convenience
- Automatic coding and data processing
- High level of anonymity for participants
- International access and reach
- Access to specialized populations