Natural Language Processing: Core Concepts and Phases
1. Introduction to Natural Language Processing (NLP)
Definition of NLP
Natural Language Processing (NLP) is a branch of Artificial Intelligence and Computational Linguistics that enables computers to understand, interpret, generate, and interact with human languages.
Applications of NLP
- Machine Translation (e.g., translating English to Hindi)
- Chatbots and Virtual Assistants
- Speech Recognition
- Sentiment Analysis
- Information Retrieval (Search Engines)
- Text Summarization
- Question Answering Systems
- Spam Detection
- Text Classification
Challenges in NLP
- Ambiguity – Words or sentences may have multiple meanings.
- Context Understanding – Meaning often depends on surrounding text.
- Language Diversity – Different languages have different grammar structures.
- Idioms and Slang – Difficult for machines to interpret literally.
- Syntax Variations – Same meaning can be expressed in many ways.
- Pragmatic Knowledge – Understanding implied meaning and real-world knowledge.
Levels and Phases of NLP
NLP processing is generally divided into the following levels:
1. Lexical Analysis
- Converts text into words (tokens).
- Identifies word forms and meanings.
Example:
“Cats are playing”
Tokens: Cats | are | playing
2. Morphological Analysis
- Studies the internal structure of words.
- Breaks words into morphemes (smallest meaningful units).
Example:
- Unhappiness = Un + Happy + ness
- Playing = Play + ing
Purpose:
- Identify root words
- Determine tense, number, gender, etc.
3. Syntactic Analysis (Parsing)
- Examines grammatical structure.
- Checks whether a sentence follows language rules.
Example:
“The boy eats an apple.”
Correct syntax:
Subject + Verb + Object
4. Semantic Analysis
- Determines the meaning of a sentence.
- Resolves word meanings.
Example:
“The bank is closed.”
Semantic analysis determines whether “bank” means:
- Financial institution
- Riverside
5. Discourse Integration
- Considers meaning across multiple sentences.
- Uses previous sentences to interpret the current one.
Example:
Ram bought a car. He loves it.
“He” refers to Ram and “it” refers to the car.
6. Pragmatic Analysis
- Understands intended meaning using context and world knowledge.
Example:
“Can you open the window?”
Literal meaning:
- Asking about ability
Actual meaning:
- Request to open the window
Syntax vs. Semantics
| Syntax | Semantics |
|---|---|
| Deals with grammatical structure | Deals with meaning |
| Checks sentence correctness | Checks sentence interpretation |
| Focuses on arrangement of words | Focuses on understanding meaning |
| Example: “Boy the apple eats” is syntactically incorrect | Example: “Colorless green ideas sleep furiously” is syntactically correct but semantically odd |
Example
Sentence:
“The dog chased the cat.”
Syntax
- Subject = Dog
- Verb = Chased
- Object = Cat
Semantics
- Dog performs action
- Cat receives action
Ambiguity in NLP
Ambiguity occurs when a word, phrase, or sentence has more than one interpretation.
1. Lexical Ambiguity
A word has multiple meanings.
Example
“I went to the bank.”
Possible meanings:
- Financial institution
- Riverside
Resolution
Context is used to determine the correct meaning.
2. Morphological Ambiguity
A word can be analyzed into morphemes in different ways.
Example
“unlockable”
Interpretation 1:
- un + lockable (Not capable of being locked)
- Interpretation 2: unlock + able (Capable of being unlocked)
3. Syntactic Ambiguity
A sentence can have multiple grammatical structures.
Example
“I saw the man with a telescope.”
Meaning 1:
- I used a telescope to see the man.
Meaning 2:
- The man had a telescope.
4. Semantic Ambiguity
Sentence meaning is unclear despite correct grammar.
Example
“Every student read a book.”
Possible meanings:
- Same book read by all students.
- Different books read by different students.
Morphological Analysis
Definition
Morphological analysis studies word formation and identifies:
- Root word (stem)
- Prefixes
- Suffixes
- Inflections
Example
| Word | Morphological Breakdown |
|---|---|
| Unhappy | Un + Happy |
| Teachers | Teacher + s |
| Playing | Play + ing |
| Rewritten | Re + Write + en |
Importance
- Part-of-speech tagging
- Information Retrieval
- Machine Translation
- Text Mining
Semantic Parsing
Definition
Semantic parsing converts natural language into a machine-understandable representation of meaning.
Example
Sentence:
“Book a flight from Kolkata to Delhi.”
Semantic Representation:
Action: Book
Object: Flight
Source: Kolkata
Destination: DelhiUses
- Question Answering
- Chatbots
- Virtual Assistants
- Database Query Systems
Important Concepts (Exam Notes)
Syntax
- Study of grammatical structure.
- Concerned with sentence formation.
Semantics
- Study of meaning.
- Concerned with interpretation.
Lexical Ambiguity
- One word, multiple meanings.
- Example: bank, bat.
Morphological Ambiguity
- Multiple possible morpheme decompositions.
- Example: unlockable.
Discourse Integration
- Connects meanings across sentences.
- Resolves references and maintains context.
Quick Revision Table
| Concept | Key Idea |
|---|---|
| NLP | Enables computers to understand human language |
| Lexical Analysis | Tokenization of words |
| Morphological Analysis | Study of word structure |
| Syntax | Grammar and sentence structure |
| Semantics | Meaning of words and sentences |
| Semantic Parsing | Converts text into formal meaning representation |
| Lexical Ambiguity | Word has multiple meanings |
| Morphological Ambiguity | Multiple morpheme interpretations |
| Discourse Integration | Connects information across sentences |
| Pragmatics | Intended meaning using context |
One-Line Exam Definition
NLP is the field of AI that enables computers to understand, process, and generate human language through lexical, syntactic, semantic, discourse, and pragmatic analysis.