Natural Language Processing Assignment 2

1. Explain about Semantic Parsing

Answer
Semantic parsing is a core task in Natural Language Processing (NLP) focused on converting natural language (NL) sentences into formal meaning representations (MRs) that machines can interpret and execute . These MRs are structured outputs, such as logical forms, database queries, or programming language code, enabling computers to “understand” and act on human language .

Key Components and Process

1. Input-Output Mapping:

   • Input: A natural language utterance (e.g., “What is the capital of France?”).
   • Output: A formal representation (e.g., capital(France, X) in Prolog or SQL query SELECT capital FROM countries WHERE name = 'France').
   • The goal is to resolve ambiguity (e.g., lexical, syntactic, or semantic) to produce accurate MRs .

2. Techniques and Approaches:

   • Rule-based Systems: Early methods relied on handcrafted grammars and lexicons to map language to MRs .
   • Statistical Methods: Later approaches used supervised learning on annotated datasets (e.g., Geoquery corpus) to learn mappings .
   • Neural Networks: Modern systems employ sequence-to-sequence models, transformers, or graph-based architectures to handle complex structures .

3. Applications:

   • Question Answering: Translating questions into executable queries for databases .
   • Dialogue Systems: Enabling task-oriented virtual assistants (e.g., booking flights) by parsing user intents .
   • Code Generation: Converting NL descriptions into programming code (e.g., generating Python scripts from instructions) .

Challenges

• Ambiguity: Words like “bank” (financial institution vs. river edge) require context resolution .
• Scalability: Handling diverse domains and languages .
• Evaluation: Metrics like exact match accuracy or task-specific performance (e.g., query execution success) .

Example:

• NL: “List all flights from New York to London.”
• MR: SQL: SELECT * FROM flights WHERE origin = 'New York' AND destination = 'London' .

---

2. Explain about Predicate-Argument Structure

Answer
Predicate-Argument Structure (PAS) is a semantic framework that represents the relationship between a predicate (typically a verb) and its associated arguments (e.g., subject, object, adjuncts) in a sentence . It captures the “who did what to whom” essence of meaning, abstracting away from syntactic variations .

Core Components

1. Predicate: The core action or state (e.g., eat, run, believe).
2. Arguments: Participants or attributes linked to the predicate:
   • Subject: Actor (e.g., “The cat sat”).
   • Object: Entity affected (e.g., “She ate pizza”).
   • Adjuncts: Time, location, or manner (e.g., “at 7 PM”) .

Example:

• Sentence: “Mary gave John a book.”
• PAS: give (Predicate) ➔ Mary (Agent), John (Recipient), book (Theme) .

Applications

1. Machine Translation: Aligning PAS between languages ensures semantic equivalence (e.g., reordering arguments in Japanese vs. English) .
2. Paraphrase Detection: Identifying if two sentences share the same PAS (e.g., “The cat chased the mouse” vs. “The mouse was chased by the cat”) .
3. Information Extraction: Extracting events and their participants from text .

Challenges and Methods

• Free Word Order Languages: Techniques like dependency parsing resolve arguments in languages like Turkish .
• Deep Learning: Neural models (e.g., transformers) automate PAS extraction by learning semantic dependencies .

PAS Examples

Sentence	Predicate	Arguments
”She reads a novel”	reads	Agent: She, Theme: novel
”The meeting starts at 9 AM”	starts	Agent: meeting, Time: at 9 AM

3. Explain about Meaning Representation Systems

Answer
Meaning Representation Systems (MRS) formalize the semantics of natural language into structured formats, enabling machines to process and reason over text . These systems are critical for tasks requiring inference, knowledge extraction, or dialogue understanding.

Types of Meaning Representations

1. Abstract Meaning Representation (AMR):

   • Focuses on predicate-argument structures, abstracting away from syntax .
   • Example: “The cat sat on the mat” → (s / sit-01 :ARG0 (c / cat) :location (m / mat)).

2. FrameNet:

   • Represents scenarios (frames) and their participants (e.g., “buying” frame includes buyer, seller, goods) .

3. Dependency-based MRS:

   • Uses syntactic dependencies to encode semantic relationships .

Design Principles

• Expressiveness: Capture nuances like tense, modality, and coreference .
• Compositionality: Combine sub-structures to represent complex meanings .
• Interoperability: Align with knowledge bases (e.g., Wikidata) for downstream tasks .

Applications

• Question Answering: MRs enable precise query formulation over knowledge graphs .
• Machine Translation: Intermediate MRS improves cross-lingual transfer .
• Dialogue Systems: Track conversational context using MRS to resolve pronouns and ellipsis .

Example in Dialogue Systems:

• User: “Book a table for two at 7 PM.”
• MRS: (book :action reserve :participants user :time 19:00 :party-size 2) .

Challenges

• Ambiguity Resolution: Distinguishing between “bank” (financial) vs. “bank” (river) .
• Scalability: Balancing detail with computational efficiency .

Key Features of Meaning Representation Systems

System	Focus	Use Case
AMR	Predicate-argument structure	Cross-lingual NLP
FrameNet	Scenario-based roles	Event detection
SRL	Verb-centric argument labeling	Information extraction

---

4. Explain about N-Gram Model

Answer
The N-gram model is a foundational probabilistic language model in NLP that predicts the next word in a sequence based on the preceding $N-1$ words . It simplifies language generation and understanding by approximating context using fixed-length sequences of words (N-grams).

• N-gram: A contiguous sequence of $N$ words (e.g., “the cat sat” is a trigram, $N=3$) .
• Goal: Estimate the probability $P(w_n|w_{n-1},...,w_{n-N+1})$ of the next word $w_n$ given the previous $N-1$ words .
• Example: For the sentence “I love to eat pizza”, a trigram model predicts “pizza” based on “to eat” .

1. Probability Estimation:

   • Uses chain rule to decompose joint probabilities:
     $P(w_1,w_2,...,w_T)={\prod }_{t=1}^{T}P(w_t|w_{t-1},...,w_{t-N+1})$
   • Maximum Likelihood Estimation (MLE):
     $P(w_n|w_{n-1},...,w_{n-N+1})=\frac{\text{Count}(w_{n-N+1},...,w_n)}{\text{Count}(w_{n-N+1},...,w_{n-1})}$ .
   • Smoothing: Techniques like Laplace smoothing address zero-probability issues for unseen N-grams .

2. Common N-gram Variants:

   • Unigram ($N=1$): Ignores context (e.g., $P(\text{pizza})$) .
   • Bigram ($N=2$): Uses the previous word (e.g., $P(\text{pizza | eat})$) .
   • Trigram ($N=3$): Uses the two preceding words (e.g., $P(\text{pizza | to eat})$) .

Applications

• Machine Translation: Improves fluency by selecting likely word sequences .
• Speech Recognition: Resolves ambiguities in acoustic signals (e.g., “recognize speech” vs. “wreck a nice beach”) .
• Spell Checking: Identifies and corrects errors using context .

Challenges and Limitations

1. Sparsity: Rare or unseen N-grams result in zero probabilities (addressed via smoothing) .
2. Context Window: Fixed $N-1$ context limits long-range dependencies (e.g., failing to connect “I lived in France for 10 years. I speak…” to “French”) .
3. Scalability: Large $N$ increases computational complexity and data requirements .

---

Comparison of N-gram Variants

Model	Context Length	Strengths	Weaknesses
Unigram	0	Simple, fast	Ignores context
Bigram	1	Balances context and speed	Limited context
Trigram	2	Better context capture	High sparsity

---

Example:

• Sentence: “The cat sat on the mat.”
• Bigram probabilities:
  • $P(\text{cat | The})$, $P(\text{sat | cat})$, $P(\text{on | sat})$, etc. .

---

5. Explain about Language Models

Answer
A language model (LM) is a probabilistic framework that assigns likelihoods to sequences of words, enabling machines to generate or interpret human-like text . It forms the backbone of NLP tasks like translation, summarization, and dialogue systems.

Evolution of Language Models

1. Traditional Models:

   • N-gram Models: Dominated early NLP (1990s–2010s) by predicting words based on limited context .
   • Limitations: Struggled with long-range dependencies and scalability .

2. Neural Language Models:

   • RNNs/LSTMs: Introduced variable-length context but faced vanishing gradients .
   • Transformers: Revolutionized LMs with self-attention, enabling global context capture (e.g., BERT, GPT) .

3. Large Language Models (LLMs):

   • Scale: Billions of parameters trained on massive datasets .
   • Capabilities: Zero-shot learning, code generation, and multi-modal reasoning .

Key Concepts

1. Probability Distribution:

   • Models $P(w_1,w_2,...,w_T)$ over word sequences .
   • Example: $P(\text{"I love programming"})>P(\text{"I love programming rocks"})$.

2. Training Objective:

   • Autoregressive: Predict next word (e.g., GPT) .
   • Masked Language Modeling: Predict masked tokens (e.g., BERT) .

3. Evaluation Metrics:

   • Perplexity: Lower values indicate better model fit .
   • BLEU/ROUGE: Assess generated text quality .

Applications

• Machine Translation: Translate text by maximizing target language probability .
• Text Generation: Write stories, emails, or scripts (e.g., GPT-4) .
• Sentiment Analysis: Classify text based on contextualized embeddings .

Challenges

1. Bias and Fairness: Propagation of societal biases in training data .
2. Computational Cost: Training LLMs requires significant resources .
3. Hallucinations: Generating plausible but incorrect information .

Evolution of Language Models

Era	Model	Key Innovation
Pre-2010s	N-gram	Statistical smoothing
2010s	RNN/LSTM	Sequence modeling
2017–Present	Transformer	Self-attention mechanism
2020s	LLMs (GPT-4)	Scalability and multi-tasking

Example:

• Prompt: “Translate ‘Hello’ to French.”
• LM Output: “Bonjour” (generated using context-aware probability distributions) .