Unit 1 Information Theory and Communication Technologies

View PDF

1. Shannon Capacity

A digital signal is transmitted through a channel that is limited by its bandwidth (in Hertz) and by unwanted noise and interference. The absolute maximum data rate that can be transmitted over this channel without errors is called the Channel Capacity.

Shannon’s Formula:

Claude Shannon proved that this maximum capacity ($C$) depends entirely on the available bandwidth ($B$) and the Signal-to-Noise Ratio ($SNR$ or $S/N$).

$C=B\cdot {\log }_2\left(1+\frac{S}{N}\right)$

• S (Signal Power): The strength of the signal, which decreases over distance due to attenuation.

• N (Noise): Includes thermal noise generated by vibrating atoms in circuits, plus interference from other users.

  • Thermal Noise Formula: $N=k\cdot T\cdot B$

  • Where $k$ is the Boltzmann constant ($1.38\times 10^{-23}$ Ws/K), $T$ is temperature in Kelvin, and $B$ is the bandwidth.

Calculating Signal-to-Noise Ratio (SNR): Because power levels vary drastically (from microwatts to watts), engineers use logarithmic decibels (dB) to represent power ratios simply.

• $SN{R}_{dB}=10\cdot {\log }_{10}(S/N)$

• dBm: Power relative to 1 milliwatt ($10\cdot {\log }_{10}(P/1\text{mW})$)

• dBW: Power relative to 1 Watt ($10\cdot {\log }_{10}(P/1\text{W})$)

Example Calculation: If a channel has a 20 MHz bandwidth at 300 K, the thermal noise is $N=(1.38\times 10^{-23})\cdot 300\cdot (20\times 10^6)=8.28\times 10^{-15}$ W. If the output signal power is $1\mu W$ ($10^{-6}$ W), the SNR is $1.208\times 10^8$ (or 70.8 dB). Using Shannon’s formula, the maximum capacity $C$ would be roughly 470 Mbps.

2. Multimedia Data

All real-world information (text, audio, video) must be converted into binary bits (0s and 1s) to be processed digitally.

Advantages of Digital Processing:

• Lower costs (no need for highly precise analog components).

• Higher channel efficiency (allows for data compression).

• High reliability (allows for automatic error correction).

• Disadvantages: Requires complex circuits and more spectrum bandwidth.

Analogue-to-Digital Conversion (ADC): Converting a continuous analog signal into digital data involves two steps:

1. Sampling: Taking snapshots of the signal’s amplitude at regular time intervals.

   • Nyquist Criterion: The sampling frequency must be at least twice the maximum signal frequency ($f_{sampling}\ge 2\cdot f_{max}$) to avoid losing information.

   • Example: Human voice over telephones (up to 3.4 kHz) is sampled at 8 kHz.

   • 

2. Quantisation: Approximating the sampled amplitudes to discrete digital steps.

   • Using $n$ bits provides $2^n$ steps. The quantisation error is limited to half a step size ($\pm \Delta A/2$).

   • A-Law: A logarithmic compression algorithm used in Europe to encode voice efficiently. In ISDN telephony, voice is sampled at 8 kHz and quantised with 8 bits, resulting in a 64 kbps data rate.

n bits	2n−1 intervals	Error less than ΔA/2	Relative to (Amax​−Amin​)
1	1	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/2$	0.500
2	3	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/6$	0.167
3	7	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/14$	0.071
4	15	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/30$	0.033
5	30	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/60$	0.017
6	62	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/124$	0.008
7	126	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/252$	0.004
8	252	$(A_{max}-A_{min})/(2^{(n+1)}-2)=(A_{max}-A_{min})/504$	0.002

Binary Logics & Hardware:

• Positive Logic: “1” = High Voltage (True), “0” = Low Voltage (False).

• Negative Logic: “1” = Low Voltage, “0” = High Voltage.

• Integration Scales: Hardware circuits have shrunk massively over time, categorized by component count: SSI (<100), MSI (<1,000), LSI (<10,000), VLSI (<100k), ULSI (<1M), SLSI (<10M), ELSI (<100M), up to GLSI (Giant Large Scale Integration) with over 100 million components.

3. Data Processing & Boolean Logics

Digital logic gates process binary data using Boolean algebra.

• AND ($\wedge$): Output is 1 only if both inputs are 1. ($Z=A\wedge B$)

• OR ($\vee$): Output is 1 if either input is 1. ($Z=A\vee B$)

• NOT ($\bar{A}$): Inverts the input. ($Z=\bar{A}$)

AND (∧)	OR (∨)	NOT (¬)
$0\wedge 0=0$	$0\vee 0=0$	$\overline{1}=0$
$0\wedge 1=0$	$0\vee 1=1$	$\overline{0}=1$
$1\wedge 0=0$	$1\vee 0=1$
$1\wedge 1=1$	$1\vee 1=1$

Key Boolean Laws:

• Commutative: $A\wedge B=B\wedge A$

• Associative: $A\wedge (B\wedge C)=(A\wedge B)\wedge C$

• Distributive: $A\wedge (B\vee C)=(A\wedge B)\vee (A\wedge C)$

• DeMorgan’s Laws: $\overline{A\wedge B}=\bar{A}\vee \bar{B}$ and $\overline{A\vee B}=\bar{A}\wedge \bar{B}$

• Shannon’s Law of Inversion: Defines how to invert entire complex logic functions.

• Absorption: $A\wedge (A\vee B)=A$

Engineering Application: To build a lift controller ($Z$) that goes up only if the door is closed ($A=1$), the lift is NOT overloaded ($B=0$, so $\bar{B}=1$), and a button is pressed ($C=1$), the logic gate circuit would be wired as: $Z=A\wedge \bar{B}\wedge C$.

4. Information Content & Entropy

Information Content ($I_i$): The amount of information a specific symbol carries is inversely proportional to how often it appears ($p_i$). A rare letter carries more “surprise” and thus more information.

$I_i={\log }_2\left(\frac{1}{p_i}\right)$

Entropy ($H$):

The average amount of information across an entire message.

$H=\sum p_i\cdot {\log }_2\left(\frac{1}{p_i}\right)$

• Max Entropy ($H_0$): This is only reached if every symbol is used equally.

Redundancy ($R$): Redundancy is the difference between Max Entropy and actual Entropy ($R=H_0-H$). Languages are naturally redundant, which helps us correct errors (e.g., guessing a missing letter in a word).

• Example: Analyzing an English newspaper article yields an actual Entropy of $H=4.14$. Because the letter ‘e’ appears ~14% of the time, the letters are not used equally. Max entropy $H_0$ would be 4.70. Therefore, the Redundancy is $R=0.56$.

• Morse Code: Uses redundancy to its advantage. The most common letter (‘e’) gets the shortest code (a single dot), while rare letters (‘q’) get long codes, creating high efficiency.

5. Source Coding

Source coding compresses data by removing redundancy and irrelevant details, heavily reducing bandwidth requirements.

Video Compression (MPEG): Uncompressed video is massive. A 1280x720 video at 50 frames per second using 24-bit color generates over 1.106 Gbps. Compression is mandatory and done in two steps:

1. Spatial Compression (JPEG): Compresses each single image frame independently using Discrete Cosine Transform (DCT) to remove irrelevant visual details.

1. Temporal Compression (MPEG): Removes redundant background data between consecutive moving frames. It transmits:

   • I-Frames: Independent, full original pictures.

   • P-Frames: Predicted frames (only stores what changed since the last I or P frame).

   • B-Frames: Bidirectional frames (calculates changes by looking at both past and future frames).

By using MPEG-2, the 1.106 Gbps video can be shrunk down to just 27 Mbps.

6. Channel Coding

While source coding removes redundancy, channel coding intentionally adds redundant bits to help receivers automatically detect and fix errors caused by channel noise (FEC - Forward Error Correction), avoiding the delay of asking for retransmissions (ARQ - Automatic Repeat reQuest).

Techniques:

1. Interleaving: Shuffles the data bits before sending. If a burst of noise destroys a chunk of bits, the receiver de-interleaves them, spreading the errors out into “single-bit” errors that are much easier to fix.

1. Parity Checks: Adding bits to ensure the total number of 1s is always even.

   • Rectangular (2D) Parity: Arranges data in rows and columns with vertical and horizontal parity bits. This grid allows the receiver to pinpoint and flip the exact single broken bit.

2. Linear Block Codes: Maps $k$ message bits into an $n$-bit code word using a Generator Matrix ($G$). At the receiver, a Parity Check Matrix ($H$) generates a “Syndrome” ($S$). If $S=0$, there are no errors. If $S\ne 0$, the syndrome mathematically points to the exact error pattern to be corrected.

   • Hamming Code: A popular block code that easily fixes single-bit errors.

2. Convolutional Codes: Unlike block codes, these use shift registers to add “memory”. The output code depends not only on the current input bit but also on the previous bits in the register (defined by constraint length $K$).

2. Viterbi Decoding: A Maximum Likelihood decoding algorithm. It uses a “Trellis Diagram” to map all possible paths the convolutional code could have taken. When paths merge, it discards the paths with high error metrics (Hamming distances) to deduce the original data.

7. Modulation Schemes

Modulation maps digital binary data onto an analog radio or optical carrier wave so it can physically travel through the air or a cable.

Analogue vs. Digital Modulation:

• Analogue: Varies Amplitude (AM), Frequency (FM), or Phase (PM) in a continuous wave.

• Digital: Varies the baseband In-phase ($I$) and Quadrature ($Q$) components of the carrier wave in discrete, exact steps.

  • $s(t)=I(t)\cos (2\pi ft)+Q(t)\sin (2\pi ft)$

Digital Modulation Types (Constellations):

Represented on a “Constellation Diagram”. The points are arranged so adjacent symbols only differ by 1 bit (Hamming distance of 1) to make error correction easier.

• BPSK (Binary Phase Shift Keying): 2 symbols ($2^1$). Transmits 1 bit per symbol.

• QPSK (Quadrature PSK): 4 symbols ($2^2$). Transmits 2 bits per symbol.

• 8PSK & 16PSK: 8 and 16 symbols ($2^3$ and $2^4$). Transmits 3 or 4 bits by shifting the phase in a circle.

• QAM (Quadrature Amplitude Modulation): Varies both amplitude and phase. Extremely efficient. (16QAM, 32QAM, 64QAM, 256QAM).

  • The Trade-off: 64QAM transmits 6 bits per symbol, making it very fast. However, because the 64 dots are packed tightly together on the constellation diagram, even a tiny amount of noise will cause the receiver to guess the wrong dot. Therefore, higher QAMs require a vastly superior Signal-to-Noise Ratio (Eb/No) to function without high bit error rates.

Modulation scheme	Spectral efficiency (b/s/Hz)
BPSK	1
QPSK	2
8PSK	3
16PSK / 16QAM	4
64QAM	6
256QAM	8

8. Internet

The internet evolved from early Circuit-Switched voice telephone networks into modern Packet-Switched computer networks.

Brief History: It began as the ARPANET, funded by the US Department of Defense during the Cold War to create a resilient, decentralized data network. It exploded into public relevance when Tim Berners-Lee invented the World Wide Web (WWW) at CERN in 1991, allowing easy navigation via hyperlinks.

Key Network Protocols:

• MAC: The physical hardware address of a device.

• PPP / PPTP / L2TP: Point-to-Point and Tunneling protocols used by ISPs and VPNs to transport data securely.

• ARP / RARP: Resolves an IP address to a physical MAC address.

• DHCP: Dynamically assigns IP addresses to new devices joining a network.

• OSPF / IGRP / BGP: Routing protocols. They calculate metrics (bandwidth, delay) to find the shortest/best path for data packets across internal and exterior global networks.

• UDP (User Datagram Protocol): Unreliable transport. Sends packets fast without checking sequence numbers or asking for lost data. Ideal for real-time video/voice where waiting for delayed data ruins the experience.

• TCP (Transmission Control Protocol): Reliable transport. Tracks sequence numbers and demands retransmission (ARQ) if packets are lost. Crucial for web browsing and file downloads, but adds delay.

• SMTP & MIME: Protocols for sending emails and embedding structured multimedia (images/video) into them.

Network Scale & The All-IP Future: Networks scale from PAN (Personal, like Bluetooth), to LAWN/LAN (Local/Campus), MAN (Metropolitan), WAN (Nationwide), and GAN (Global/Trans-oceanic).

Modern telecommunications are converging onto an All-IP architecture. Instead of maintaining separate networks for TV, Telephone, and Internet, all multimedia services are packetized as IP data and routed over a unified convergence layer, regardless of the physical access technology (5G, Wi-Fi, optical fiber, or satellite) underneath.

Links:

Unit 1 Information Theory and Communication Technologies

Unit 2 Wireless Communication Technologies

Unit 3 Cellular Mobile Networks

Unit 4 Free Space Optical Communications

Unit 5 Network Security and Management

---

Social Network Analysis

Conversational AI