EE 671 INFORMATION THEORY AND CODING
Fall 2004
Catalog data:
Information measures, characterization of information sources, coding for discrete sources.
Discrete channel characterization, channel capacity,
noisy channel coding theorems. Various error control coding
and decoding techniques. Introduction to waveform channels
and rate distortion theory.
Textbook:
Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, Wiley Interscience, 1991.
References:
R. B. Ash, Information Theory,
Wiley Interscience, 1965.
R. G. Gallager, Information Theory and Reliable
Communication, Wiley, 1968.
Research papers.
Monday 10:00-12:00, Wednesday 14:00-15:00, all in KB 230.
Prerequisites:
1) At least EE 571-level probability theory.
2) Statistics.
3) Transform techniques.
4) Basic understanding of analog and digital communication
systems.
5) Programming skills in one high-level language.
Topics:
1) Entropy, relative entropy and mutual information.
2) The asymptotic equipartition property.
3) Entropy rate of a stochastic process.
4) Data compression: Huffman coding, arithmetic coding.
5) Channel capacity and the channel coding theorem.
6) Differential entropy.
7) The Gaussian channel.
8) Maximum entropy principle.
9) Information theory and statistics.
10) Rate distortion theory.
11) The multiple access channel.
Grading:
Homeworks (25%), one in-class, open-textbook exam (35% each),
final (40%).
You can expect about ten homework assignments.
Attendance is mandatory!
Failure to turn in
more then 60% of the homeworks will result in an automatic F regardless
of your exam performance. Between 60-80% completion will cost you a
letter grade!
Exam date:
November 11, Thursday
Homework assignments
Hw #1: Cover & Thomas 2.13, 2.19, 2.22, 2.25 due (10-11)
Hw #2: Cover & Thomas 2.31, 3.3, 3.4, 3.7 (due 10-15)
Hw #3: Cover & Thomas 4.1, 4.5, 4.9, 4.10 (due 10-18)
Hw #4: Cover & Thomas 5.8, 5.12, 5.22, 5.25 (due 10-25)
Hw #5: Cover & Thomas 8.1, 8.2, 8.3, 8.4 (due 11-3)
Hw #6: Cover & Thomas 8.8, 8.9, 8.10, 9.1 (due 11-8)
Hw #7: Cover & Thomas 9.3, 9.4 (a,b only), 9.5, 9.6 (due 11-24)
Hw #8: Cover & Thomas 10.1, 10.2, 10.3, 10.4 (due 12-8)
Hw #9: Cover & Thomas 11.1, 11.2, 11.3, 11.5 (due 12-10)
C. E. Shannon, "A mathematical theory of communication," BSTJ, 1948.