Reddit reviews Information Theory, Inference and Learning Algorithms

http://neuralnetworksanddeeplearning.com/chap1.html
For neural networks and understanding the fundamentals behind backpropagation.

http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Fourth%20Printing.pdf
(book is free, also an online course on it too called statistical learning)

http://www.amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/0521642981
the author of this also has a set of lectures online:
http://videolectures.net/david_mackay/

Personally, I found it easier to learn linear algebra from the perspective of a physicist. I never really liked the pure theoretical approach. But here's a dude that likes that approach:
https://www.youtube.com/channel/UCr22xikWUK2yUW4YxOKXclQ/playlists

and you can't forget strang:
http://ocw.mit.edu/courses/mathematics/18-085-computational-science-and-engineering-i-fall-2008/video-lectures/

I think the best community for questions on any of the exercises in these book or concepts in this lecture is CrossValidated. I think its doubly helpful to answer other people's questions as well.

Here's some lighter reading I've liked:

• Analog Circuit Design: Art, Science and Personalities
  
• The Chip : How Two Americans Invented the Microchip and Launched a Revolution
  
• Gödel, Escher, Bach: An Eternal Golden Braid
  
• Feynamn Lectures on Computation
  
  Readable, but more hardcore
  
  Hardware:
  
  
• The Design of CMOS Radio-Frequency Integrated Circuits, Second Edition
  
  
  Mathy:
  
  
• A New Kind of Science. I liked this book a lot, it is worth the price just for the interesting pictures. Make up your mind for yourself, and don't let negative reviewers unduly influence you.
  
• Probability Theory: The Logic of Science
  
• Information Theory, Inference and Learning Algorithms
  
  
  Programmy:
  
  
• Mastering Regular Expressions
  
  Unixy:
  
  
• Unix Power Tools
  
  
  Off the wall:
  
  
• Molecular Biology of the Cell
  
  

I would start with Cover & Thomas' book, read concurrently with a serious probability book such as Resnick's or Feller's.

I would also take a look at Mackay's book later as it ties notions from Information theory and Inference together.

At this point, you have a grad-student level understanding of the field. I'm not sure what to do to go beyond this level.

For crypto, you should definitely take a look at Goldreich's books:

Foundations Vol 1

Foundations Vol 2

Modern Crypto

I was glad when someone pointed out David MacKay's book to me. Now I can pass it along to you. I don't know if it is directly relevant to what you are pursuing in physics, but it is a wonderful, and FREE, book. Check out the amazon reviews and see if it would be worth your time.

I always like finding intuitive explanations to help grapple with the 'true' math. It's really hard to extract meaning sometimes from hard books, but at some point, the 'true' story and the kind of challenging practice that goes with it is something you still need. If you just want to see information theory from a high level, Kahn's Academy is probably a great place to start. But when you say 'deep learning research'... if you want to write an original white paper (or even read an information theoretic paper on deep learning) you'll need to wade deep into the weeds and actually get your hands dirty. If you do want to get a good foundation in information theory for machine learning, I went through the first few chapters so far of David MacKay's information theory book and that's been great so far, excited to go through it properly at some point soon. I've heard Cover and Thomas is considered more the 'bible' of the field for undergrad/early grad study, but it takes a more communication centric approach instead of a specific machine learning based approach.

Um... though reading your comment again, do you also not know probability theory and statistics? Wasserman's all of statistics is a good source for that, but you'll need a very high level of comfort with multivariate calculus and a reasonable level of comfort with proof based mathematics to be able to weather that book.

Why don't you start looking at the kinds of papers you'd be interested in? Some research is more computational than theoretical. You've got a very long road ahead of you to get a proper foundation for original theoretical research, but getting very clear on what exactly you want to do might help you narrow down what you want to study? You really, really can't do wrong with starting with stats though, even if you do want to focus on a more computer science/practical implementation direction.

I found this book really helped me http://www.amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/0521642981