Top products from r/probabilitytheory

By old-fashioned definitions. See General Irreducible Markov Chains and Non-Negative Operators by Nummelin or Markov Chains and Stochastic Stability by Meyn and Tweedie.

Markov chains used to be defined the way you say, because there were no good methods to handle general state spaces. Now there are, so the old-fashioned definition leaves out the most interesting applications for no good reason.

I really loved Harvard Stat 110 (both the book and the videos for the lectures)

http://www.amazon.com/Introduction-Probability-Edition-Dimitri-Bertsekas/dp/188652923X/ref=sr_1_1?ie=UTF8&qid=1394424420&sr=8-1&keywords=bertsekas+probability

You can find the video lectures from http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-041sc-probabilistic-systems-analysis-and-applied-probability-fall-2013/ or taking the course on edX https://www.edx.org/course/mitx/mitx-6-041x-introduction-probability-1296

*Solutions to the book exercises can be found on the book's website. Perfect for self-taught learner.

Jaynes' Probability Theory is fantastic.

Most (all?) rigorous treatments Bayesian methods require a rigorous foundation in probability theory - I think that is self explanatory.

The usual foundation for probability theory is measure theory. So, you can't have rigorous foundation in probability theory without knowing measure theory. There are other foundations, but the vast majority of the time we use measure theory - for example, convergence results like the SLLN use the measure theoretic concept of almost-sure convergence.

So, for instance, I could direct you to be a rigorous Bayesian book - for example, this book - but they will assume you already know things like the martingale convergence theorem, Radon-Nikodym, and Borel-Cantelli which are typically covered in measure-theoretic probability.

I started reading Hamming's book:
http://www.amazon.com/The-Art-Probability-Richard-Hamming/dp/0201406861

I like his style, but it's going to be coming at it from more of an engineering point of view rather than pure mathematics.

I'm only part way through it myself, but here's one I've been recomended in the past that I've been enjoying so far:

Probability Theory: The Logic of Science by E.T. Jaynes

http://www.amazon.com/Probability-Theory-The-Logic-Science/dp/0521592712

http://omega.albany.edu:8008/JaynesBook.html

The second link only appears to have the first three chapters in pdf (though it has everything as postscript files), but I would be shocked if you couldn't easilly find a free pdf off the whole thing online with a quick search.

• Machine Learning: A Probabilistic Perspective
  
• Probabilistic Graphical Models: Principles and Techniques
  
• Bayesian Data Analysis
  
• Data Analysis Using Regression and Multilevel/Hierarchical Models