Reddit reviews Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition (Developer Reference)

the operating system will sometimes page out parts of applications which are probably not going to be doing a lot of work -- a lot of background services and system applications will just sit idle for very long periods at a time -- and free up the RAM they'd otherwise be occupying to do other things. (if you want to see the pager at work, go download Process Explorer and look at the columns "Private Bytes" and "WS Private". "Private Bytes" is the amount of memory each application needs for its own private use; "WS Private" is the amount that the memory manager thinks it's entitled to, given its activity patterns and the system's current memory pressure. you can also pull up Performance Monitor, and look at the paging file usage counter under "Paging File > % Usage > Total", to see how much of your paging file is in use at any given time.)

as for what that RAM can be used for, off the top of my head:

• there's a piece of Windows called [the prefetcher](
  http://en.wikipedia.org/wiki/Prefetcher) which monitors your usual use cases and speculatively loads files off the disk to help you get stuff done faster by having the files you need to do it pre-cached in RAM. the less RAM it has to play with, the less it can improve your performance.
  
• file reads and writes are backed by the cache manager's cached copies of files; if the cache manager is running low on space to play with -- i.e., if you've got all sorts of idle data sitting around in RAM -- it has less space to cache files, and has to write things back to disk more often. (if you're doing heavily I/O-bound tasks, you might even end up with more disk accesses than if you'd left the paging file on!)
  
  there's a lot more about the design and implementation of Windows in Windows Internals by Russinovich, Solomon, and Ionescu, which I heartily recommend. (there's a sixth edition which is half-released right now and covers Windows 7, but the rest won't be out for a few more months.)

>As we explore the details of the internal structure and operation of Windows, you’ll see how these original design goals and market requirements were woven successfully into the construction of the system.

Its actually from book called Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition written by THE Mark Russinovich and David Solomon.

I have some knowledge on assembly and RE (Win32), but I never really got that much deep into it. I learned things as I needed in order to do my stuff.
I think the important thing here is to have a solid understanding of the architecture you're working on (for example x86/x86_64), the OS internals, and C. Without this you are going to have a very unpleasant and even painful experience.
If you don't have it yet, take the time and learn it, and then go back to RE. It will be worth it and save you a lot of effort. If you already have it, then things shall come easier to you.
If you are using Windows, I highly recommend David A. Solomon's and Mark Russinovich's Windows Internals book. It's an unmatched resource about Windows and will help you greatly.

My driver was for Windows. If that's something you want to explore, I'd highly recommend Programming the Windows Driver Model and Windows Internals. I used the 2nd and 4th editions respectively at the time I wrote it. I think the 6th edition of Internals is coming out soon though.

If I had known about the existence of these two books (or rather, their earlier editions) a decade ago, I would have been well on my way to being a better Windows programmer. These two are MUST READS for win32 programming (if you don't want to use .NET framework or other abstraction layers.)

Windows Internals, by Russinovich and Solomon

Windows via C/C++, by Richter and Nasarre

Again, if you want to do non-.NET Windows programming, these books are pure gold. (The older books by Charles Petzold are kind of getting out of date, but were good too.) Also, check out Raymond Chen's blog The Old New Thing for tons of cool bits of Windows dev history

EDIT: Also a must read: Advanced Windows Debugging this is a must if you want to learn how to properly debug "unmanaged" (i.e. non-.NET) applications. Also a pretty good intro to reverse engineering or cracking apps on Windows.