Reddit reviews Group Theory in a Nutshell for Physicists

Zee just came out with a group theory for physics book this month

http://www.amazon.com/Group-Theory-Nutshell-Physicists-Zee/dp/0691162697/ref=sr_1_1?ie=UTF8&qid=1458956358&sr=8-1&keywords=zee+group+theory

I can't promise that its what you're looking for or that its a stellar book (since it came out this month), but from how well received his GR and QFT book is, I wouldn't expect anything so much less.

Interesting question! I'm not a physicist, but I can offer some insights into language, math, and how to learn them.

MATH AS A LANGUAGE

The idea that math is a language has some merits. For our purposes, we might define a language as an ordered triple (M,S,G). No, not monosodium glutamate! Here, these represent

• M = set of meanings or ideas;
  
• S = set of symbols and words to represent those meanings; and
  
• G = set of grammatical rules for combining words.
  
  Like natural languages such as English, math
  
  
• has its own set of meanings, its own symbols and words, and its own grammar for combining those symbols;
  
• allows people to communicate about complex abstractions in a standardized format by defining them in terms of simpler abstractions;
  
• evolves and grows in a fairly decentralized way; and
  
• facilitates the sharing of ideas between a vast array of human endeavors.
  
  LANGUAGE OF MATH VS. NATURAL LANGUAGES
  
  Languages such as English and Spanish have different symbols and words, as well as different grammars. However, the meanings and ideas in English and Spanish are largely the same. For example, as humans, we all have a need for water, so English and Spanish both have a word that represents the idea of water.
  
  As a result, if you already know one natural language such as English, then when you learn Spanish, you're mostly learning a new way to describe ideas you've already had.
  
  On the other hand, to learn math, you need to learn not only a new set of symbols and a new grammar, but also an entirely new set of ideas. Whereas we all need water, we don't all have a need to discuss topology.
  
  EXAMPLE
  The term "compact set" isn't a different word for a small collection of objects (something with which you might already be familiar). Instead, it's a term that represents a novel and fundamental idea from the field of topology. Specifically, a compact set is a set for which every open cover has a finite subcover.
  
  That idea builds upon several other ideas (the notion of a topology, of an open set, of an open cover, etc.). The only way to fully understand that idea is to start by learning the more basic ideas of which it is composed, then consider a range of examples and non-examples, then identify the properties common to those examples, then establish theorems relating those properties to other properties, and so on.
  
  Keep in mind that all of this depends on the setting, as well. Let's take the underlying notion of "open set." Depending on the book, this may be defined as
  
  
• a certain kind of subset of the set of real numbers;
  
• a certain kind of subset of R^n ;
  
• a certain kind of subset of a metric space; or
  
• a certain kind of subset of a topological space.
  
  The definitions are all consistent with each other, but they may appear quite different, since they're each tailored for a particular level of generality.
  
  POSSIBLE APPROACH
  The approach advised by /u/docmedic is probably the best: skim through a book or a set of course materials on the subject you want to learn, find out the prerequisites for that subject, find out the prerequisites for the prerequisites, and so on until you find materials that don't assume more than you currently know. Then work your way back up.
  
  That said, you don't necessarily need to learn everything in the prerequisite subjects in order to progress to the next set of topics; you could talk to professors and students to help you figure out which parts are needed and which aren't.
  
  Also, you may not need one hundred percent mastery of the topics you do learn. Math is best learned cyclically, I think. Try to gain as deep an understanding as you feasibly can, move on to a more advanced topic, and eventually go back to the prerequisite topic as you encounter it in more advanced contexts to see what new insights you've picked up.
  
  Lastly, don't be discouraged if a particular book is too dense. One of the comments mentioned Principles of Mathematical Analysis by Rudin as a place to start for analysis. That's an excellent book, but it works within a fairly abstract setting, and it's famously concise.
  
  Starting right away with a more abstract setting is fine if you look at lots of examples and can make sense of the abstraction, but it's probably a good idea to get several books nonetheless. I'd get some that provide lots of motivation, in order to improve your understanding, and others that are concise, so that you can better determine what's important.
  
  SOME "MATH FOR PHYSICISTS" BOOKS
  
  
• Mathematics of Classical and Quantum Physics
  
• Mathematics for Physicists
  
• Group theory in a nutshell for physicists
  
  This is just a sample.
  
  CONCLUSION
  There is not likely to be a "fast-track" to learning the language of math, at least not in the sense that you seem to be hoping for, but there might be ways to make this endeavor more manageable.
  
  If you want any help getting up to speed on advanced undergraduate math (it sounds like that's about your current level), let me know.
  
  Good luck!
  Greg at Higher Math Help

Group Theory in a Nutshell for Physicists by Anthony Zee.