Reddit reviews Colorblind: The Rise of Post-Racial Politics and the Retreat from Racial Equity (City Lights Open Media)

You know I tried typing it up 5 times and I just can't do it succinctly, so I'm just going to point you here:

http://www.amazon.com/Colorblind-Post-Racial-Politics-Retreat-Racial/dp/0872865088

It's a book that specifically confronts exactly your attitude.

I will say this. When a white man and a black man have equal qualifications and certifications on paper for a job, all else being equal, I would assume the black man to be more determined and ambitious. Not because of an inherent quality of blackness, but because I am well aware that in this country blacks more often than not come from worse initial circumstances and face far more adversity than the average white. Not true for all, obviously, but all else being equal, I think it's reasonable to assume a black man needed a lot more perseverance and true grit to get in that seat next to a white guy and hold equal qualifications.

You can call that racism, but it's not quite the same as the traditional (actual) meaning of the word, which is belief in inherent qualities and hierarchy between the races.

Color blindness in this context means anti-affirmative action, which they see as a dog whistle for racists. See https://www.amazon.com/dp/0872865088/ref=cm_sw_r_cp_apa_ynxszb7HE5S28

This quote is exactly what they're referring to:
"You start out in 1954 by saying, “Nigger, nigger, nigger.” By 1968 you can't say “nigger” — that hurts you. Backfires. So you say stuff like forced busing, states' rights and all that stuff. You're getting so abstract now [that] you're talking about cutting taxes, and all these things you're talking about are totally economic things and a byproduct of them is [that] blacks get hurt worse than whites. And subconsciously maybe that is part of it. I'm not saying that. But I'm saying that if it is getting that abstract, and that coded, that we are doing away with the racial problem one way or the other. You follow me — because obviously sitting around saying, “We want to cut this,” is much more abstract than even the busing thing, and a hell of a lot more abstract than “Nigger, nigger.” - Lee Atwater

Some of the people arguing against affirmative action are surely racists, but to say that it's an inherently racist position is ridiculous.