Reddit reviews Race and Manifest Destiny: Origins of American Racial Anglo-Saxonism

If you look into the history of whiteness and blackness, it's really not. A "black" person for a long time was not an African black we refer to now, at various times Irish, Italians, and Native Americans were referred to as "black" in the United States because they were racially and culturally inferior, thus not white. Whiteness was used to demarcate issues of politics and class.

Here's a good book that discusses some of it.