You're getting a lot of praise for this article & you do make some good points, esp re the average human's capacity for evil. But I think some of your assumptions are weak.
E.g., the average antebellum slave-owner wasn't wealthy; the persistence of several terms &/or phrases isn't the same thing as "speaking the language"; all great fortunes, the world over, not just in the US, are based on great crimes; American exceptionalism doesn't necessarily come from slave owners; and I don't agree that slave owners were particularly influential save in their own, hermitic society.
None of this is to deny the continuation of racism. As we see with anti-Asian racism, or anti-Semitism, a history of slavery isn't a necessary component of racist belief & activity.