I believe it is, because you argue that slave owners were "wealthy & influential". Generally the two go hand in hand. But if the average slave owner was more likely to be a small-hold farmer with five or fewer slaves, then the degree of their influence over the nation as a whole is questionable.
After all, isn't this what prompted secession in the first place? The perception that the south's influence over the nation was rapidly waning?
Then there's the notion that, say, a single term such as "master bedroom" means that Americans have a slave-holder mentality. I'm honestly curious, is the term used abroad? In other English speaking countries where slavery had less impact, or even non-English-speaking countries?
One more point, given the media-dense nature of society, it takes hardly any time at all to eliminate something like "master bedroom" from common usage. I believe the process is already at work. But will having done so lead to any kind of authentic racial equity?