I didn't read Charlie & the Chocolate Factory as a kid. When I did read it, I didn't take much of it seriously. Willie Wonka, after all, wasn't a good guy. Nevermind the alleged racism for a moment. Consider that when Wonka told his workers they didn't have to come in anymore—he destroys good-paying jobs, replacing them with slaves. Even if he had replaced the workers with say robots, the damage to the community would be the same. In short, Willy Wonka was a very bad man. I'm not entirely certain Dahl was endorsing any of his employment practices.
Dahl was quintessentially English in many ways. A RAF pilot & Battle of Britain veteran, he was depicted in a Life Magazine article around the time his wife, Patricia Neal, suffered a stroke while pregnant, as being unable to physically comfort any of their children because even the slightest expression of physical affection was foreign to him.
Finally, while he might have shared that infuriating English upper-class anti-Semitism, he was hardly more sympathetic to the Nazis, & Charlie, for all of its flaws, isn't Mein Kampf.