It was UN Resolution 181 that "handed" the land over to the Jews to form Israel. British policy, per the 1939 White Paper still in force in '48, was to create an Arab majority Palestine with a "participating" Jewish minority. Arab invasions of the nascent provided Israelis with the opportunity to take more land they were allotted by the UN.

Israel doesn't compare with the Raj because it's not colonized by a foreign power. Where would Israelis go if the land were returned to the Palestinians?

It happened numerous times to the British people, as England was invaded & conquered by Saxons, Angles, Jutes & Normans.

Finally, the "black & brown people" doesn't work because Israelis are by no means all white. Many, including ironically, extremist politician Itamar ben-Gvir, are Arabs themselves, while others are Ethiopian in origin. The Israel as "white supremacist" entity doesn't follow.