Why is it considered racist to not want ethnic minorities in our native lands?

They are always griping about how we whites are in their lands and their people are being oppressed blah blah blah, and nobody every says anything about it. Yet when we complain about them being in our country abusing our generous welfare system we get called racist. Why the double standards? Can't the minorities practice what they preach?