Conservatives and libertarians have a tendency to assume that every single thing you do is a personal choice, and a consequence of this thinking is that you assume that you choose to participate in systems of oppression. This is the reason why a white male may have a hard time understanding that he contributes to (or benefits from) racism or sexism. For these people, you choose to be racist or sexist. So the accusation is met with push-back. Unfortunately, this assumption of responsibility is an enormous barrier to the fight for racial and gender equity. You’ll deny the problem if you think you’re the cause. It seems that it’s more salient for them to believe they might be personally at fault for limiting other peoples’ freedoms than to believe that they just can’t control everything.