Should affirmative action even be legal?

It just seems wrong to let race or gender play a part in selecting candidates for jobs and school. It was useful at one time when racism and ism was rampant but now I don't see the need for it. I would support measures to give consideration to those who grew up poor and underpriveliged but not based on race or gender. What are your thoughts?