lubricate it w/ baby oil before pushing the pin out.
...and i don't believe that pushing it out stresses the plastic, in fact it relieves the stress coz after you take the pin out, some of the whitish stress marks disappear. what stresses out the plastic is the knurling that's bulging the plastic from inside the hole. but yeah, having knurled pins doesn't equate to a 100% chance of it breaking/cracking. it's all up to you if you want to do it or not.