SB Lars,
You should remove the firing pin stop befor trying
to move the sight.
Field strip your slide,remove firing pin stop,firing pin and spring, 80 series plunger and spring, extractor. secure in large smooth jaw vise w-padding
to protect finish, leather-rubber-layerd masking tape.
Loosen rear sight set screw,witness sight position to judge movment,apply min two layers masking tape
to sight or face of punch-nylon-brass-steel?
I like a large hammer 16oz , hold punch firmly and squarely
to sight, strike one time sharply to break sight loose.
IF NOT "REMOVE AND RE-APPLY FRESH TAPE" REPEAT PROCESS.
bets luck JB