Berwick ran out 3-2 winners in an exciting game at East Fife's Bayview Stadium.
They took the lead on 37 minutes through Paul Currie who curled the ball with his right foot into the top corner from just outside the box.
The day got worse for East fife defender Allan Walker put the ball into his own net to make it 2-0 going into half time.
In the second half the home side fought back to make it 2-2. The first came from the penalty spot after Nathan Austin was brought down by John Fairburn and striker Alan Cook stepped up and placed the penalty into the bottom corner.
Centre-back Steven Campbell then pegged Rangers back with a long-range strike.
However, Rangers substitute David Gold grabbed the winner in stoppage time.