The outer bearing sits against a land in the housing. The inner bearing does not, it's close, but not quite. It rests against the spacer.
I just use a big screwdriver to push the spacer to one side. Now you can drive one of the bearings out. YOU HAVE TO DO THIS BY TAPPING BOTH SIDES ALTERNATELY so it goes out square. If you drive it out crooked/sideways, it will ovalise the housing and your new bearing will be loose.