You are not saving the seal so jam in a large, flat-tipped screwdriver under the seal lip and then twist the screwdriver. It'll come.
Forget the rubber mallet and screwdriver. Use a proper punch and hammer Hammers are hard. Punches are (relatively) soft. They are made for each other. Screwdrivers and bearings are hard. Hard against hard may cause one or the other to fracture and send metal chips flying about at high speed. Not good. Use a punch or something made of un-hardened steel (rebar for example).
Since you driving out the bearing against its inner race, your bearing are toast and have to be changed.
Slide the inner spacer side to side and hit on the inner race with the punch and hammer. Start out with a medium hit and observe how far the bearing moves. If it moves a lot, unlikely, then tone it down. If it doesn't move, more likely, ramp it up. Go from side to side so the bearing comes out "square". If you hit only on one side the bearing comes out sideways which will ovalise the housing's bore.