I say it's serious in that it is unusual (I'm unaware of anyone else complaining about that hole being worn) and could denote a problem such as being hit or improper assembly/disassembly. That 1/2" bolt shouldn't rotate or otherwise move once installed. I've had mine in and out several times and they still require a hammer and punch to get them out of upright.
Of course, that's not to say that it can't be worn (or that the next time I have to take mine apart, it might won't be worn!) and if yours is, I would certainly have a bush pressed in. I think I would have an aluminum bush pressed in. Given that the bolt should be tight in the hole (bush) I might worry that the steel bolt might "grab" a steel bush and when pounding the bolt in or out, it might tend to drive the steel bush with it. On the other hand I might be worrying about noting and steel would be fine. I would consult the person installing the bush.
I would also always use anti-seize when installing the bolt and I would always back it up when tightening or loosening the nut on it.
In any case, checking the condition of the hole should be easy enough and if you've done everything else, it would be a good next check.
Have you checked the condition of your stub-axles? I found one of mine was worn from the bearing (see attached picture). I would expect that you would notice something like that when you changed the bearings but it caught me by surprise.