You read my mind. I was thinking of riveting in stainless rivet nuts so I could use bolts. However, it would be pretty tough getting the rivets set at the two locations. I still may give it a try though.if you're going to re-rivet. maybe clean the holes and put some primer in there. just as a precaution.
maybe consider threading for a screw but that might be worse.
Surprised those holders don't come out easier.