It's got nothing to do with the intellect of those lifting them, but rather it's all about the origins. In the 1700s, tired of using heavy rocks to lift weights, people invented the precursors to that modern gym staple, the dumbbell.
They took two church bells and placed a rod between them. This would be fine, but you also have to take into account that bells have clappers that move around and make noise. So they took the clappers away, which turned the bells silent. That's why they're called dumbbells!