This is a particular and very small set of "affluent dudes". Except for Elon Musk who is way, way out of everyone else's league affluence wise, they are pundits and entertainers. Most affluent dudes are focused on providing products and services while making a tidy profit. It is hard work because they are competing with other affluent dudes, even within their own company, to keep on being an affluent dude.

If you ever get a chance to fly first class on an airliner you will find that it isn't all movie stars and fashion models. Most of your fellow passengers will be affluent dudes in business casual. As soon at the plane takes off the laptops flip open and the tapping begins with the under the breath cursing. They are drinking coffee not champagne, because they are working. Headed to Vegas? It's all meetings at a conference that starts again at 7 AM the next morning. No swim up blackjack or shows.

Sure most of those affluent dudes certainly have a nice house or two, some luxurious personal possessions, and maybe even a nice family, but they are running on the treadmill just like everyone else. Don't expect any sympathy from them.