There is a hereditary factor. And hormones. I was in my early 20s when it started and i've never been overweight and a gf of mine was exactly the same. My dad is, well, male, and was in his 50s.
If you are having it removed then you will need to adjust your diet initially simply because with your gall bladder gone you can't tolerate fatty foods. But eventually you get used to eating everything again.