You need to make the right decision for you. There is no right or wrong.
Just for the record I am an only child and I love it still as an adult as much as I did as a child. I get along wonderfully with my parents and my mum says that she is glad she only had one (she couldn't have any more) because she spent so much time doing things with me as I grew up.
You sound like you are in a good place stay true to what you want not what society tries to tell you is "right".