Yeah - noone's business but yours! How you feel is how you feel- and how you feel may also change- and that doesn't give people the right to say 'I told you so'....

DH and I both come from families of 4 siblings- are have close relationships with our brothers and sisters. I have always loved the busy-ness and action around our dinner tables at Christmas, Birthdays etc, and thought I would have at least 3 kids. However, after I went through labour (in the early sleepless days) I could not believe how our parents had done it 4 times!!! I thought they were absolute crazy martyrs!! I couldn't conceive of having more at that stage.
However, then about 4 months on- I developed that 'pregnancy amnesia' that everyone tells you about. When DS1 reached 6 months I was well and truly thinking about when we would start TTC again- lol.
Now I am back to wanting at least 3 again- but you know what? My answer when anyone asks me "how many kids do you want?" or "how many kids will you have?" is always the same...."I don't know". Because really, who really knows? I could be hit by a car next week and become a paralysed from the neck down in which case I'll only have 1. I know that sounds a little morbid but its reality. I am always weary of trying to 'construct' our future too much and getting set on a particular number or thing to make you happy or complete. Truthfully, we would feel complete and happy with 1 child, and we would feel complete and happy with 7. What will happen over the next 10 years is unknown to us but written in the stars!!!
You are free to only desire 1 child- and people have no right to judge you. You are also free to change your mind later on and want 5.
I recommend telling people that you are not a fortune teller and cannot predict the future. Just enjoy your precious bub to be, and all will be revealed to you with time.