My DS was always most active when I was trying to sleep!!! I definitely think the movement during the day puts them to sleep. I wouldn't worry- I remember thinking that on SO many occasions. But the fact is, babies that small spend much more time asleep than they do awake- and its the same when they come into the world!
I'm sure you're baby is fine. There will come a time (7 months +) when you will begin to feel every move your baby makes. For now your bub still has plenty of room in there so it is likely you will only feel he or she move occasionally.