I think you should do whatever your baby wants you to do for sleep time, as he gets older he'll let you know what works for him and what doesn't.
i stopped swaddling evie's arms when she could pull them out and there was just no point wrapping them anymore. now she can get her legs out too but i still wrap her legs more as a matter of course and cos she knows it sleep time when i do, rather than it being for any other reason
I dont think it's something for you to be worrying about though - especially if it means hours and hours of tears!