I have to say I agree with general trend of this thread. Being that the baby is experiencing a different environment (as in at your place) he will need the cuddles to relax and come to some kind of terms with it, also when he goes back, the reason he he was probably so upset the first time was just having to re settle again in his home, not because you "spoiled "him or anything.
Hope it all works out, I think following your instincts cant be a bad thing.