Yes, that would make me very worried, I know it's hard to get into a dr on a Sunday but could you find a medical centre? If not, give one of the health hotlines a call, their details should all be in your babies book that they are given when born for health checks and immunizations etc.

Good luck, I hope your bub is ok!