You both have to sign for the passport so that is all that you will need. If you and your child are listed on an airport watch list to stop you leaving the country you will be stopped but that only happens in breaks up generally... Sounds like you are still with his dad you just aren't traveling with him?