If you're concerned, perhaps talk to your midwife or doctor. Generally speaking, it's only in certain situations that sex is not advisable. Otherwise it can actually be a really good idea
You might actually find it's easier to do it now, rather than after you have a baby. But ultimately, just do whatever you're comfortable with. Neither of you will be seriously harmed if you don't have sex.