The infant insert is only there for comfort and to make sure that the buckle sits low enough on their pelvis when they are little. You don't want the buckle to sit too high because it can cause damage to their organs in a crash. You want it To be sitting across their pelvis region so in an accident that area absorbs the impact. If the buckle is sitting in the right spot with the infant insert removed then it is fine to take it out.