i wouldnt say its innapropriate at all she is his grandaughter after all he would probably like seeing her as for you i still dont see a problem from what you said the 2 of you are still close despite not being with his son anymore