I would try giving him "dinner" at lunch, and even boosting his breakfast - so that at dinner time he can have a banana for dinner and its ok!
At this age I think its just a stage, and that he will grow out of it.
Rivlas made a good point - food off mums plate is always better!
hang in there, youre going a good job hun!