Nope you're doing it right, well that's how I was doing it anyways. It's hard to get on and off, but easy without them actually in the bumbo.....which I'm pretty sure the box says not to remove or put on the tray with the child in the seat..... that did not stop me trying however

There is no way I could get the tray off the clip on part once I put it together either.