Best investment that I ever made was a sewing box. Cost about $35 or 40 is from our local sewing shop and keeps everything in the one spot.

With supplies. I found that the easiest way was to get what I needed for a project and then keep the spares in the box. And then before going shopping for the next project look at what you already have IYKWIM. There is nothing worse than have 100 buttons that were cute (and sometimes very exxie) and nothing to do with them After a few projects you have several different thread colours, scraps of fabric and leftover buttons etc.

Also, spare sewing machine needles are an essential. Get the right ones when you get the machine if you aren't sure.