I haven't been to Port Fairy for years (my Grandparents lived there). There's not a whole lot to do. There's whale watching from Warrnambool (30km away, but check the dates). Erm.... You can walk around the Island at Port Fairy where the mutton birds nest (its a very pretty walk, there are very sheltered shallow pools), rocks to climb over and the lighthouse (but you can't go inside). There is also the battery and cannons, surf beach, a more quiet beach and the pea soup. Further towards Portland there are the Crags (sandstone washed out rock formations, great photo ops). Not much to do in town, at New Years there always used to be the carnival, and joy flights out of the airstrip. Fishing charters and nice golf course too. Sorry, can't think of anything else, its been a really long time since I've been there.