olympics: when is a sport not a sport?

I'm enjoying the Olympics. The sheer diversity of sports and athletic disciplines is staggering.
Last time the Olympics was on I remember having a pub conversation with some of the guys from Home (that's you Jim and Rich!) about what makes a sport a sport. The conversation came out of the fact that some of the events seemed a little silly (thinking now of things like that gymnastic nonsense when the gymnast throws a ball around or waves a stick with some ribbon on the end - stuff like that).
I seem to remember we put forward the idea that to be included in the Olympics a sport had to have an objective element - longer, faster, higher , the most goals etc. Something that was objectively measurable. So anything with a panel of judges and scores for artistic merit etc. HAD to go.
Off the top of my head that means that these 'sports' would no longer be in the Olympics - synchronised swimming and diving, trampolining, gymnastics etc.
Some of these are great to watch (the gymnastics is amazing) but I'm not sure about it as a sport.
Am I being a sports fascist?