Fragrance. Is using perfume or cologne good or bad?

More and more ppl seem to be allergic...yet when a fragrance can be perfect. I've returned a fragrance when someone close to me said it made them ill...but women - and men - love their scents... Should we be on guard for other people or not?