You'd have a point if most of your calories weren't burned by a little thing called metabolism, often without lifting a finger.
1) Not everyone's metabolism (chain of chemical reactions in your body) is as quick as everyone else's. 2) A lot of people consume too much calories without burning them off. 3) Stress, in take of carbohydrates, salts, and liquids does affect how you weigh.
Well, no kidding, and calories are burned by your metabolism (which will often vary according to personal habits) until your body gets proper nutrition, then other calories you take in are excess. So if you take in 3500 and burn 3500, you stay the same. Why should the extra calories/fat be broken down without burning more than your metabolism already does? I hardly see how it's unfair is my point.