It seems more terrifying that an invisible force is pulling you toward the centre of the earth instead of you just falling from something. amirite?

Actually, it's a common conception that the earth's rotation is attracting us to the earth (adds about 1/8th gm.) and not just the mass of the earth.
It's contraintuitive...

Misconception

You are not wrong. I have also pondered on this.