That is a good look. in my opinion
Why not a nice looking snuggly warm top? You can always find something at a thrift store - I like salvation army store. Last year's fashions in decent condition are fine for me, I often get complements on things I wear.
I go for warmth, not how it looks.
Usually a long-sleeved tee shirt with a flannel shirt as a jacket.
I'm at the point in my life where I am more concerned about my comfort that what people may think.
I would wear a sweater made from a burlap sack, covered with Whoopie Goldbergs mugshot all over it....before I would pick being cold..