I'm so glad I kept it modest. In '99 I paid $75k for my 1k sq ft. house. Built a garage in '04 bringing the total up to $100k with a construction loan. Refinanced last year at 4% and my mortgage payment is stupid-cheap.
Of course, if I had lost my job at any point during the last 10 years, I might've found myself in a similar situation even with a small cheap house.