I know several people that bought a large wooded piece of property, cut down a whole lot of the trees, milled them with a portable lumber mill, and then built the house entirely out of that wood, along with cinder blocks, roofing material, insulation, windows, paint and caulk, wiring and plumbing, and gutters from other places. Probably 90% of the material was from wood harvested on the site though.

"Normal folk, however, used local timber, bricks, and stones to build their homes."

That's simply not true. Prefabricated kit houses have been popular in the USA since the 19th Century. I lived in a 1920s Sears kit house in Virginia that had been shipped from Chicago. An older woman had seen them dragging it into place from the rail station with horses when she was a little girl.