Housing Minister Clare O'Neil has made a comment about housing that could infuriate younger voters and see more of them switch from Labor to the Greens.