"How do you know your family was poor?"
"Every  time I passed someone in town,  they would say, "There goes Joe. His poor family!"