Two Fathers and Two Sons Leaving Town

Classic Problem
Two fathers and two sons leave town.

This reduces the population of the town by three.

Solution
This is an instance of the riddle Two Fathers and Two Sons who are Three People.

The $3$ people involved are grandfather, father and son.