The count is correct based on how you have it set up - it counts how many linked records are in a particular field, and you have one linked record in each field.
Linking records to the same table does not work the same way linking records on separate tables does. When you link records between separate tables, a reciprocal field is created on the table you’re linking to, and the relationship is bidirectional. When you link to the same table, there is no reciprocal - the field link only goes in one direction.
One way to get the result you’re looking for is to create a second table, called something like "Families". You can do this most easily by editing your existing linked record field and pointing it at a new table. This will take all the existing values in that field and add them to a new table as a unique list. On this new table, a count field should function as your expecting.
