You cannot do this with formula fields. You can do this with scripting.
Counting all the hours which a record does not occupy is not any easier than counting hours which a record does occupy. Your technique may also need adjusting if begin/end times ever are not exactly on the hour.
The exact approach will depend on your base structure, but I would approach writing this script by finding overlapping time frames and collapsing them into one bigger time frame. Once all time frames have no overlaps, then adding up all the time frames.