Well, it's not right, but it is common.
Could be any number of holes that allow hot air through the floor or cowl or firewall, but if you're sure it's coming from the heater area, it's probably a combination of two things.
First is the heater TEMP control valve under the hood is not closing all the way. If it's an old one, this is common and a new one will, if not completely eliminate it, at least help.
The second is that the flapper valve inside the heater box is not closing all the way. The fact that you can feel air coming in is indicative of this. Either the flapper door (damper door) is not closing all the way, the seals are gone, or both.
So yeah, you might have to do a full re-seal, but check to make sure it's not just a simple adjustment or valve replacement.
Also, is your fresh air inlet hooked up, or are you pulling air from inside the cabin?
And since a lot of the control cables and knobs are sticky with rust, is it possible that your knobs are not pushed all the way in? If the controls are stiff at all, you should lube them to rule that out. A little WD-40 goes a long way here.
Good luck.
Paul