Have you re-checked your points gap? You should actually have a dwell-meter in your stable of tools as well. If you're going to keep points you're going to find it very handy.
No change in the timing when you pull the hose off is normal. If you've got the hose connected to the correct port on the carb, then there will be no vacuum signal at idle. It only comes in as you open the throttle.
With your timing light, you should be able to see the timing jump up just after you've started opening the throttle. Verify that this is happening.
Verify with the light too, that your timing event is relatively stable. If the marks are jumping around, you have something wrong/loose inside the distributor.
Incorrect dwell and/or a faulty condenser can easily create a hard cold-start issue.
The timing itself is very forgiving between the stock 6 degrees, and up to 12 or 13 degrees advanced.
And remember, the idle goes down as you retard the timing, and goes up as you advance. And every time you mess with the gap/dwell, you change the timing.
With some distributors too, any change in idle speed will make the mechanical advance move a bit. Most EB distributors won't have that issue, as their weights are a bit heavier and don't come in until a few hundred rpm later, but it's still common to see after all these years.
Just a few things to think about.
And now that you have it idling so you can mess with it, start looking all over again for any vacuum leaks. Use water or carb spray around every port, seam, gasket and seal that you can think of.
Just watch out for new paint if you use carb spray!
Paul