These can be a super PITA! In my experiences, it's not the brakes catching the drum but the from center hole is too tight around the axle lip it slides over. When this is the case, a drum puller or hammer does little good. Prying on it with a large screw driver will usually just bust the drum edge. I've found the best way is to drive the lugs through(you've already done that) Then put pressure on it with a screwdriver or bar and twist the drum off like it was a giant nut. It takes a few rotations to get over the tight spots, but it usually works. After doing this three times on my 76, I ground the axle lip down a fuzz(cool tech. term), to keep it from sticking the next time.