The ones that are an integral part of the cable or shaft - heater switch, choke, headlight switch, etc. have to be damaged to be removed. Either by melting them off with heat or breaking them into pieces.
As you noted, the others are held on with a small allen head set screw on the bottom. Those pull right off once you loosen the set screw.