The last time I did one I made a tool. I used a piece of 1/2" round stock and welded a big washer that I cut in half on the end. I held it toward the top and pushed it in a ways and let it drag the bottom as I pulled all the crap out. Like you said, sand dirt and all kinds of good stuff in there.
I had the carrier out and was replacing the seals, thats when I found all the fun stuff. After I got most of the chunks I used a rag with diesel on it and pulled it back through from the inside of the housing to the knuckle a few times. I used a dry rag the last time or two and called it good.
Not a fun job but definitely needs done when its plugged up.
Good luck!