The retainer can be pretty snug in there, but will come out without taking the rest apart, may take some soaking and lots of tapping with a brass drift or plastic mallet, but should come free, just don't bust it, you might make a small hook type, slide hammer puller and hook the edge near the shaft, once it is a little loose.
Thanks for that detail. Did not want to keep trying only to find out that that was something holding it from behind.
Will persevere.
randy