Here's some pictures of what I mean, I didn't have the right diameter keyed shaft but you get the idea. If you go this route make sure you drill the hole as close to center as possible, if you have access to a lathe you can drill it on that would be ideal.
Sent from my iPhone using Tapatalk