Had that happen to me when changing the rear tire.
The workshop had to continuously spray carb cleaner to get the rust out. It's due to rain getting in there, perhaps insufficient grease on that shaft.
I think it took about 30mins of spraying carb cleaner and tapping with a rubber mallet to finally getting it off.