Raider camps are really bad for that kind of delay.
Eventually it'll glitch or give up and move on to the rest of the queue. I'm going to guess it's some sort of pathing issue and AI order, like the game is contemplating something else elsewhere on the map and has queued up any further instructions from you, the player, as 'to be done when the AI actually gets that far.' Pretty sure the AI also hangs sometimes because one settler is too stupid to walk to his bed and the game flails because he can't get there so it keeps trying and failing to make him move.
If you have a lot of active settlements (and/or raider outposts), it can take quite a while for settlers to assign to things properly.
