I actually thought that the preposition (in or with) after "luck" would be necessary.If I wanted to use a preposition, I'd choose "with". We can wish someone "Good luck with threading that needle".
Yes, that's completely normal. In both cases a verb follows. threading and finding.So far I have had no luck finding a job.
In this case it's a different construction. It's not followed by a verb. It's more of a place/location/things (your new job) so it needs the preposition to describe the relationship.- We’d all like to wish you luck your new job.