Could a UK exit tax apply to someone who only spent a few years in the UK?

Yes. Exit taxes in other countries generally apply as long as you have been resident for a minimum period, often a number of years out of the previous decade.

Any UK system might have a minimum residence requirement, although this has not been set out. In principle a long-term foreign resident who moved to the UK for work and then left could face the same rules as a lifelong British resident, provided they met any time-based tests in the law.