The long strange trip
The remote centralized government of the Imperium is possessed of great industrial and technological might, but it is also unable, due to the sheer distances and travel times involved, to exert total control everywhere within its starspanning realm.
Extensive home rule provisions allow planetary governments to choose their own forms of government, raise and maintain their own armed forces for local secunty, pass and enforce laws governing local conduct, and regulate (within limits) commerce. Defense has been a cooperative effort of local indigenous forces, which is stiffened by Imperial naval bases manned by extremely sophisticated forces.
Conflicting local interests have often settled their differences by force of arms, with Imperial forces looking quietly the other way, unable to intervene en masse. Imperial forces have been able to operate as a police force so as to not jeopardize their primary mission of defense of the realm. Only when local conflicts threaten either the security or the economy of an area have Imperial forces taken an active hand, and then with overwhelming speed and force.
In this environment, the soldier for hire has found active employment. The tolerance exhibited by the forces of the Imperium have made the hired military force a practical matter. Imperial policy has allowed the marketplace to provide military force to those who can afford it; one line of thought has been that a hired military can be more effectively controlled if it gets out of hand, as opposed to nationalistic or politically motivated military forces.