You do have to fit the whole assembly. HIDs don't have filaments. They use an electric arc drawn between two electrodes suspended inside the bulb. The bulb is filled with Xenon gas at very low pressure.
The arc has to be struck. That's the first job for the ballast. It provides a 25kV spike to start the arc. After that they maintain a nominal operating voltage of about 90 volts. The current is very tightly regulated to avoided the arc either extinguishing itself or burning too hard, which would cause the lamp (proper word for bulb) to explode.
The principle is similar to mercury arc and SON (high pressure sodium vapour) street lights. SON lamps have been popular for street and stadium lighting since the 1980s due to their much better spectral characteristics over a mercury vapour lamp. Both are unsuited to automotive applications as the startup time is of the order of several minutes. Further, they cannot be switched off and then immediately back on again without a rest period which can also extend to a few minutes.
Whether HIDs provide you with better vision is debatable. They are, without question, visibly and measurably brighter. But the light that appears white is not actually spectrally continuous; rather it is comprised of a number of line spectra which together gives the appearance of being white.
This can lead to problems with visual perception. It's a topic I don't fully understand but I can see why light comprised of a continuous spectrum would be more pleasing to the human eye. After all, that's how sunlight behaves.
