The greatest used class of compounds used in dyestuffs are azo compounds. There are azo dyes that have been found to be potential carcinogens. These were first regulated in the EU by Germany and the list is commonly called the German banned dye list. If a product has not been made using these compounds, it is sometimes called azo free. It is misleading, since the number of dyes on that list is very small as compared to the total number of azo dyes which are still used.
In short, some azo-based dyes (Azo dye group III A1 and A2) shed carcinogenic aryl amines as the garments are worn (they contain metallic elements) creating health risks, according to the Federal Institute for Risk Assessment. Azo-dyed garments may also contain toxic chemicals such as chlorine bleach.
For example, your vibrant yellow sweater may contain an azo dye with 4-hydro-xyphenylazobenzene, or 1,2,4-trichlorobenzene (and you didn't have to pay extra!).
Given that baby's skin has a natural sensitivity and an undeveloped derma, the health risk of exposure to aromatic azo dyes is very high. The key benefit of azo-free dyes is the removal of the metallic component, which can create a textile-induced chemical sensitivity, or worse.