A National Osteoporosis Society survey found a fifth of under-25s are cutting out or reducing dairy in their diet.
It said it was concerned many young adults were putting their health at risk by following eating fads.
Cutting out dairy can be healthy if enough calcium is consumed from other sources, such as nuts, seeds and fish.
The charity surveyed 2,000 adults, including 239 under the age of 25 and 339 aged 25-35.
Milk and dairy food, such as cheese and yoghurt, are important sources of calcium for strong bones.
The charity’s survey suggests that many young people seek dietary advice from bloggers and vloggers on the internet
Although some of this advice can be good, the charity is concerned some people become too restrictive about what they eat.
A recent Food Standards Agency survey found that nearly half of 16-24 year olds said they had an intolerance to cow’s milk and dairy products, compared to just 8% of over-75s.
Yet only 24% had actually had their condition diagnosed by a doctor.
Prof Susan Lanham-New, head of nutritional sciences at the University of Surrey and clinical advisor to the National Osteoporosis Society, said: Diet in early adulthood is so important because by the time we get into our late 20s it is too late to reverse the damage caused by poor diet and nutrient deficiencies and the opportunity to build strong bones has passed.