The progress made against AIDS in the last decade has been extraordinary. In the last decade, close to 10 million people in developing countries have been given antiretroviral treatment. In sub-Saharan Africa, 60% of people in urgent need of such medicines now have access to them, something unthinkable just a few years ago.

As a result we are seeing steadily declining rates of AIDS-related deaths and of new HIV infections: a third fewer in 2013 than in 2011. Of the 25 countries that have achieved a 50% reduction in the rate of new HIV infections in the last 10 years, more than half are in sub-Saharan Africa.

This success has been possible through a combination of factors: advocacy and community mobilisation, driven by groups such as Act-Up, Treatment Action Group, and their many counterparts; courageous and compassionate political leadership; remarkable advances in science and technology; and tremendous innovation in delivery, which has enabled new drugs and tests to reach parts of the developing world at unprecedented speed.

It is a global response that has treated AIDS medicines as a “global public good,” funded through a combination of domestic and international financial support and made available at the lowest possible price.

The response to AIDS has generated extraordinary hope. Hope that it may indeed be possible to end major epidemics in poor countries. Hope that we see reflected in headlines that predict “the end of AIDS,” or the possibility of an “AIDS-free generation” in our lifetimes.

Sadly, this is not the whole story. This progress contrasts with the failure to respond effectively to HIV among so-called “key affected populations”—people who inject drugs (PWID), sex workers, men who have sex with men (MSM), and prisoners, as well as migrant and transgender populations, among whom the response to AIDS has often been a story of indifference and neglect.

It has long been assumed that key affected populations represent a modest share of the epidemic, and that the prevalence of HIV among them is largely confined to countries with low level and “concentrated” epidemics. These key populations, however, represent most of the affected persons outside sub-Saharan Africa, and an increasingly recognised share of infections in urban settings within sub-Saharan Africa.

In low and middle income countries, men who have sex with men and female sex workers are 19 and 13 times more likely to have HIV, respectively, than the rest of the population. Key populations and their sex partners account for as much as 51% of new infections in Nigeria, 33% in Kenya, 28% in Mozambique, 80% in Morocco, 47% in the Dominican Republic, and 65% in Peru. MSM alone account for more than 33% of new infections in China, and projections indicate that MSM could make up half or more of all new infections in Asia by 2020.

We have also failed to control the epidemic among gay men in the northern hemisphere: the rate of new infections in the US has remained stable for a decade, as it has in France and the UK.

Compared with the general population, HIV prevalence is 20 times higher among people who inject drugs, 35 per cent of whom are infected with HIV with prevalence up to 70 per cent or more in some communities in Asia and in the Russian Federation.

The lack of progress in tackling HIV among people who inject drugs in eastern Europe—where injecting drug use accounts for more than a third of reported cases—translates into a growing regional, and now possibly generalising, epidemic.

This is despite the fact that a clear understanding exists of what needs to be done. Achievable coverage levels of needle exchange programs, opioid substitution therapy, and antiretroviral treatment could result in large decreases in HIV incidence and prevalence rates in settings with high prevalence among people who inject drugs.

Key affected populations tend to be concentrated in cities. For many MSM and transgender people, for example, cities offer new freedoms for expressing sexuality and identity. At the same time, people in cities experience social dislocation and insufficient economic opportunities relative to demand. This produces contexts and patterns of vulnerability to HIV, including sex work and drug use, fuelled by inadequate access to prevention and treatment, as well as human rights abuses.

We ought to be gravely concerned that if we continue to fail in responding to these epidemics, there is a serious risk that AIDS will become increasingly concentrated everywhere.

This is for two key reasons.

Firstly, epidemics such as those that are growing among gay men and drug users in African cities are tremendously difficult to tackle because these populations are so marginalised.

Secondly, these epidemics are occurring  in the current context of economic transition and growing urbanisation accompanied by increasing economic and social disparities, all of which affecthealth and access to health services, including, but not limited to, HIV prevention and treatment.

And so it is clear that our future efforts to deliver global public goods, including HIV interventions to key affected populations, are tremendously dependent on addressing the social, economic, and political determinants of health.  These include the well known drivers of health inequities: lack of educational and economic opportunity; lack of appropriate, accessible health and community services; and lack of participation in decision making around health by those who are most affected.

The response to the AIDS epidemic has certainly come a long way in the past three decades, but we urgently need a shift in the collective mindset to change the course of the epidemic for those key affected populations. There is a crucial need for a renewed health agenda to continue to drive equity, human rights, and hope to those who need it most. If we do not deliver the right response, we will fail to deliver an end to AIDS globally.

Competing interests: None declared.

Michel Kazatchkine is the UN Secretary General’s Special Envoy for HIV/AIDS in Eastern Europe and Central Asia. This is an edited version of the Alison Chesney and Eddie Killoran Memorial Lecture he delivered this week at the City Health 2013 conference in Glasgow.