David Cameron is facing a major backlash after it emerged he has put a man in overall charge of women's issues.
Although Nicky Morgan has been installed as Women's Minister in the wake of Maria Miller's resignation, Downing Street confirmed that new Culture Secretary Sajid Javid will retain final responsibility for the brief.
The number of female Cabinet members is also lower now than it was when the coalition came to power.
Mr Cameron failed to head off criticism about the lack of gender balance in his government despite promoting Ms Morgan and bringing in Andrea Leadsom as Economic Secretary to the Treasury.
While Ms Morgan will attend Cabinet, she will not have full voting rights - meaning the only female full members of Mr Cameron's top team are Home Secretary Theresa May, International Development Secretary Justine Greening and Northern Ireland Secretary Theresa Villiers.
In 2010 there were five women members of the Cabinet, although at that time no others had the right to attend the meetings.
Number 10 stressed that the mini-reshuffle meant there were four ministers rather than three with responsibility for women's issues - Mr Javid, Ms Morgan, Helen Grant and Liberal Democrat Jenny Willott.
But the aides confirmed: "Equalities and women is still in the Department for Culture, Media and Sport at secretary of state level."
They were unclear whether there will still be a Commons questions session focusing on women and equalities, and which minister would respond.
Gloria De Piero, shadow minister for women and equalities, said: "David Cameron's decision to replace Maria Miller with Sajid Javid means that there is now no full member of the Cabinet speaking for women.
"There are now just three women running Government departments out of a possible 22, demonstrating that when it comes to women, it's out of sight, out of mind for this out-of-touch Government."
Daisy Sands of gender rights campaign group The Fawcett Society said: "The women's equality brief has sat, variously, in the then Department for Trade and Industry, the Home Office and most recently the Department for Culture, Media and Sport.
"It now rests with someone who is not a full member of the Cabinet, though she will have the right to attend meetings.
"Ahead of the 2010 General Election, the Prime Minister pledged to make a third of his Ministerial List female by the end of his first term. With just a year to go, he is governing the country with a Cabinet almost 90% male."
The decision to give Ms Morgan ministerial responsibility for equalities was criticised in light of her opposition to gay marriage.
PinkNews publisher Benjamin Cohen said: "It is unfortunate that the new Minister for Women opposes the rights of some women, lesbians, to have the right to marry.
"I hope that in her new role, Nicky Morgan will represent all women regardless of the gender of the person they love."