Extra safeguards are to be placed around the taxman's controversial new powers to seize money directly from people's bank accounts, the Government has announced.

The Government has been consulting on plans to enable HM Revenue and Customs (HMRC) to recover cash straight from bank and building society accounts, including joint accounts and Isas, belonging to people or businesses who refuse to settle tax or tax credit debts.

It said today that all debtors who are being considered for this process will get a face-to-face visit from an HMRC officer, which will give them the chance to challenge and settle their affairs.

Those who pay up in full or agree to a repayment plan will not go through the process of having money clawed from their bank account.

Face-to-face visits will also help HMRC officers to identify people who are potentially vulnerable and offer them support. A specialist unit will be set up to deal with cases involving vulnerable people and a dedicated helpline will be set up to help people discuss any direct action to recover their debts.

Debtors who refuse to respond to correspondence from the taxman will also see their account put on hold for 30 days, giving them time to arrange payment or object, before any money is taken. This is an extension on the Government's previous plans, which would have involved putting the account on hold for 14 days before money was taken out.

People will also be able to appeal against HMRC's decision to a County Court on specified grounds, such as hardship.

David Gauke, Financial Secretary to the Treasury, said: "We're strengthening the guarantees we can offer taxpayers that the powers will only be used when debtors have consistently refused to talk to HMRC and settle their debts, and their use will be subject to the toughest scrutiny and oversight possible."

The plans, which were first unveiled in the Budget, aim to target what the Government describes as a small minority of "can pay, won't pay" debtors who owe at least £1,000. They are expected to bring in around £100 million a year.

But fears have been raised that the moves, which mean that HMRC does not have to go to a court before taking such action, will lead to people who are vulnerable or have been wrongly targeted having their bank accounts frozen and drive people to stash cash under the mattress.

The legislation to allow the direct recovery of debts (DRD) is expected to be taken forward as part of the 2015 Finance Bill.

It will apply to England, Wales and Northern Ireland only as Scotland's legal system already allows processes that allow HMRC to recover debts in a similar way to DRD.

HMRC estimates that DRD will apply to around 17,000 cases a year, with the average debt of those affected being £5,800.

The Government said that DRD will initially be applied to a smaller number of cases in the first year of its operation in 2015/16, to allow HMRC to start the process on a "small, targeted basis" and gain experience and feedback.

A full review of DRD, covering its implementation and impact on customers, will be carried out by HMRC after the policy has been up and running for two years.

Statistics will also be published on the number of times this power is used and on the number of appeals that are raised.

Only debtors who will be left with a minimum of £5,000 across their accounts would be made subject to a DRD, and only funds up to the value of what they owed the taxman will be put on hold under the plans.

Around half of these cases would involve debtors with more than £20,000 in their bank or building society.

The Government acknowledged concerns raised by some respondents, including the banking community, that the taxman's new power to dip into accounts could interfere with insolvency proceedings and potentially give HMRC an advantage over other creditors who are also trying to claw money back.

Banks were concerned that, at present, mortgages tend to be paid ahead of other types of debt.

Its consultation response said: "The Government is clear that this is not the intention of DRD, but acknowledges respondents' concerns that HMRC could inadvertently find itself in such a position.

"It is therefore committing to ensure that HMRC does not receive any advantage during insolvencies through its use of DRD."

Addressing concerns it had heard that some people may think their money is not safe and instead put cash "under the mattress", the Government said it will work to minimise any impact on banks and building societies and make sure all communications with the debtor are clear.

Andrew Tyrie, chairman of the Treasury Committee, said: "There's a lot for Parliament to scrutinise here. The committee made it abundantly clear earlier this year that prior independent oversight is essential.

"On the basis of the Government's consultation paper, published today, it appears that the proposed new right of appeal will take place before withdrawal of any money by HMRC.

"It also appears, however, that the money would be frozen, prior to any appeal.

"The committee will take a further look at these proposals. Safeguards for taxpayers are crucial."

Joanna Elson, chief executive of the Money Advice Trust, the charity that runs National Debtline, said the revised proposals "represent something of an improvement".

She continued: ""However, these new powers to raid bank accounts may still be subject to HMRC error, which is far from unknown.

"We remain concerned that this could cause further financial problems for people who are already in difficulty.

"It is crucial that anyone struggling to repay tax debts are signposted to a charity-run debt advice service such as National Debtline as early as possible."