The future of the intelligence and security services could be undermined unless there are major reforms to make them more transparent and accountable, Nick Clegg has warned.
The Deputy Prime Minister set out a series of changes he wanted to see, but claimed he was being blocked by his Tory coalition partners from introducing measures to bring the work of Britain's spies into the open.
The proposals follow a series of damaging revelations from US whistleblower Edward Snowden about the UK agencies' internet monitoring capabilities.
They exposed details of major data-harvesting operations by government listening post GCHQ and its US counterpart, the National Security Agency.
Mr Clegg said respected security think tank the Royal United Services Institute (Rusi) had agreed to carry out an independent expert review of "big data" and privacy issues in a bid to secure consensus on changes, with a report due after the next election.
In a speech at Rusi Mr Clegg said: "I have, and will continue to have, the highest respect for the professionalism of our agencies. They conduct their work in good faith and the scope of their activities is constrained by laws, including the Human Rights Act, which require them to operate in the public interest.
"That being said, I am concerned about two things which risk undermining their reputation and by extension their long-term effectiveness.
"First, our current framework assumes the collection of bulk data is uncontroversial as long as arrangements for accessing it are suitably stringent. I don't accept that.
"I agree that strong access controls are vital to prevent employees from going on fishing expeditions once a store of data exists. But the case for collection itself has to be made, not assumed, and it must be shown to be proportionate.
"This is particularly true in a world where, as we have seen in recent weeks in the context of NHS data, people are starting to question the uses to which their data are put and to demand that government does more to obtain their informed consent.
"Second, the public interest cannot be democratically determined behind closed doors. Decisions exercised in obscurity cannot be relied on to command public confidence when they come to light.
"It is not enough for the agencies to claim they accurately interpret the correct balance between privacy and national security, they must be seen to do so, and that means strong, exacting third party oversight."
He said the review, led by Rusi's director general Professor Michael Clarke, would look at "proportionate use of bulk data but also the question of access to communications data held by private companies too".
Setting out a series of proposals for reform, Mr Clegg suggested:
:: annual transparency reports detailing the requests for data which government makes from internet service providers and telecoms firms;
:: a website for information about the work of the agencies, with the suggested address www.surveillance.gov.uk;
:: allowing appeals against decisions of the Investigatory Powers Tribunal and publication of the reasons for rulings. At present appeals have to go to the European Court of Human Rights;
::a single watchdog - known as the Inspector General for the UK intelligence services - replacing the present Interception of Communications Commissioner and Intelligence Services Commissioner.
Mr Clegg said: " To start with we should introduce more transparency and openness where we can do, without jeopardising operations.
"Secrecy is essential for the agencies to conduct their operations, but if blanket secrecy becomes an unthinking, default response then public trust will suffer.
"The assumption should always be for openness where possible, secrecy where necessary."
He also called for substantial reform of the parliamentary Intelligence and Security Committee (ISC), claiming it was "on probation" in its current form.
"The public face of agency accountability, the parliamentary Intelligence and Security Committee (ISC) has been criticised for being overly deferential, and I think anyone who saw the recent public hearing with the three agency heads would agree that the pre-cooked questioning, whilst welcome as a new innovation, was no match for the kind of raw grilling the US agencies receive in front of their congressional oversight committees," he said.
"The danger, whether in perception or in reality, is that we have a closed shop - especially when the chair of the committee previously served as one of the sponsor ministers for the agencies in Whitehall."
The committee is currently chaired by Tory former foreign secretary Sir Malcolm Rifkind, but Mr Clegg said in future the ISC should be chaired by an opposition MP " to avoid accusations that the committee is too cosy with the government of the day" and its hearings should be held in public wherever possible.
Mr Clegg acknowledged that his proposals would not be implemented by this Government because he had failed to persuade his Tory coalition colleagues of the need for change.
But he said that the next government would not be able to "duck" the issue and indicated that the Liberal Democrats would raise it in any coalition negotiations that might follow the 2015 election.
"W hilst no material changes are going to occur during the remaining months of this parliament I've absolutely no doubt in my mind that in the early stages of the next parliament whoever is in government, whoever is represented in parliament, will need to grapple with these changes," he said.
"That's why the work of this Rusi panel, the work of the ISC, the wider public debate, is so important to prepare the ground for what I think will be a very important set of decisions to be taken in the early stages of the next parliament.
"At that point no party, whatever they think now, will be able to duck those choices."
He added: "The strain between the existing regulatory frameworks we have got and the development of technology, the gap is now becoming so great I really don't think it's an issue of the argy-bargy between political parties or the line-by-line negotiating between political parties, it's just the next government - whatever its composition - just has to face up to this and do something about it.
"Personally I think it would be extraordinary if it wasn't reflected in the legislative programme or the policy programme of any future government after the election next year."
Shadow home secretary Yvette Cooper yesterday called on the Government to engage in a serious public debate about issues surrounding privacy, data and the private sector.
Labour wants new powers for police and security services to crack down on cyber crimes but only with extra checks on how crime agencies are using sensitive data, she added.
Mr Clegg said he was "pleased to see Labour has been reflecting on its poor record on civil liberties when in government and that the shadow home secretary now says she is in favour of reform".
Emma Carr, deputy director of privacy campaign group Big Brother Watch, said: "The Deputy Prime Minister remains the only party leader to address these major issues head on and he should be applauded for doing so.
"With the shadow home secretary and the Deputy Prime Minister recognising that the existing legal framework governing surveillance is not fit for purpose, it is clear that no new snooping powers should be considered until these critical issues for our privacy and the digital economy have been addressed.
"The independent review is an important part of ensuring that our laws are fit for the internet age, oversight is as strong as it needs to be and as much information as is possible is made public.
"It is abundantly clear from the experiences of countries like the US that more can be done to explain to the public how powers are being used, without compromising security."
Liberty's director Shami Chakrabarti said: "It is good to see politicians waking up to concern about privacy in particular, and hopefully our rights and freedoms more generally.
"But no political party has a monopoly over this and the reforming rhetoric of the pre-election trail is soon diluted in office. Parliamentary responses to revelations of blanket surveillance have so far proved embarrassingly inadequate so, sadly, we must still look to the courts to kerb gross abuse of power by the secret state."