What Would The Tech Requirements For Metadata Retention Be?

The new Federal Government push for metadata retention has got off to a very rocky start, with government ministers offering nonsensical, incoherent and inconsistent explanations of what is proposed. Assuming that anyone in the Coalition ever manages to come up with a coherent explanation of what is proposed, what would the technical and legislative requirements of such a system be?

IP picture from Shutterstock

According to communications minister Malcolm Turnbull in an interview with the ABC’s AM today, the only data which ISPs would be required to store under the regulations would be the IP addresses associated with specific customers when they connect. That data is already retained by ISPs, though not necessarily for the two-year period that has been suggested. (It’s also very easy to conceal that address, but that’s another story.)

However, we haven’t seen any of the proposed legislation. We’ve seen Attorney-General George Brandis demonstrate he has no idea about what might be involved, and we’ve seen Prime Minister Tony Abbott apparently unable to decide whether the proposal would involve tracking browser usage history or not. We’ve also seen that all this data would apparently be available to security agencies without a warrant, which raises significant questions about judicial process and privacy.

The Internet Society of Australia earlier this week outlined a list of 10 questions (note: PDF link) that need to be addressed when we see an actual policy, rather than incoherent, incomplete and self-serving soundbites. In fact there are considerably more than 10 questions in the list, and they’re all worth considering. We’ve reproduced those questions below with annotations.

The document begins by noting recent criticisms of the implementation of the National Broadband Network (NBN) and suggesting that these problems could easily occur without metadata retention:

Accordingly, the Internet Society of Australia expects the Government to ensure that the design and implementation of the metadata retention proposal is not rushed, chaotic or inadequate, by requiring a rigorous business case and/or regulatory impact assessment process which takes into account the costs and risks of the proposal across industry and the economy as a whole, as well as the direct costs to the Commonwealth budget and risks to the Commonwealth. We also recommend that the Government conduct a full privacy impact assessment of the proposal in accordance with the Office of the Australian Information Commissioner’s guidelines, in addition to any usual parliamentary processes to scrutinise and improve legislation before it is adopted.

To be blunt, this is likely to be a challenge for the government, which has been keen to criticise projects such as the NBN but hasn’t actually done much to produce cost-benefit analyses for its own proposals. It’s also not evident how this approach will be balanced with its stated aim of reducing red tape (and of balancing the introduction of new regulations with reductions elsewhere).

1. What is the definition of metadata to be retained?

“If carriers or other organisations are to be obliged to retain metadata, they need to know what metadata is to be retained. The scope of the data required to be retained will have significant impact on the cost and risk in implementing the proposal. Is it only Internet connection duration and location information, such as that from authentication systems? Is it IP packet headers, or a subset of the information contained in the packet header, or the full content of some of the packets, for example the contents of the packets which include email subject headings? Will information about the content of the packets themselves be required to be retained? Must the metadata of every packet be retained or only session information?”

Throughout this week, we’ve heard contradictory definitions of what metadata will be retained. This is a vital question, but evidently not one that there’s a clear answer to, Turnbull’s attempts notwithstanding.

2. Which entities are required to retain metadata (Retention Entities)?

“Will it be restricted to only licensed carriers transmitting information across the public Internet? Will organisations which operate private internal IP networks or virtual private networks be required to retain the metadata of information passing across their private networks, or only if and once the communication leaves the private network to the public Internet?”

We’d also like to know how the legislation proposed to deal with VPNs, anonymising technologies and people who roam from overseas carriers.

3. Whose metadata is required to be retained?

“Is it the metadata of all individuals, companies, media organisations, members of parliament, political parties, governments and agencies (including the law enforcement and national security agencies themselves)? Will it apply to the metadata of communications by autonomous devices, like smart meters? If there are to be exceptions, what is the basis for those exceptions and how will the exceptions be implemented in practice?”

Tracking the IP addresses of anything on the “Internet of Things” would dramatically increase the amount of data to be tracked, especially with the predicted explosion in smart devices.

4. What method of metadata retention must a Retention Entity employ?

“Will it be sufficient for Retention Entities to maintain records in a large range of devices across their networks or will the metadata need to be centralised into a single server or data centre? If so, will the centralisation need to occur in real time (which might considerably increase the network overhead and thus require capacity upgrades across the entire network) or can it be batched and transmitted in periods of lower network traffic? If so, how frequently must it be batched and transmitted? What will be the consequences of failing to do? What format is the metadata required to be collected and stored in? Will the format be standardised or different for different types of communications and storage medium or vendor equipment? What minimum level of security must the Retention Entity establish and maintain in relation to retained metadata? Will a Retention Entity be restrained from outsourcing and/or offshoring the performance of its retention obligations? If not, does the Retention Entity remain primarily liable for those obligations?”

These technical questions haven’t yet arisen much in public discussion, which has mostly focused on privacy and cost implications. For ISPs and telcos, however, these will be crucial questions.

5. When must metadata retention commence?

“The Government has indicated that there is an immediate serious risk to the Australian community from terrorism which metadata retention and access will assist to mitigate. Accordingly the Government will seek to implement some form of voluntary informal metadata retention arrangements by direct discussions with the communications industry prior to the introduction of legislation. However, implementation of a metadata retention system is likely to require adequate time to properly plan, design, implement and test before it ‘goes live’. Too rapid implementation is likely to:
5.1 unexpectedly incur or bring forward capital costs which have not been previously budgeted for or funded which may create short term competitiveness or even liquidity issues, particularly for smaller Retention Entities;
5.2 increase total costs of implementation due to uncertainties in the specification of the form of metadata retention required to be implemented and changing requirements through the various review and parliamentary processes; and
5.3 increase the risk of unidentified defects in design and implementation, thereby increasing the total risk of project failure, loss or disclosure of retained metadata and future requirements to incur additional costs of rectification.”

Frankly, it’s hard to see how a short-term plan can be implemented when the relevant metadata still hasn’t been defined and meetings with telcos have only just begun.

6. Who will pay the cost of metadata retention?

“Will there be some public subsidy to private organisations to meet the capital and operational expenses of implementing and operating metadata retention? Or, will the cost need to be absorbed by customers and/or shareholders? If there is to be some form of public subsidy, on what basis will it be calculated and allocated between Retention Entities? What will the costs of operation of the subsidy system be and how will that be allocated between the public and private sectors? A practical mechanism may be to require relevant law enforcement or national security agencies to subsidise the Retention Entities’ capital implementation costs and then pay the true operational cost of each access request they make from their existing budget allocations. This would create a practical budgetary incentive upon agencies to restrict the requirements of (and thus cost of) metadata retention systems and the number of access requests to only the most important and to limit ‘fishing expeditions’.”

This is going to be a challenging question for a government which was determined to slash budget expenditure this year. The apparent inclination would be to insist ISPs pay the cost, but if they pass it on to consumers, that’s likely to cause a major outcry.

7. What authorisation will be required to access metadata?

“Will metadata be available only to law enforcement (ie Police) and national security agencies? What are the range of agencies permitted to seek access to retained metadata and the purposes for which they may seek access? Will it be limited to intelligence and policing agencies for counter-terrorism purposes or extend to ‘ordinary’ criminal or civil law enforcement activity. For example, will ASIC, local governments, the Victorian Taxi Directorate and the RSPCA continue to have the ability to access retained metadata for the enforcement of the statutes for which they are responsible, as currently? In what circumstances will a warrant or formal authorisation be required? Will that be an independent process? What oversight will be in place? What sanctions will be applied to individual officers who inappropriately authorise access? What sanctions will apply to agencies and officers who inappropriately use or disclose metadata which has been accessed? Will the Retention Entity be permitted to access its retained metadata for its own business (including billing and marketing) or other purposes? Will private parties to litigation (for example, unfair dismissal, breach of confidence or divorce cases) be able to demand the provision of metadata upon subpoena? Will metadata of, or held by, agencies be available under Freedom of Information requests?

The proposal was launched in the context of counter-terrorism, but subsequent comments from the Prime Minister and others suggest it will be available in any crime investigation context.

8. How long must metadata be retained and how will it be disposed of?

“Is the two year period foreshadowed by the government the specific, minimum or maximum period for which the data is to be retained? Will agencies be authorised to access metadata which is more than two years old? What obligations will Retention Entities have to ensure that retained metadata is disposed of and fully expunged after the expiry of the two year period?”

On one level, the storage period is arbitrary: one lesson of the information age is that it’s surprisingly difficult to delete data.

9. Who will bear the risks of metadata retention?

“It is likely that any ‘deep pool’ of metadata will pose an attractive target to hackers, ranging from the purely curious through the disorganised anti-social to organised crime and terrorist organisations themselves. As the Manning and Snowden cases make clear, no information system is ever completely secure, so there is a real probability that retained metadata will be accessed inappropriately or without authorisation, in a way that causes real personal and economic harm. Who bears the costs of that harm: is it the individual whose privacy is interfered with; the business who suffers loss or damage from the disclosure of its confidential information; the Retention Entity which is retaining the data; or the taxpayer through the government? What mechanisms, for example statutory indemnities or immunities, will be put in place to give effect to that risk allocation? What disclosure regimes will be in place in order to report such breaches?”

We haven’t heard a single word said about this. Guaranteeing security of data is going to be a major, major challenge.

10. What ongoing review and reporting of metadata retention will occur?

“Is the metadata retention intended only to be in place for the next five years, which the Director General of ASIO has identified as the peak risk period for returning jihadists, or will it be in place indefinitely? This will affect the way Retention Entities amortise any of their unsubsidised capital costs of implementing retention systems. Will there be a review of metadata retention? Who will conduct the review, by what process and when? What statistics and key performance indicators of the effectiveness of the proposal in achieving its stated aims will be collected, analysed and published to enable a review to occur?”

If the Government is serious in its intentions and mindful of its civic obligations, then all those questions will be able to be answered. If the whole exercise is a cynical exploitation of terrorism to try and introduce a major change to how citizens are tracked, that will always be evident. It will be interesting to see the answers.