A new method for testing the security of cryptographic protocols such as TLS

15 March 2023

Seeking to improve IT security, researchers from the PESTO project team have devised an “innovative and effective” method for identifying vulnerabilities in implementations of TLS, an essential protocol for securing data exchanges over internet. This method has already shown its worth, having been used to discover four new vulnerabilities, one of which was critical.

It is the reason for the “s” after the “http” in the address bar on your browser and the small padlock next to it. The TLS protocol is one of the most widely-used protocols for encrypting exchanges on IT networks, including for the internet of things (IoT). Used since the 1990s, TLS is used to establish a secure connection between two end-points, thus preserving the privacy and integrity of any data travelling through it, whether in the form of usernames, passwords, bank card numbers, telephone numbers or email addresses. However, this protocol is not invulnerable, and has been subject to a range of attacks. Security breaches of varying levels of criticality have been detected, including the much-publicised Heartbleed breach in 2014.

A 1st bridge between two worlds: formal verification and fuzzing
Seeking to “significantly improve the scope and detection capabilities of testing algorithm”, Lucca Hirschi and Steve Kremer, Inria researchers with the PESTO project team (a joint undertaking involving Inria and Loria), have developed “an innovative and effective” method for identifying TLS protocol breaches. Working alongside Max Ammann, a former Master’s student at Loria who now works as a security engineer, Hirschi and Kremer developed this method as part of ProtoFuzz, a Young Researcher project supported through a generic call for proposals from the ANR (the French National Research Agency). This involved two different strategies coming together, as Lucca Hirschi explains: “We were among the first to create a bridge between the formal verification of cryptographic protocols – an area of focus for PESTO – and fuzzing, an effective method for testing implementations, such as TLS implementations. Fuzzing involves generating random entries which are then executed on the target program, the behaviour of which is observed and fed into a feedback loop that can be used to prioritise interesting tests.” The researchers also build on a formally defined attacker known as “DY” or “Dolev-Yao” – the names of its creators -, which explains the name of their new method: “DY model-guided fuzzing”.

Breaches of different levels of severity in wolfSSL

The TLS protocol has multiple implementations, meaning that different programs implementing this protocols have been described in computer language. The researchers focused on two of these: “OpenSSL, which is widely used, and wolfSSL, which was primarily designed for embedded devices, which have low computation power”.  No new vulnerabilities were identified in OpenSSL, but four were in wolfSSL, one of which was judged to be critical, with a severity score of 9.1 out of 10 under the Common Vulnerability Scoring System (CVSS). “Such a breach could cause Denial of Service attack and potential leaks of sensitive information elsewhere on the server”, explains Lucca Hirschi. “Another vulnerability would enable ‘buffer overflow’ attacks. This bug in server implementation was found to have remote code execution (RCE) potential: an attacker could have used it to re-write the memory space in order to execute code remotely.”  The researchers shared their vulnerability report with the teams in charge of development of wolfSSL on 12th August 2022, in accordance with the ethical principle of responsible disclosure. The stakeholders agreed to spend three months working together to seek out solutions and patches, without which the breaches would be disclosed. If you’re reading this article, that means the collaboration was successful: “As soon as we notified them, the wolfSSL team was very responsive. On 28th October, all of the fixes were deployed.” The researchers’ work was submitted for review at a conference of the Institute of Electrical and Electronics Engineers (IEEE), an international association “dedicated to advancing technology for the benefit of humanity.”

Find out more

Read the original article in Pesto Team’s page