Author: Anna Knack, Research Assistant, RAND Europe
As connected and autonomous systems enter the mainstream, policymakers are faced with new challenges to ensure citizens’ safety and security in day-to-day activities. Non-embedded software is increasingly available for smart systems and could potentially make the maintenance and security of those systems more difficult. Non-embedded software is downloadable on a device and does not come integrated into a product at the time of the product’s placement in the market. For example, apps or software updates on smartphones, tablets or personal computers are widely known examples of non-embedded software and can be produced by third-party developers to customise users’ devices.
Beyond the ubiquitous smartphone, emerging smart systems include driverless cars and smart home appliances. Smart systems integrate technologies such as artificial intelligence, sensors and data analytics that allow the system to determine how to act based on its surroundings. For example, driverless vehicles travel on existing transportation networks without the need for a human driver and can identify hazards such as children playing near the roadway and quickly determine how to reduce risks, potentially quicker than humans would react. Smart homes can be customised according to users’ preferences using connected and AI-based devices; one such example, Samsung’s SmartThings, can be connected to compatible appliances via a mobile app that will be able to tell users if there is a problem at home while the user is away. For example, if the user has left an oven on it can be switched off remotely with the user’s smartphone, or if a door is left open, the climate control will automatically adjust to conserve energy.
Apps and other non-embedded software are starting to become available for smart homes and autonomous or semi-autonomous vehicles. Amazon has invited third-party developers to create new software and capabilities for Amazon Echo or ‘Alexa’ - a voice-activated Bluetooth speaker. Alexa can be connected to apps to play personalised music playlists, smart home appliances to turn them on and off or adjust settings, and can even make purchases by accessing credit card details through user’s Amazon accounts. Car manufacturers like BMW and Volvo produce built-in apps for their cars with relatively limited, but expanding functions. Non-embedded apps such as Apple CarPlay and AndroidAuto are operated by plugging a smartphone to the vehicle. They use the car screen as a monitor, displaying an iOS or Android-style interface to access traffic news, navigation or music apps. Furthermore, non-embedded software such as Logitech ZeroTouch are already out in the market and can connect smart homes to cars, allowing users to command Alexa to make purchases, play audiobooks or entertain children with voice-based games.
Challenges to securing non-embedded software could jeopardise the safety of users and bystanders
With the Internet of Things (IoT), virtually every consumer device from smartphones to robotic vacuum cleaners will need software upgrades. Software and system upgrades are set to be delivered to increasingly diverse platforms, and reliability and quality assurance will become all the more challenging. More car manufacturers are likely to follow Tesla Motors and Ford by offering automatic and non-embedded over-the-air updates, similar to iOS updates on iPhones. This could mean that problems that would otherwise lead to recalls and inconvenient trips to the dealership could be resolved by a Tesla or Ford engineer remotely. The risks that a single flaw triggered by a software update or virus could create are yet to be empirically demonstrated, but there is a need to understand any potential risks.
Smart devices generate large volumes of data, but these systems also contain vulnerabilities that could be exploited or that could lead to users’ personal data to be accessed. For example, in smartphones and tablets, non-embedded software such as apps can allow external software developers access to personal data from the user’s phone. Data gathered by smart home devices is already being used as digital forensic evidence by the police to prosecute accused criminals, which demonstrates the usefulness of the data. Simultaneously, smart devices contain vulnerabilities that could make them a target for cyber-attacks. For example, in the United States, two hackers were able to completely override a driver’s control of a Jeep Cherokee on a highway as part of a controlled demonstration of cybersecurity threats in transportation.
If a mobile device is compromised by the user disabling some or all of its security features, this is often referred to as “jailbreaking.” Owners themselves may seek to “jailbreak” or access their own smart homes, driverless vehicles and other smart systems to gain control over elements that manufacturers have locked down for operational and security reasons. For instance, a few reasons owners might seek to jailbreak their own driverless vehicles is to customise their vehicle using non-embedded software to enter into “performance racing mode.” Accountability or liability over operational or security issues that arise thereafter, and that could lead to safety concerns for users and bystanders become increasingly complex to establish and safety issues become more difficult to foresee.
Designing automation into safety features can both help ensure the safety of users, but paradoxically can also obfuscate safety and security risks to users. The automation of a safety activity partly removes people’s awareness of it overall. The convenience of over-the-air updates, for example, has a trade-off – while connectivity removes the user from the act of watching updates take place, it also allows remote operation of the device, facilitating potential vulnerabilities through remote scheduled access to the user’s device onboard memory.
Modernising testing, certification and risk management systems can bolster the resilience of smart systems
Current safety analysis techniques are becoming increasingly less effective due to their incompatibility with today’s emerging software-intensive systems. New testing, certification and risk management systems need to be aligned to the emerging technological landscape.
There will be a need for human-in-the-loop control systems that incorporate human interaction and guidance with autonomous or semi-autonomous systems. Safety monitoring systems that combine the strengths of the automated system while allowing humans to reassume control when there are uncertainties in the environment could make for a better system. For instance, if a flaw in a software update for semi-autonomous or autonomous vehicles were to cause a risk of collision, human passengers should be able to regain control of the vehicle to possibly prevent the collision.
Moreover, smart systems will need to be equipped with intrusion detection systems. Unlike current intrusion detection systems that are designed for conventional computer systems, the security of cyber-physical systems is more complex. In the case of surgical robots, software attestation mechanisms are being developed involving secure communication protocols between the verifier, the telesurgical robot and the medical personnel. As attack vectors in connected and autonomous systems such as smart homes increase, network-level security and privacy control to detect intrusion of smart home software will be required.
There may also be a case for more systems-level testing, certification and risk management systems. Safety and security have traditionally been treated in isolation, with safety experts seeing their role as counteracting losses due to unintentional actions by actors without malevolent intentions, and security experts seeing their role as preventing losses due to intentional actions by malicious actors. This may be resolved by taking a common top-down, system engineering approach to security and safety where the overall role of the entire socio-technical system is considered as a whole. Such an integrated system may bring more efficiency to use of resources and may lead to earlier detection of issues in the development process.
Policymakers need legal and regulatory frameworks that are “fit for purpose” to address challenges surrounding product safety and security
Legal systems need to be upgraded. At present, there are few specific laws, if any, which impose responsibilities, designed for smart systems and non-embedded software. It can be slow for the law to adapt, and that can pose special difficulties for new and emerging technologies where high speed of development and deployment are common.
Laws need to regulate third-party access to users’ personal data if such access can potentially harm users. For non-embedded software such as apps, the UK’s Information Commissioner’s Office has issued “Privacy in mobile apps: Guidance for app developers.” Similarly, the EU has issued “Guidelines on the protection of personal data processed by mobile applications provided by European Union institutions.” As non-embedded software becomes available for other devices beyond smartphones and tablets, regulations for apps and non-embedded software in other types of smart systems will need to be developed.
Regulations need to be able to establish where liability lies when accidents or crimes involving smart systems occur. At present, when a car crash involving an autonomous vehicle occurs, liability currently focuses on the individual driver or owner of the vehicle, but if, for instance, a car crash occurs between a vehicle driven by a human-driver and a vehicle on autopilot, this can create uncertainty as to who is culpable – the human driving at the time of an accident, the software developer, the car manufacturer, or the AI-driver. In the UK, the insurance company Adrian Flux claims it has launched the UK’s first car insurance policy for autonomous vehicles. The policy stipulates that customers will be covered for loss or damage by “failure of the manufacturer’s vehicle operating system or authorised in-car software,” but does not mention any provisions regarding non-embedded software. As laws and regulations surrounding the nascent area of smart systems develop, regulations will need to provide more clarity.
Care should be given to enforcing the need to ascertain software safety before assuming that the wider technology is safe for use. Furthermore, further software applications that can be added onto systems, to provide further functionality such as entertainment, navigation, or performance monitoring needs to be safe in its own right, but also to not interact with the embedded software in any way that could lead to unsafe outcomes.
Non-embedded software creates new levels of complexity for ensuring the safety and security of connected and autonomous systems. Vulnerabilities can be exploited intentionally by malicious actors or can lead to unforeseen risks that lead to dangers for users and the people surrounding them. If these challenges are to be addressed, legal and regulatory frameworks as well as testing, certification, insurance and risk management systems need to be future-proofed. Policymakers must be agile and forward-thinking to ensure that these systems adapt along with the rapidly evolving technological landscape.