Light@ASPI_CTSPersuasive technologies in China: Implications for the future of national securityKey FindingsThe rapid adoption of persuasive technologies—any digital system that shapes users’ attitudes and behaviours by exploiting physiological and cognitive reactions or vulnerabilities—will challenge national security in ways that are difficult to predict.Emerging persuasive technologies such as generative artificial intelligence (AI), ambient technologies and neurotechnology interact with the human mind and body in far more intimate and subconscious ways, and at far greater speed and efficiency, than previous technologies. This presents malign actors with the ability to sway opinions and actions without the conscious autonomy of users.Regulation is struggling to keep pace.Over the past decade, the swift development and adoption of these technologies have outpaced responses by liberal democracies, highlighting the urgent need for more proactive approaches that prioritise privacy and user autonomy. That means protecting and enhancing the ability of users to make conscious and informed decisions about how they’re interacting with technology and for what purpose.China’s commercial sector is already a global leader in developing and using persuasive technologies.The Chinese Communist Party (CCP) tightly controls China’s private sector and mandates that Chinese companies—especially technology companies—work towards China’s national-security interests. This presents a risk that the CCP could use persuasive technologies commercially developed in China to pursue illiberal and authoritarian ends, both domestically and abroad, through such means as online influence campaigns, targeted psychological operations, transnational repression, cyber operations and enhanced military capabilities.ASPI has identified several prominent Chinese companies that already have their persuasive technologies at work for China’s propaganda, military and public-security agencies. They include:Midu—a language intelligence technology company that provides generative AI tools used by Chinese Government and CCP bureaus to enhance the party-state’s control of public opinion. Those capabilities could also be used for foreign interference (see page 4).Suishi—a pioneer in neurotechnology that’s developing an online emotion detection and evaluation system to interpret and respond to human emotions in real time. The company is an important partner of Tianjin University’s Haihe Lab (see page 16), which has been highly acclaimed for its research with national-security applications (see page 17).Goertek—an electronics manufacturer that has achieved global prominence for smart wearables and virtual-reality (VR) devices. This company collaborates on military–civil integration projects with the CCP’s military and security organs and has developed a range of products with dual-use applications, such as drone-piloting training devices (see page 20).ASPI has further identified case studies of Chinese technology companies, including Silicon Intelligence, OneSight and Mobvoi, that are leading in the development of persuasive technologies spanning generative AI, neurotechnologies and emerging ambient systems.We find that those companies have used such solutions in support of the CCP in diverse ways—including overt and attributable propaganda campaigns, disinformation campaigns targeting foreign audiences, and military–civil fusion projects.IntroductionPersuasive technologies—or technologies with persuasive characteristics—are tools and systems designed to shape users’ decision-making, attitudes or behaviours by exploiting people’s physiological and cognitive reactions or vulnerabilities.1Compared to technologies we presently use, persuasive technologies collect more data, analyse more deeply and generate more insights that are more intimately tailored to us as individuals.With current consumer technologies, influence is achieved through content recommendations that reflect algorithms learning from the choices we consciously make (at least initially). At a certain point, a person’s capacity to choose then becomes constrained because of a restricted information environment that reflects and reinforces their opinions—the so-called echo-chamber effect. With persuasive technologies, influence is achieved through a more direct connection with intimate physiological and emotional reactions. That risks removing human choice from the process entirely and steering choices without an individual’s full awareness. Such technologies won’t just shape what we do: they have the potential to influence who we are.Many countries and companies are working to harness the power of emerging technologies with persuasive characteristics, such as generative artificial intelligence (AI), wearable devices and brain–computer interfaces, but the People’s Republic of China (PRC) and its technology companies pose a unique challenge. The Chinese party-state combines a rapidly advancing tech industry with a political system and ideology that mandate companies to align with CCP objectives, driving the creation and use of persuasive technologies for political purposes (see ‘How the CCP is using persuasive technologies’, page 21). That synergy enables China to develop cutting-edge innovations while directing their application towards maintaining regime stability domestically, reshaping the international order, challenging democratic values and undermining global human-rights norms.There’s already extensive research on how the CCP and its military are adopting technology in cognitive warfare to ‘win without fighting’—a strategy to acquire the means to shape adversaries’ psychological states and behaviours (see Appendix 2: Persuasive technologies in China’s ‘cognitive warfare’, page 29).2Separately, academics have considered the manipulative methods of surveillance capitalism, especially on issues of addiction, child safety and privacy .3However, there’s limited research on the intersection of those two topics; that is, attempts by the Chinese party-state to exploit commercially available emerging technologies to advance its political objectives. This report is one of the first to explore that intersection.Chinese technology, advertising and public-relations companies have made substantial advances in harnessing such tools, from mobile push notifications and social-media algorithms to AI-generated content. Many of those companies have achieved global success. Access to the personal data of foreign users is at an all-time high, and Chinese companies are now a fixed staple on the world’s most downloaded mobile apps lists, unlike just five years ago.4While many persuasive technologies have clear commercial purposes, their potential for political and national-security exploitation—both inside and outside China—is also profound.This report seeks to break through the ‘Collingridge dilemma’, in which control and understanding of emerging technologies come too late to mitigate the consequences of those technologies.5The report analyses generative AI, neurotechnologies and immersive technologies and focuses on key advances being made by PRC companies in particular. It examines the national-security implications of persuasive technologies designed and developed in China, and what that means for policymakers and regulators outside China as those technologies continue to roll out globally.Persuasive-technology capabilities are evolving rapidly, and concepts of and approaches to regulation are struggling to keep pace. The national-security implications of technologies that are designed to drive users towards certain behaviours are becoming apparent. Democratic governments have acted slowly and reactively to those challenges over the past decade. There’s an urgent need for more fit-for-purpose, proactive and adaptive approaches to regulating persuasive technologies. Protecting user autonomy and privacy must sit at the core of those efforts. Looking forward, persuasive technologies are set to become even more sophisticated and pervasive, and the consequences of their use are increasingly difficult to detect. Accordingly, the policy recommendations set out here focus on preparing for and countering the potential malicious use of the next generation of those technologies.Full ReportFor the full report, pleasedownload here.26 Nov 2024Persuasive technologies in ChinaFri, 11/22/2024 - 14:35markopetreski@…AttachmentDownload8.47 MBProgram link/program/international-cyber-policy-centreReferences1First defined by Brian J Fogg in Persuasive technology: using computers to change what we think and do, Morgan Kaufmann, 2003.2www.rand.orgSee, for example, Nathan Beauchamp-Mustafaga, Chinese next-generation psychological warfare, RAND Corporation, Santa Monica, 1 June 2023, online; Elsa B Kania, ‘Minds at war: China’s pursuit of military advantage through cognitive science and biotechnology’, PRISM, 2019, 8(3):82–101, online; Department of Defense, Annual report to Congress; Military and security developments involving the People’s Republic of China, US Government, 19 October 2023, online.3Shoshana Zuboff, The Age of Surveillance Capitalism: the fight for a human future at the new frontier of power, Ingram Publisher Services, 2017.4businessofapps.comExamples of Chinese-owned apps that are among the most downloaded globally include Tiktok, CapCut (a ByteDance-owned video editor) and the e-commerce platforms Temu and Shein. See David Curry, ‘Most popular apps (2024)’, Business of Apps, 30 January 2024, online.5Richard Worthington, ‘The social control of technology by David Collingridge’, American Political Science Review, 1982, 76(1):134–135; David Collingridge,The social control of technology, St Martin’s Press, New York, 1980.AuthorDaria ImpiombatoAnalystFull bioDr Nathan AttrillChina AnalystFull bioAlbert ZhangAnalystFull bioFergus RyanSenior AnalystFull bioBethany AllenHead of China Investigations and AnalysisFull bioAcknowledgementsASPI acknowledges the Ngunnawal and Ngambri peoples, who are the traditional owners and custodians of the land upon which this work was prepared, and their continuing connection to land, waters and community. We pay our respects to their cultures, country and elders past, present and emerging.The authors would like to thank ASPI colleagues who worked on and reviewed this project, including Danielle Cave, James Corera, Dr Fitriani, Bart Hogeveen and Tilla Hoja, and external reviewers Simeon Gilding, Jacinta Keast, Kenton Thibaut and Alia Yofira.About ASPIThe Australian Strategic Policy Institute was formed in 2001 as an independent, non partisan think tank. Its core aim is to provide the Australian Government with fresh ideas on Australia’s defence, security and strategic policy choices. ASPI is responsible for informing the public on a range of strategic issues, generating new thinking for government and harnessing strategic thinking internationally. ASPI’s sources of funding are identified in our annual report, online atwww.aspi.org.auand in the acknowledgements section of individual publications. ASPI remains independent in the content of the research and in all editorial judgements. It is incorporated as a company, and is governed by a Council with broad membership. ASPI’s core values are collegiality, originality and innovation, quality and excellence, and independence.ASPI’s publications—including this paper—are not intended in any way to express or reflect the views of the Australian Government. The opinions and recommendations in this paper are published by ASPI to promote public debate and understanding of strategic and defence issues. They reflect the personal views of the authors and should not be seen as representing the formal position of ASPI on any particular issue.ASPI Cyber, Technology & SecurityASPI’s Cyber, Technology and Security (CTS) analysts inform policy debates in the Indo-Pacific through original, rigorous and data-driven research. CTS is a leading voice in global debates on cyber, emerging and critical technologies, foreign interference and issues related to information operations and disinformation. CTS has a growing mixture of expertise and skills with teams of researchers who concentrate on policy, technical analysis, information operations and disinformation, critical and emerging technologies, cyber capacity building and internet safety, satellite analysis, surveillance and China-related issues. To develop capability in Australia and across the Indo-Pacific region, CTS has a capacity-building team that conducts workshops, training programs and large-scale exercises for the public, private and civil-society sectors.CTS enriches regional debate by collaborating with civil-society groups from around the world and by bringing leading global experts to Australia through our international fellowship program. We thank all of those who support and contribute to CTS with their time, intellect and passion for the topics we work on. If you would like to support the work of the CTS, contact:[email protected].FundingFunding for this report was provided by the US State Department’s Global Engagement Center.Important disclaimerThis publication is designed to provide accurate and authoritative information in relation to the subject matter covered. It is provided with the understanding that the publisher is not engaged in rendering any form of professional or other advice or services.© The Australian Strategic Policy Institute Limited 2024This publication is subject to copyright. Except as permitted under the Copyright Act 1968, no part of it may in any form or by any means (electronic, mechanical, microcopying, photocopying, recording or otherwise) be reproduced, stored in a retrieval system or transmitted without prior written permission. Enquiries should be addressed to the publishers. Notwithstanding the above, educational institutions (including schools, independent colleges, universities and TAFEs) are granted permission to make copies of copyrighted works strictly for educational purposes without explicit permission from ASPI and free of charge.First published November 2024Published in Australia by the Australian Strategic Policy InstituteCyber, Technology & Security