PacTrain accepted at DAC-2025
Our paper on PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning has been accepted at the Design Automation Conference DAC (2025) (CCF-A).
Abstract:
Large-scale deep neural networks (DNN) exhibit excellent performance for various tasks. As DNNs and datasets grow, distributed training becomes extremely time-consuming and demands larger clusters. A main bottleneck is the resulting gradient aggregation overhead. While gradient compression and sparse collective communication techniques are commonly employed to alleviate network load, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. This paper introduces PacTrain, a novel framework that accelerates distributed training by combining pruning with sparse gradient compression. Active pruning of the neural network makes the model weights and gradients sparse.
By ensuring the global knowledge of the gradient sparsity among all distributed training workers, we can perform lightweight compression communication without harming accuracy. We show that the PacTrain compression scheme achieves a near-optimal compression strategy while remaining compatible with the all- reduce primitive. Experimental evaluations show that PacTrain improves training throughput by 1.25 to 8.72× compared to state-of-the-art compression-enabled systems for representative vision and language models training tasks under bandwidth-constrained conditions.
Stay tuned for the pre-print.
References
Yisu Wang, Ruilong Wu, Xinjiao Li , Dirk Kutscher; PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning; Design Automation Conference (DAC) 2025
HKUST Internet Research Workshop (HKIRW) 2025
We are organizing the 2025 HKUST Internet Research Workshop (HKIRW) in the week before the IETF-122 meeting in Bangkok. This workshop aims to bring together researchers in computer networking and systems around the globe to a live forum discussing innovative ideas at their early stages. The mission of the workshop is that promising but not-yet-mature ideas can receive timely feedback from the community and experienced researchers, leading them into future IRTF work, Internet Drafts, or IETF working groups.
The workshop will operate like a “one day Dagstuhl seminar” and will focus on discussion and ideas exchange and less on conference-style presentations. The objective is to identify topics and connect like-minded people for potential future collaboration.
Please see https://hkirw.github.io/2025/ for details.
References
Report: ACM Conext-2024 Workshop on the Decentralization of the Internet
On Monday, December 9th, 2024, we held our Decentralization of the Internet (DIN) workshop at ACM CoNEXT-2024. It brought together network researchers, law and policy experts, and digital right activists to discuss the observed consolidation and centralization of the existing Internet applications, services, and the infrastructure in recent years. This trend has economic as well as technical implications for attributes commonly associated with the Internet, such as user-centricity and permissionless innovations.
The Decentralization of the Internet Research Group (DINRG) of the Internet Research Task Force (IRTF) has been working on identifying the root causes and consequences of Internet centralization at IETF meetings and focused workshops in the past, which has led to significant insights, especially with regard to the centralization of infrastructure and control power. This recent DIN workshop at ACM CoNEXT-2024, organized by my DINRG co-chairs Lixia Zhang and myself, provided a forum for academic researchers to present and discuss on-going efforts on this topic, and to create a greater awareness of this important issue in the broader network research community. The workshop attracted a diverse set of researchers who are working on Internet decentralization in fields such as Internet technologies, economics and law-making. The workshop featured two keynotes, two technical paper presentation sessions, and an interactive panel discussion.
Keynotes
The keynotes were presented by two renowned experts:
Keynote: Cory Doctorow: DISENSHITTIFY OR DIE!
Cory Doctorow, member of the Electronic Frontier Foundation (EFF), gave a talk titled DISENSHITTIFY OR DIE! How computer scientists can halt enshittification to make a new, good internet and condemn today's enshitternet to the scrapheap of history. Cory’s talk vividly explained the historic development of a process that he called enshittification, a process in which the providers of online products and services changed their policies subtly and gradually over time, grabbing the control of user data for profitability. Doctorow also discussed potential remedies and countermeasures, including removing the barriers for users to exit platforms and reinstalling the end-to-end principle in future application developments.
Keynote: Michael Karanicolas: The Fediverse Papers: Constitutional, Governance, and Policy Questions for a New Paradigm of Networking
Michael Karanicolas, the executive director of the UCLA Institute for Technology, Law and Policy, talked about the Fediverse Papers: Constitutional, Governance, and Policy Questions for a New Paradigm of Networking. Michael provided an overview of the history of digital speech and content governance. He highlighted the challenges in supporting effective content moderation in today’s Internet contexts, including issues around monetization, legislation, privacy, and the need for governance mechanisms to meet users, content owners, and governments’ expectations. He emphasized the importance of intentionality and a structured process to identify the essential policy questions and to evaluate various design choices for the future of decentralized platforms.
Decentralized Systems
Bluesky and the AT Protocol: Usable Decentralized Social Media
Authors: Martin Kleppmann, Paul Frazee, Jake Gold, Jay Graber, Daniel Holmgren, Devin Ivy, Jeromy Johnson, Bryan Newbold, Jaz Volpert
Abstract: Bluesky is a new social network built upon the AT Protocol, a decentralized foundation for public social media. It was launched in private beta in February 2023, and has grown to over 10 million registered users by October 2024. In this paper we introduce the architecture of Bluesky and the AT Protocol, and explain how the technical design of Bluesky is informed by our goals: to enable decentralization by having multiple interoperable providers for every part of the system; to make it easy for users to switch providers; to give users agency over the content they see; and to provide a simple user experience that does not burden users with complexity arising from the system’s decentralized nature. The system’s openness allows anybody to contribute to content moderation and community management, and we invite the research community to use Bluesky as a dataset and testing ground for new approaches in social media moderation.
ReP2P Matrix: Decentralized Relays to Improve Reliability and Performance of Peer-to-Peer Matrix
Authors: Benjamin Schichtholz, Roland Bless, Florian Jacob, Hannes Hartenstein, Martina Zitterbart
Abstract: Matrix is a decentralized middleware for low-latency group communication, most renowned for its use in the Element instant messenger. Proposals for peer-to-peer (P2P) Matrix architectures aim to decentralize the current architecture further, which is based on federated servers. These proposals require that the receiver and the originator, or another peer that already successfully received the message, are simultaneously online. We introduce relay-enhanced P2P Matrix (ReP2P Matrix) in order to improve message delivery between peers that are online at different times. The design maintains the advantages of P2P Matrix and integrates well into it, e.g., it reuses existing mechanisms for authentication and authorization. Using an extended real-world group messaging traffic dataset, we evaluate P2P Matrix by comparing it to P2P Matrix without relays. The results show that relays do not only improve reliability in message delivery, but also increase the share of low delivery latencies by 50% points in groups with up to 30 members.
On Empowering End Users in Future Networking
Authors: Tianyuan Yu, Xinyu Ma, Lixia Zhang
Abstract: In today's Internet, end users communicate largely via cloud-based apps, and user data are stored in cloud servers and controlled by cloud providers. Recent years have witnessed multiple efforts in developing decentralized social apps with various design approaches, although the community at large is yet to fully understand the effectiveness, viability, and limitations of these different designs. In this paper, we make a proposition that a necessary condition of moving towards Internet decentralization is enabling direct user-to-user (U2U) communications, and discuss the design choices in several decentralization efforts and identify their limitations. We then articulate why a DNS-derived namespace is the best choice in U2U app developments in general, and use a recently developed decentralized app, NDN Workspace (NWS), as an example to show how NWS' use of DNS-derived namespace enables secure U2U communications.
Technologies for Decentralization
Atomicity and Abstraction for Multi-Blockchain Interactions
Authors: Huaixi Lu, Akshay Jajoo, Kedar S. Namjoshi
Abstract: A blockchain enables secure, atomic transactions among untrusted parties. Atomicity is not guaranteed, however, for transactions whose operations span several blockchains; multi-chain atomicity must be enforced by a protocol. Such protocols are known only for special cases, such as cryptocurrency swaps, which are limited only to two chains. We propose a novel two-phase protocol that facilitates atomic executions of general multi-chain (>= 2) transactions. We formally analyze the protocol correctness and show that the proposed abstraction considerably simplifies the development of multi-chain applications. Our experiments with a prototype implementation show that the performance of the general atomicity protocol is comparable to that of custom-built implementations.
Communication Cost for Permissionless Distributed Consensus at Internet Scale
Authors: David Guzman, Dirk Trossen, Jörg Ott
Abstract: The diffusion of information that evolves a distributed computing state is a fundamental operation of a permissionless distributed consensus system (DCS). This permissionless participation decentralized the consensus over the distributed computing state, e.g., in cryptocurrencies and voting systems. For this, a permissionless DCS implements protocols to establish relationships among peers, which is then used to diffuse information. The relation establishment constitutes the control plane of the DCS, while the state diffusion is the data plane. The prevalent mechanism to realize both is a randomized peer-centric iterative diffusion. In this paper, we contrast this approach against a multicast-based design, focusing our comparison on the costs (bytes transmitted) for maintaining the relations, the control plane. We develop suitable models to account for those costs, parameterized through Internet-scale experimental insights we derived from existing DCS deployments. Our results show that the communication costs can be reduced by 30 times.
Towards a Decentralized Internet Namespace
Authors: Yekta Kocaogullar, Eric Osterweil, Lixia Zhang
Abstract: The Domain Name System (DNS) has been providing a decentralized global namespace to support all Internet applications and usages over the last few decades. In the recent years, a number of blockchain-based name systems have emerged with the claim of providing better namespace decentralization than DNS. The community at large seems uncertain with regard to which of these systems is the best in providing decentralized Internet namespace control. In this paper, we first deconstruct the design of DNS, identify its three essential components and explain who controls each of them. We then examine the Ethereum Name Service (ENS) as a representative example of blockchain-based naming systems, gauge the degree of its decentralization. Finally, we conduct a comparative analysis between DNS and ENS to assess the validity and affordability of each design and the (de)centralization in their namespace control and name system operations.
Panel Discussion: Decentralization of the Internet – Quo Vadis?
An interactive panel discussion with (from left to right) Michael Karanicolas (UCLA), Paul Mockapetris (ThreatSTOP), Dan Massey (USC ISI, NSF), and Cory Doctorow (EFF), articulated various next steps for countering Internet centralization. Among many things discussed, the panel and audience identified the notion of enabling direct user-to-user communication without reliance on third parties, and the required functionality to support that, such as how to provide user owned identities, tools for user mutual authentications and secure communications.
These and additional related topics will be further discussed at the IRTF DIN research group, which is a forum with open participation to serve the purpose of continuous international collaborative research on Internet decentralization.
References
Appointed as IRTF Chair
I am delighted that I have been appointed as the next Chair of the Internet Research Task Force (IRTF) by the Internet Architecture Board (IAB).
I have been involved in the IRTF for many years. It is a unique organization that conducts research of importance to the evolution of the Internet protocols, applications, architecture and technology. It has initiated and supported many important technology developments for the Internet in the past, in fields such as network architecture, security and privacy, congestion control, and many more.
The IRTF focuses on longer term research issues, and its various research groups are enabling international collaboration for continuous research on critical topics for the Internet by working with academic and industry research communities.
My term starts in March 2025. I am sincerely grateful for all the support I have received, I am looking forward to working with this community to help making the Internet work better through good research work.
References
ACM Conext-2024 Workshop on the Decentralization of the Internet
Our ACM CoNEXT-2024 workshop on the decentralization of the Internet on Monday, December 9th 2024 in LA has an exciting agenda – don't miss it! Check out the workshop homepage for up-to-date information.
09:00 Session 1: Keynotes
- Keynote by Cory Doctorow: DISENSHITTIFY OR DIE! How computer scientists can halt enshittification to make a new, good internet and condemn today's enshitternet to the scrapheap of history.
- Keynote by Michael Karanicolas: The Fediverse Papers: Constitutional, Governance, and Policy Questions for a New Paradigm of Networking
11:00 Session 2: Decentralized Systems
- Martin Kleppmann, et al.; Bluesky and the AT Protocol: Usable Decentralized Social Media
- Benjamin Schichtholz et al.; ReP2P Matrix: Decentralized Relays to Improve Reliability and Performance of Peer-to-Peer Matrix
- Tianyuan Yu et al.; On Empowering End Users in Future Networking
14:00 Session 3: Technologies for Decentralization
- Huaixi Lu et al.; Atomicity and Abstraction for Multi-Blockchain Interactions
- David Guzman et. el; Communication Cost for Permissionless Distributed Consensus at Internet Scale
- Yekta Kocaogullar et al.; Towards a Decentralized Internet Namespace
15:00 Session 4: Decentralization of the Internet – Quo Vadis?
- Organizers: Lixia Zhang & Dirk Kutscher
- Interactive panel discussion with Cory Doctorow, Michael Karanicola, and paper authors
IRTF DINRG Meeting at IETF-121
The IRTF DINRG Meeting at IETF-121 takes place on 2024-11-06 at 13:00 to 14:30 UTC.
| 1 | DINRG Chairs’ Presentation: Status, Updates | Chairs | 05 min |
| 2 | Distributing DDoS Analytics among ASes | Daniel Wagner | 20 min |
| 3 | The Role of DNS names in Internet Decentralization | Tianyuan Yu | 20 min |
| 4 | Taxonomy of Internet Consolidation & Effects of Internet Consolidation | Marc McFadden | 15 min |
| 5 | DINRG – Next Steps | Chairs & Panelists | 30 min |
| 6 | Wrap-up & Buffer | Chairs | 00 min |
Documents and Links to Resources
- United We Stand: Collaborative Detection and Mitigation of
Amplification DDoS Attacks at
Scale - https://datatracker.ietf.org/doc/draft-mcfadden-consolidation-taxonomy/
- https://datatracker.ietf.org/doc/draft-mcfadden-cnsldtn-effects/
Notes
Please remember that all sessions are being recorded.
IRTF ICNRG Meeting at IETF-121
The ICNRG Meeting at IETF-121 takes place on 2024-11-05, 13:00 to 14:30 UTC.
ICNRG Agenda
| 1 | ICNRG Chairs’ Presentation: Status, Updates | Chairs | 05 min |
| 2 | FLIC Update | Marc Mosko | 15 min |
| 3 | CCNx Content Object Chunking | Marc Mosko | 15 min |
| 4 | Reflexive Forwarding Update | Hitoshi Asaeda | 20 min |
| 5 | ICN Challenges for Metaverse Platform Interoperability | Jungha Hong | 15 min |
| 6 | Distributed Micro Service Communication | Aijun Wang | 15 min |
| 7 | Buffer, Wrap Up and Next Steps | Chairs | 05 min |
Please remember that all sessions are being recorded.
Material
- https://datatracker.ietf.org/doc/draft-irtf-icnrg-flic/
- https://datatracker.ietf.org/doc/draft-mosko-icnrg-ccnxchunking/
- https://github.com/mmosko/ccnpy
- https://datatracker.ietf.org/doc/draft-irtf-icnrg-reflexive-forwarding/
- https://datatracker.ietf.org/doc/draft-hong-icn-metaverse-interoperability/
- https://datatracker.ietf.org/doc/draft-li-icnrg-damc/
New Internet Draft draft-irtf-icnrg-reflexive-forwarding-00
We updated our Internet Draft draft-irtf-icnrg-reflexive-forwarding-00 on Reflexive Forwarding for CCNx and NDN Protocols.
Current Information-Centric Networking protocols such as CCNx and NDN have a wide range of useful applications in content retrieval and other scenarios that depend only on a robust two-way exchange in the form of a request and response (represented by an Interest-Data exchange in the case of the two protocols noted above). A number of important applications however, require placing large amounts of data in the Interest message, and/or more than one two-way handshake. While these can be accomplished using independent Interest-Data exchanges by reversing the roles of consumer and producer, such approaches can be both clumsy for applications and problematic from a state management, congestion control, or security standpoint. This specification proposes a Reflexive Forwarding extension to the CCNx and NDN protocol architectures that eliminates the problems inherent in using independent Interest-Data exchanges for such applications. It updates RFC8569 and RFC8609.
The recent update includes a generalization of the main protocol specification, so that Reflexive Forwarding can be used in both CCNx and NDN.
Invited Talk at Airbus Workshop on Networking Systems
On October 10th, 2024, I was invited to give a talk at the 2nd Airbus Workshop on Networking Systems. The workshop largely discussed connected aircraft scenarios and technologies and features talks on security and reliability, IoT sensor fusioning, and future space and 6G network architectures.
My talk was on Connected Aircraft – Network Architectures and Technologies, and discussed relevant scenarios from my perspective, such as passenger services and new aircraft management applications. For the technology discussion, I focused on large-scale low-latency multimedia communication over the expected heterogeneous and dynamic aircraft connectivity networks and discussed current and emerging technologies such as Media over QUIC, ICN.
I also introduced the recently established Low-Altitude Systems and Economy Research Institute at HKUST(GZ), a cross-disciplinary research institute for the low-altitude domain (with similar but not identical requirements) and some of our recent projects such as Named Data Microverse.
Dagstuhl Seminar on Greening Networking: Toward a Net Zero Internet
We (Alexander Clemm, Michael Welzl, Cedric Westphal, Noa Zilbermann, and I) organized a Dagstuhl seminar on Green Networking: Toward a Net Zero Internet.
Making Networks Greener
As climate change triggered by CO2 emissions dramatically impacts our environment and our everyday life, the Internet has proved a fertile ground for solutions, such as enabling teleworking or teleconferencing to reduce travel emissions. It is also a significant contributor to greenhouse gas emissions, e.g. through its own significant power consumption. It is thus very important to make networks themselves "greener" and devise less carbon-intensive solutions while continuing to meet increasing network traffic demands and service requirements.
Computer scientists and engineers from world-leading universities and international companies, such as Ericsson, NEC, Netflix, Red Hat, and Telefonica came together in a Seminar on Green Networking (Toward a Net Zero Internet) at Schloss Dagstuhl – Leibniz Center for Informatics, between September 29th and October 2nd, 2024. Organized by leading Internet researchers from the Hong Kong University of Science and Technology (Guangzhou), the University of Oxford, the University of Oslo and the University of California, Santa Cruz, they met to identify and prioritize the most impactful networking improvements to reduce carbon emission, define action items for a carbon-aware networking research agenda, and foster/facilitate research collaboration in order to reduce carbon emissions and to positively impact climate change.
Interactions between the Power Grid, Larger Systems, and the Network
In addition to pure networking issues, the seminar also analyzed the impact of larger systems that are built with Internet technologies, such as AI, multimedia streaming, and mobile communication networks. For example, the seminar discussed energy proportionality in networked systems, to allow systems to adapt their energy consumption to actual changes in utilization, so that savings can be achieved in idle times. Such a behavior would require better adaptiveness of applications and network protocols to cost information (such as carbon impact).
Moreover, networked systems can interact with the power grid in different ways, for example adapting energy consumption to current availability and cost of renewable energy, which can be helpful for joint planning of grid and network/networked-systems/cloud, achieving maximum efficiency/savings.
The seminar attendees are working with international research and standardization organizations such as the Internet Engineering Task Force (IETF) and ETSI, and it is expected that the seminar will make contributions to future research and standardization agendas in such organizations to bring the Internet to Net Zero emissions.
Organizers
- Alexander Clemm (Los Gatos, US)
- Dirk Kutscher (HKUST - Guangzhou, CN)
- Michael Welzl (University of Oslo, NO)
- Cedric Westphal (University of California, Santa Cruz, US)
- Noa Zilberman (University of Oxford, GB)