LightNE: A Lightweight Graph Processing System for Network Embedding
- Jiezhong Qiu ,
- Laxman Dhulipala ,
- Jie Tang ,
- Richard Peng ,
- Chi Wang
Proceedings of the 2021 International Conference on Management of Data (SIGMOD 2021) |
We propose LightNE, a cost-effective, scalable, and high quality network embedding system that scales to graphs with hundreds of billions of edges on a single machine.
In contrast to the mainstream belief that distributed architecture and GPUs are needed for large-scale network embedding with good quality, we prove that we can achieve higher quality, better scalability, lower cost and faster runtime with shared-memory, CPU-only architecture.
LightNE combines two theoretically grounded embedding methods NetSMF and ProNE.
We introduce the following techniques to network embedding for the first time:
(1) a newly proposed downsampling method to reduce the sample complexity of NetSMF while preserving its theoretical advantages;
(2) a high-performance parallel graph processing stack GBBS to achieve high memory efficiency and scalability;
(3) sparse parallel hash table to aggregate and maintain the matrix sparsifier in memory;
and (4) Intel MKL for efficient randomized SVD and spectral propagation.