{"id":355241,"date":"2017-01-19T02:01:26","date_gmt":"2017-01-19T10:01:26","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-research-item&p=355241"},"modified":"2018-10-16T20:43:53","modified_gmt":"2018-10-17T03:43:53","slug":"information-theoretic-thresholds-community-detection-sparse-networks","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/information-theoretic-thresholds-community-detection-sparse-networks\/","title":{"rendered":"Information-theoretic thresholds for community detection in sparse networks"},"content":{"rendered":"

We give upper and lower bounds on the information-theoretic threshold for community detection in the stochastic block model. Specifically, consider the symmetric stochastic block model with q<\/span><\/span><\/span><\/span> groups, average degree d<\/span><\/span><\/span><\/span>, and connection probabilities c_{<\/span>in}<\/span><\/span>\/<\/span><\/span><\/span>n<\/span><\/span><\/span><\/span> and c_{<\/span>out}<\/span><\/span>\/<\/span><\/span><\/span>n<\/span><\/span><\/span><\/span> for within-group and between-group edges respectively; let \u03bb<\/span>=<\/span>(<\/span>c_{i<\/span>n}<\/span><\/span>\u2212<\/span>c_{<\/span>out}<\/span><\/span>)<\/span>\/<\/span><\/span><\/span>(<\/span>q<\/span>d<\/span>)<\/span><\/span><\/span><\/span>. We show that, when q<\/span><\/span><\/span><\/span> is large, and \u03bb<\/span>=<\/span>O<\/span>(<\/span>1<\/span>\/<\/span><\/span><\/span>q<\/span>)<\/span><\/span><\/span><\/span>, the critical value of d<\/span><\/span><\/span><\/span> at which community detection becomes possible—in physical terms, the condensation threshold—is<\/p>\n

d<\/span>c<\/span><\/span>=<\/span>\u0398<\/span><\/span>(\\<\/span>log(<\/span><\/span>q)\/<\/span><\/span>q<\/span>\u03bb^<\/span>2<\/span><\/span><\/span><\/span>)<\/span><\/span><\/span>,<\/span><\/span><\/span><\/span><\/div>\n

with tighter results in certain regimes. Above this threshold, we show that any partition of the nodes into q<\/span><\/span><\/span><\/span> groups which is as `good’ as the planted one, in terms of the number of within- and between-group edges, is correlated with it. This gives an exponential-time algorithm that performs better than chance; specifically, community detection becomes possible below the Kesten-Stigum bound for q<\/span>\u2265<\/span>5<\/span><\/span><\/span><\/span> in the disassortative case \u03bb<\/span><<\/span>0<\/span><\/span><\/span><\/span>, and for q<\/span>\u2265<\/span>11<\/span><\/span><\/span><\/span> in the assortative case \u03bb<\/span>><\/span>0<\/span><\/span><\/span><\/span> (similar upper bounds were obtained independently by Abbe and Sandon). Conversely, below this threshold, we show that no algorithm can label the vertices better than chance, or even distinguish the block model from an\u00a0Erdos-Renyi random graph with high probability.
\nOur lower bound on d<\/span>c<\/span><\/span><\/span><\/span><\/span> uses Robinson and Wormald’s small subgraph conditioning method, and we also give (less explicit) results for non-symmetric stochastic block models. In the symmetric case, we obtain explicit results by using bounds on certain functions of doubly stochastic matrices due to Achlioptas and Naor; indeed, our lower bound on d<\/span>c<\/span><\/span><\/span><\/span><\/span> is their second moment lower bound on the q<\/span><\/span><\/span><\/span>-colorability threshold for random graphs with a certain effective degree.<\/p>\n","protected":false},"excerpt":{"rendered":"

We give upper and lower bounds on the information-theoretic threshold for community detection in the stochastic block model. Specifically, consider the symmetric stochastic block model with q groups, average degree d, and connection probabilities c_{in}\/n and c_{out}\/n for within-group and between-group edges respectively; let \u03bb=(c_{in}\u2212c_{out})\/(qd). We show that, when q is large, and \u03bb=O(1\/q), the […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_publishername":"COLT","msr_publisher_other":"","msr_booktitle":"","msr_chapter":"","msr_edition":"Conference on Learning Theory 2016","msr_editors":"","msr_how_published":"","msr_isbn":"","msr_issue":"","msr_journal":"","msr_number":"","msr_organization":"","msr_pages_string":"","msr_page_range_start":"","msr_page_range_end":"","msr_series":"","msr_volume":"","msr_copyright":"","msr_conference_name":"Conference on Learning Theory 2016","msr_doi":"","msr_arxiv_id":"","msr_s2_paper_id":"","msr_mag_id":"","msr_pubmed_id":"","msr_other_authors":"","msr_other_contributors":"","msr_speaker":"","msr_award":"","msr_affiliation":"","msr_institution":"","msr_host":"","msr_version":"","msr_duration":"","msr_original_fields_of_study":"","msr_release_tracker_id":"","msr_s2_match_type":"","msr_citation_count_updated":"","msr_published_date":"2016-06-01","msr_highlight_text":"","msr_notes":"","msr_longbiography":"","msr_publicationurl":"https:\/\/arxiv.org\/abs\/1607.01760","msr_external_url":"","msr_secondary_video_url":"","msr_conference_url":"","msr_journal_url":"","msr_s2_pdf_url":"","msr_year":0,"msr_citation_count":0,"msr_influential_citations":0,"msr_reference_count":0,"msr_s2_match_confidence":0,"msr_microsoftintellectualproperty":true,"msr_s2_open_access":false,"msr_s2_author_ids":[],"msr_pub_ids":[],"msr_hide_image_in_river":0,"footnotes":""},"msr-research-highlight":[],"research-area":[13561],"msr-publication-type":[193716],"msr-publisher":[],"msr-focus-area":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-355241","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-algorithms","msr-locale-en_us"],"msr_publishername":"COLT","msr_edition":"Conference on Learning Theory 2016","msr_affiliation":"","msr_published_date":"2016-06-01","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"https:\/\/arxiv.org\/abs\/1607.01760","msr_doi":"","msr_publication_uploader":[{"type":"url","title":"https:\/\/arxiv.org\/abs\/1607.01760","viewUrl":false,"id":false,"label_id":0}],"msr_related_uploader":"","msr_citation_count":0,"msr_citation_count_updated":"","msr_s2_paper_id":"","msr_influential_citations":0,"msr_reference_count":0,"msr_arxiv_id":"","msr_s2_author_ids":[],"msr_s2_open_access":false,"msr_s2_pdf_url":null,"msr_attachments":[{"id":0,"url":"https:\/\/arxiv.org\/abs\/1607.01760"}],"msr-author-ordering":[{"type":"text","value":"Jess Banks","user_id":0,"rest_url":false},{"type":"text","value":"Cristopher Moore","user_id":0,"rest_url":false},{"type":"text","value":"Joe Neeman","user_id":0,"rest_url":false},{"type":"user_nicename","value":"praneeth","user_id":33279,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=praneeth"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[],"msr_group":[],"msr_project":[],"publication":[],"video":[],"msr-tool":[],"msr_publication_type":"inproceedings","related_content":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/355241","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/355241\/revisions"}],"predecessor-version":[{"id":410462,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/355241\/revisions\/410462"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=355241"}],"wp:term":[{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=355241"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=355241"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=355241"},{"taxonomy":"msr-publisher","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publisher?post=355241"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=355241"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=355241"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=355241"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=355241"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=355241"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=355241"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=355241"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=355241"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}