A fully automated method for discovering community structures in high dimensional data

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    8 Scopus citations

    Abstract

    Identifying modules, or natural communities, in large complex networks is fundamental in many fields, including social sciences, biological sciences and engineering. Recently several methods have been developed to automatically identify communities from complex networks by optimizing the modularity function. The advantage of this type of approaches is that the algorithm does not require any parameter to be tuned. However, the modularity-based methods for community discovery assume that the network structure is given explicitly and is correct. In addition, these methods work best if the network is unweighted and/or sparse. In reality, networks are often not directly defined, or may be given as an affinity matrix. In the first case, each node of the network is defined as a point in a high dimensional space and different networks can be obtained with different network construction methods, resulting in different community structures. In the second case, an affinity matrix may define a dense weighted graph, for which modularity-based methods do not perform well. In this work, we propose a very simple algorithm to automatically identify community structures from these two types of data. Our approach utilizes a k-nearest-neighbor network construction method to capture the topology embedded in high dimensional data, and applies a modularity-based algorithm to identify the optimal community structure. A key to our approach is that the network construction is incorporated with the community identification process and is totally parameterfree. Furthermore, our method can suggest appropriate preprocessing/normalization of the data to improve the results of community identification. We tested our methods on several synthetic and real data sets, and evaluated its performance by internal or external accuracy indices. Compared with several existing approaches, our method is not only fully automatic, but also has the best accuracy overall.

    Original languageEnglish (US)
    Title of host publicationICDM 2009 - The 9th IEEE International Conference on Data Mining
    Pages968-973
    Number of pages6
    DOIs
    StatePublished - Dec 1 2009
    Event9th IEEE International Conference on Data Mining, ICDM 2009 - Miami, FL, United States
    Duration: Dec 6 2009Dec 9 2009

    Publication series

    NameProceedings - IEEE International Conference on Data Mining, ICDM
    ISSN (Print)1550-4786

    Conference

    Conference9th IEEE International Conference on Data Mining, ICDM 2009
    CountryUnited States
    CityMiami, FL
    Period12/6/0912/9/09

    Keywords

    • Community structure
    • Image clustering
    • Modularity

    ASJC Scopus subject areas

    • Engineering(all)

    Fingerprint Dive into the research topics of 'A fully automated method for discovering community structures in high dimensional data'. Together they form a unique fingerprint.

    Cite this