Sklearn Basics 4: Train a Coclustering¶

The steps to train a coclustering model with Khiops are very similar to what we have already seen in the basic classifier tutorials.

We start by importing the sklearn estimator KhiopsCoclustering and defining a helper function:

import os
import platform
import subprocess
import pandas as pd
from khiops import core as kh
from khiops.sklearn import KhiopsCoclustering

# If there are any issues you may Khiops status with the following command
# kh.get_runner().print_status()

For this tutorial, we use the dataset CountriesByOrganization that contains the relation country-organization for a large number of countries and organizations (it is bit outdated though). The objective is to build a coclustering between Country and Organization and see which countries resemble the most in terms of organizations.

Let’s first load this dataset and check its content:

countries_data_file = os.path.join(
    "data", "CountriesByOrganization", "CountriesByOrganization.csv"
)
X_countries = pd.read_csv(countries_data_file, sep=";")
print("CountriesByOrganization dataset:")
display(X_countries)

CountriesByOrganization dataset:

           Country Organization
    Afghanistan         AsDB
    Afghanistan      COLOMBO
    Afghanistan          ECO
    Afghanistan       ICCROM
    Afghanistan          NAM
...            ...          ...
   Zimbabwe          WHO
   Zimbabwe         WIPO
   Zimbabwe          WMO
   Zimbabwe          WTO
   Zimbabwe       WTOURO

[11192 rows x 2 columns]

Now, let’s build the coclustering model.

Note that a coclustering model is learned in an unsupervised way and aims to cluster jointly rows and columns of a matrix. So we need to provide a column name to be able to deploy it on a specific column. We do this by setting the fit parameter id_column:

khcc_countries = KhiopsCoclustering()
khcc_countries.fit(X_countries, id_column="Country")

KhiopsCoclustering()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Now let’s access the coclustering training report to obtain the cluster information of the Country dimension. Since in each dimension there is a hierarchical cluster, so we only access the leaf clusters:

countries_clusters = khcc_countries.model_report_.coclustering_report.get_dimension(
    "Country"
).clusters
countries_leaf_clusters = [cluster for cluster in countries_clusters if cluster.is_leaf]
print(f"Number of leaf clusters: {len(countries_leaf_clusters)}:")
for index, cluster in enumerate(countries_leaf_clusters, start=1):
    print(f"cluster {index:02d}: {cluster.name}")

Number of leaf clusters: 12:
cluster 01: {Germany, France, Netherlands, ...}
cluster 02: {United States of America, Canada, Japan, ...}
cluster 03: {Poland, Hungary, Turkey, ...}
cluster 04: {Kazakhstan, Kyrgyzstan, Azerbaijan, ...}
cluster 05: {Venezuela, Nicaragua, Ecuador, ...}
cluster 06: {Trinidad and Tobago, Barbados, Grenada, ...}
cluster 07: {Niger, Ivory Coast, Benin, ...}
cluster 08: {Tanzania, Uganda, Kenya, ...}
cluster 09: {Qatar, Saudi Arabia, United Arab Emirates, ...}
cluster 10: {Tunisia, Algeria, Morocco, ...}
cluster 11: {India, Malaysia, Indonesia, ...}
cluster 12: {Papua New Guinea, Fiji, Nepal, ...}

The composition of the clusters is also available. For the first one we have:

print(f"Members of the cluster {countries_leaf_clusters[0].name}:")
for value_obj in countries_clusters[0].leaf_part.values:
    print(value_obj.value)

Members of the cluster {Germany, France, Netherlands, ...}:
Germany
France
Netherlands
Denmark
Sweden
Belgium
Finland
Italy
Norway
Spain
Portugal
Austria
United Kingdom
Luxembourg
Switzerland
Greece
Ireland
Iceland

The coclustering is a complex model, so it is better to visualize it with the Khiops Co-visualization app. Let’s export the report to a .khcj file and open it:

countries_report = os.path.join("exercises", "countries.khcj")
khcc_countries.export_report_file(countries_report)
# explorer_open(countries_report)

Finally, let’s deploy the coclustering model on the training data countries_df:

countries_predictions = khcc_countries.predict(X_countries)
print("Predicted clusters (first 10)")
display(countries_predictions[:10])

Predicted clusters (first 10)

array([['Afghanistan', '{India, Malaysia, Indonesia, ...}'],
       ['Albania', '{Poland, Hungary, Turkey, ...}'],
       ['Algeria', '{Tunisia, Algeria, Morocco, ...}'],
       ['Andorra', '{Poland, Hungary, Turkey, ...}'],
       ['Angola', '{Niger, Ivory Coast, Benin, ...}'],
       ['Antigua and Barbuda',
        '{Trinidad and Tobago, Barbados, Grenada, ...}'],
       ['Argentina', '{Venezuela, Nicaragua, Ecuador, ...}'],
       ['Armenia', '{Kazakhstan, Kyrgyzstan, Azerbaijan, ...}'],
       ['Australia', '{United States of America, Canada, Japan, ...}'],
       ['Austria', '{Germany, France, Netherlands, ...}']], dtype=object)

Exercise¶

We’ll build a coclustering model for the Tokyo2021 dataset. It is extracted from the Athletes table of the Tokyo 2021 Kaggle dataset and each record contains three variables: - Name: the name of a competing athlete - Country: the country (or organization) it represents - Discipline: the athlete’s discipline

The objective with this exercise is to make a coclustering between Country and Discipline and see which countries resemble the most in terms of the athletes they bring to the Olympics. We start by loading the contents into a dataframe:

tokyo_data_file = os.path.join("data", "Tokyo2021", "Athletes.csv")
X_tokyo = pd.read_csv(tokyo_data_file, encoding="latin1")
print("Tokyo2021 dataset (first 10 rows):")
display(X_tokyo.head(10))

Tokyo2021 dataset (first 10 rows):

                Name                   Country           Discipline
  AALERUD Katrine                    Norway         Cycling Road
      ABAD Nestor                     Spain  Artistic Gymnastics
ABAGNALE Giovanni                     Italy               Rowing
   ABALDE Alberto                     Spain           Basketball
    ABALDE Tamara                     Spain           Basketball
        ABALO Luc                    France             Handball
     ABAROA Cesar                     Chile               Rowing
    ABASS Abobakr                     Sudan             Swimming
 ABBASALI Hamideh  Islamic Republic of Iran               Karate
    ABBASOV Islam                Azerbaijan            Wrestling

Train the coclustering for the variables `Country` and `Discipline`¶

Call fit parameters with the following parameters: - X=X_tokyo[["Country", "Discipline"]] - id_column="Country"

khcc_tokyo = KhiopsCoclustering()
khcc_tokyo.fit(X_tokyo[["Country", "Discipline"]], id_column="Country")

KhiopsCoclustering()

Obtain the number and names of the clusters of the `Country` dimension¶

tokyo_clusters = khcc_tokyo.model_report_.coclustering_report.get_dimension(
    "Country"
).clusters
tokyo_leaf_clusters = [cluster for cluster in tokyo_clusters if cluster.is_leaf]
print(f"Number of leaf clusters: {len(tokyo_leaf_clusters)}:")
for index, cluster in enumerate(tokyo_leaf_clusters):
    print(f"cluster {index:02d}: {cluster.name}")

Number of leaf clusters: 39:
cluster 00: {Ghana, Kosovo, Republic of Moldova, ...}
cluster 01: {Jamaica, Ethiopia, Trinidad and Tobago, ...}
cluster 02: {Kenya, Fiji}
cluster 03: {Uzbekistan, Azerbaijan, Mongolia, ...}
cluster 04: {Serbia, Islamic Republic of Iran}
cluster 05: {Turkey, Tunisia, Venezuela, ...}
cluster 06: {Chinese Taipei, Thailand, Indonesia, ...}
cluster 07: {Switzerland, Austria, Hong Kong, China, ...}
cluster 08: {Colombia, Morocco, Ecuador, ...}
cluster 09: {Ukraine, Belarus, Slovakia}
cluster 10: {Kazakhstan, Croatia, Greece}
cluster 11: {Japan}
cluster 12: {Argentina}
cluster 13: {Republic of Korea}
cluster 14: {Egypt}
cluster 15: {Israel, Dominican Republic}
cluster 16: {Mexico}
cluster 17: {Zambia, Saudi Arabia, Honduras, ...}
cluster 18: {Romania}
cluster 19: {Great Britain, Ireland}
cluster 20: {New Zealand}
cluster 21: {Australia}
cluster 22: {Canada}
cluster 23: {People's Republic of China}
cluster 24: {United States of America}
cluster 25: {Italy}
cluster 26: {Poland, Lithuania}
cluster 27: {Germany, Belgium}
cluster 28: {India}
cluster 29: {Czech Republic, Nigeria, Slovenia, ...}
cluster 30: {Spain}
cluster 31: {South Africa}
cluster 32: {Netherlands}
cluster 33: {ROC}
cluster 34: {Hungary, Montenegro}
cluster 35: {Norway, Denmark, Portugal}
cluster 36: {Sweden, Angola, Bahrain}
cluster 37: {Brazil}
cluster 38: {France}

Print the members of one of the clusters¶

print(f"Members of the cluster {tokyo_leaf_clusters[29].name}:")
for value_obj in tokyo_leaf_clusters[29].leaf_part.values:
    print(value_obj.value)

Members of the cluster {Czech Republic, Nigeria, Slovenia, ...}:
Czech Republic
Nigeria
Slovenia
Puerto Rico

Check the results with the covisualization app

tokyo_report = os.path.join("exercises", "tokyo.khcj")
khcc_tokyo.export_report_file(tokyo_report)

# To visualize uncomment the lines below
# khcc_tokyo.export_report_file("./tokyo_report.khcj")
# kh.export_report_file("./tokyo_report.khcj")

Deploy the learned coclustering model on the training data and check the obtained clusters¶

tokyo_predictions = khcc_tokyo.predict(X_tokyo[["Country", "Discipline"]])
print("Predicted clusters (first 10)")
display(tokyo_predictions[:10])

Predicted clusters (first 10)

array([['Norway', '{Norway, Denmark, Portugal}'],
       ['Spain', '{Spain}'],
       ['Italy', '{Italy}'],
       ['France', '{France}'],
       ['Chile', '{Zambia, Saudi Arabia, Honduras, ...}'],
       ['Sudan', '{Ghana, Kosovo, Republic of Moldova, ...}'],
       ['Islamic Republic of Iran', '{Serbia, Islamic Republic of Iran}'],
       ['Azerbaijan', '{Uzbekistan, Azerbaijan, Mongolia, ...}'],
       ['Netherlands', '{Netherlands}'],
       ['Australia', '{Australia}']], dtype=object)