Attribute Reduction of Hybrid Decision Information Systems Based on Fuzzy Conditional Information Entropy

Symbols and Notations Used in the Paper

1  Introduction

1.1 Research Background

In information systems, the attributes of data are not of equal importance. Some of them may not have a direct or significant impact on decision-making. These attributes are considered as redundant. The objective of attribute reduction is to eliminate these redundant attributes, while maintaining the same classification accuracy, to facilitate more concise and efficient decision-making. Attribute reduction is also called feature selection.

With the emergence of the era of big data, the intricacies and diversities of data have been constantly on the rise. Many datasets encompass both nominal and real-valued attributes, which are referred to as hybrid data. Such hybrid data can be found in various domains, including finance, healthcare, and e-commerce. The hybrid data encompass diverse attributes, necessitating distinct methodologies and strategies for their manipulation. The complexity intensifies the challenges of attribute reduction. Primarily, the disparities among nominal attribute values prove challenging to assess accurately using Euclidean distance. Furthermore, data may encompass uncertainties such as fuzzy attributes and anomalous attribute values. The reduction of attributes in hybrid data is a pivotal research topic in the field of data mining and lies at the heart of rough set theory. This line of research not only enhances the efficiency and precision of data processing but also fosters the advancement of data mining and machine learning. Consequently, exploring the attribute reduction problem in mixed information systems holds profound theoretical significance and practical relevance [1].

1.2 Related Works

The classical rough set theory is limited in its ability to handle continuous data, hence it has been enhanced to include neighborhood rough sets and fuzzy rough sets as extensions. Currently, numerous scholars have delved into the intricacies of attribute reduction by using neighborhood rough sets and fuzzy rough sets, ultimately yielding impressive outcomes.

Neighborhood rough sets can effectively handle continuous values and have been widely applied in attribute reduction. Fan et al. [2] established a max-decision neighborhood rough set model and successfully applied it to achieve attribute reduction for information systems, leading to more accurate and concise decision-making. Shu et al. [3] proposed a neighborhood entropy-based incremental feature selection framework for hybrid data based on the neighborhood rough set model. In their study, Zhang et al. [4] conducted the entropy measurement on the gene space, utilizing the neighborhood rough sets, for effective gene selection. Zhou et al. [5] put forward an online stream-based attribute reduction method based on the new neighborhood rough set theory, eliminating the need for domain knowledge or pre-parameters. Xia et al. [6] built a granular ball neighborhood rough set method for fast adaptive attribute reduction. Liao et al. [7] developed an effective attribute reduction method for handling hybrid data under test cost constraints based on the generalized confidence level and vector neighborhood rough set models. None of the above methods provides a specific strategy for handling the differences between nominal attribute values. Instead, they use Euclidean distance to measure the differences between nominal attribute values, or use category information to simply define the differences between nominal attribute values as 1 or 0.

Fuzzy rough sets are capable of effectively dealing with the ambiguity in real-world scenarios, and have also been successfully applied in attribute reduction. Yuan et al. [8] established a unsupervised attribute reduction model for hybrid data based on fuzzy rough sets. Zeng et al. [9] examined the incremental updating mechanism of fuzzy rough approximations in response to attribute value changes in a hybrid information system. Yang et al. [10] considered the uncertainty measurement and attribute reduction for multi-source fuzzy information systems based on a new multi-granulation rough sets model. Wang et al. [11] utilized variable distance parameters and, drawing from fuzzy rough set theory, developed an iterative attribute reduction model. Singh et al. [12] came up with a fuzzy similarity-based rough set approach for attribute reduction in set-valued information systems; Jain et al. [13] created an attribute reduction model by using the intuitionistic fuzzy rough set. The above methods are either not suitable for handling mixed data or do not have special techniques for handling nominal attribute values and abnormal attribute values.

Information entropy quantifies the uncertainty of knowledge and can be used for the attribute reduction. Li et al. [14] used conditional information entropy to measure the uncertainty and reduce the attributes of multi-source incomplete information systems. Vergara et al. [15] summarized the feature selection methods based on the mutual information. Li et al. [16] measured the uncertainty of gene spaces and designed a gene seleciton algorithm using information entropy. Zhang et al. [17] defined a fuzzy information entropy of classification data based on the constructed fuzzy information structure and used it for attribute reduction. Zhang et al. [18] constructed a hybrid data attribute reduction model utilizing the λ-conditional entropy derived from fuzzy rough set theory. Sang et al. [19] proposed an incremental attribute reduction method using the fuzzy dominance neighborhood conditional entropy derived from fuzzy dominant neighborhood rough sets. Huang et al. [20] defined a new fuzzy conditional information entropy for fuzzy β covering information systems and applied it to attribute reduction. The above methods also do not provide a specific solution for handling mixed data.

Akram et al. [21,22] presented the concept of attribute reduction and designed the associated attribute reduction algorithms based on the discernibility matrix and discernibility function. These methods are intuitive and easy to comprehend, enabling the calculation of all reducts. However, they exhibit a high computational complexity, rendering them unsuitable for large datasets.

The comprehensive overview of the strengths and weaknesses of diverse attribute reduction algorithms can be found in Table 1.

1.3 Motivation and Contribution

Hybrid data is a common occurrence in data mining. They have been observed that nominal attributes significantly impact the measurement of data similarity, and Euclidean distance do not work well when dealing with nominal attribute values, yet the majority of current reduction algorithms rely heavily on Euclidean distance. Therefore, the supervised distance, a novel metric specifically for measuring the difference between nominal attribute values, has been meticulously crafted. Existing attribute reduction algorithms often exhibit sensitivity to abnormal attribute values and lack an efficient mechanism to address them. Consequently, this article introduces a novel fuzzy relationship that replaces similarity measurements based on attribute values with the count of similar attributes. This innovative approach effectively filters out abnormal attribute values, enhancing the overall accuracy and robustness of the algorithm. Due to the fact that the fuzzy conditional information entropy not only considers the fuzziness of data but also has a certain anti-noise ability, this paper uses fuzzy conditional information entropy to reduce the attributes of mixed data. The article’s novelty and contribution are outlined below:

1. A new distance metric called the supervised distance is introduced. The supervised distance metric, which considers the decision attribute that affect attribute similarity, leads to more precise attribute reduction.

2. Based on the new distance measurement, a new fuzzy relationship determined by the quantity of attributes with similar values is established. This approach views the relationship from the perspective of “feedback on parity of attribute values to attribute sets”, resulting in a fuzzy relation that is robust to a few abnormal attributes.

3. The fuzzy conditional information entropy is defined based on the new fuzzy relationship and an advanced attribute reduction algorithm is developed based on the fuzzy conditional information entropy. This algorithm incorporates an innovative distance function, an innovative fuzzy relationship, and fuzzy conditional information entropy.

4. Through meticulous experimental validation, this study clearly demonstrates that the fuzzy relation offers superior performance to the equivalence relation when dealing with hybrid data, effectively handling its complexity and uncertainties.

1.4 Organization

The structure of this paper is outlined as follows. Section 2 provides a brief overview of fuzzy relationships and HDISs, laying the foundation for the subsequent sections. Section 3 describes the method in detail. Section 4 presents the experimental results and discusses them. Section 5 summarizes the paper.

2  Preliminaries

In this section, we will review some fundamental concepts of fuzzy relations and HDISs.

2.1 Fuzzy Relation

Let I=[0,1]. A fuzzy set F on X is known as a mapping μ:X→I. ∀x∈X, μ(x) is the degree of membership of x in F [23]. F can be represented as follows:

F=μ(x1)x1+μ(x2)x2+⋯+μ(xn)xn.(1)

Let |F|=∑i=1nμ(xi). Then |F| is called the cardinality of F.

Let IX denote the family of all fuzzy sets on X.

∀F∈IX and ∀G∈2X, F∩G is defined as follows:

(F∩G)(x)=F(x)∧G(x)={F(x),x∈G0,x∉G.(2)

Then |F∩G|=|F|.

If R is a fuzzy set on X×X, then R is referred to as a fuzzy relation on X. R can be represented by the matrix: M(R)=(rij)n×n (rij=R(xi,xj)∈I denotes the similarity between xi and xj).

Let IX×X denote the set of all fuzzy relationship on X.

Definition 2.1. Reference [24] Let R be a fuzzy relation on X. Then R is

1) Reflexive, if R(x,x) = 1 (∀x∈X).

2) Symmetric, if R(x,y) = R(y, x) (∀x,y∈X).

3) Transitive, if R(x,z)≥R(x,y)∧R(y,z)(∀x,y,z∈X).

If R is reflexive, symmetric, and transitive, then R is called a fuzzy equivalence relation on X. If R is reflexive and symmetric, then R is called a fuzzy tolerance relation on X.

Let R∈IX×X. ∀x∈X, the fuzzy set Rx on X is defined as follows:

Rx(y)=R(x,y)(y∈X).(3)

Then Rx is called the fuzzy information granule of x.

According to (1), Rx=R(x,x1)x1+R(x,x2)x2+⋯+R(x,xn)xn.

Then |Rx|=∑i=1nR(x,xi).

2.2 Hybrid Decision Information Systems

Definition 2.2. Reference [25] (X,C) is called an information system (IS), if ∀c∈C decides a function f:X→Vc, where Vc={f(x):x∈X}. If Q⊆C, then (X,Q) is said to be a subsystem of (X,C).

If C=B∪{d}, where B={b1,b2,⋯,bm} represents the conditional attributes set, d represents the decision attribute, then (X,C) is called as a decision information system.

Definition 2.3. Let (X,B,d) is a decision information system. Then (X,B,d) is known as a hybrid decision information system (HDIS), if B=Bc∪Br, where Bc is a set of categorical attributes, Br is a set of real-valued attributes.

Example 2.4. Table 2 shows an HDIS, where X={x1,x2,⋯,x9}, Bc={b1,b2} and Br={b3}.

3  Methodology

3.1 A New Distance Function

To accurately measure the difference between two attributes in HDISs, we have developed a novel distance function that accounts for various types of attributes and missing data. This innovative approach allows for a more comprehensive assessment of the similarity or dissimilarity between different attributes in the system.

We first assign a definition for the distance between categorical attribute values to enable further distance definition of hybrid data.

Definition 3.1. For an HDIS (X,B,d), let Vd={d(x):x∈X}={d1,d2,⋯,dr},

N(b,x)=|{y∈X:b(x)=b(y),b∈Bc,b(x)≠∗}|(∀x∈X)

Ni(b,x)=|{y∈X:b(x)=b(y),d(y)=di}|

Then N(b,x)=∑i=1rNi(b.x).

Example 3.2. (Continue with Example 2.4)

Vd={d1=Flu,d2=Rhinitis,d3=Health},N(b2,x5)=|{x∈X:b2(x)=b2(x5)=Yes}|=|{x1,x2,x4,x5,x9}|=5,N1(b2,x5)=|{x∈X:b2(x)=b2(x5)=Yes,d(x)=d1=Flu}|=|{x1,x2,x4}|=3N2(b2,x5)=|{x∈X:b2(x)=b2(x5)=Yes,d(x)=d2=Rhinitis}|=|{x5}|=1N3(b2,x5)=|{x∈X:b2(x)=b2(x5)=Yes,d(x)=d3=Health}|=|{x9}|=1.

Definition 3.3. For an HDIS (X,B,d), let |Vd|=r, b∈Bc, x∈X, y∈X, b(x)≠∗ and b(y)≠∗. Then the distance between b(x) and b(y) is defiend as follows:

ρc(b(x),b(y))=12∑i=1r|Ni(b,x)N(b,x)−Ni(b,y)N(b,y)|.(4)

The distribution of attribute values defines a distance that is well-suited for the characteristics of classification data.

Proposition 3.4. Let (X,B,d) be an HDIS. Then the following conclusions are valid:

1)   ρc(b(x),b(x))=0,

2)   0≤ρc(b(x),b(y))≤1.

Proof. 1) The conclusion is self-evidently valid.

    2) ρc(b(x),b(y))≥0 is self-evidently valid.

Since

∑i=1rNi(b,x)N(b,x)=∑i=1rNi(b,y)N(b,y)=1,

we have

ρc(b(x),b(y))≤12(∑i=1rNi(b,x)N(b,x)+∑i=1rNi(b,y)N(b,y))=1.

Definition 3.5. For an HDIS (X,B,d), let b∈Br, x∈X, y∈X, b(x)≠∗, and b(y)≠∗. Then the distance between b(x) and b(y) is defined as follows:

ρr(b(x),b(y))=|b(x)−b(y)|max{b(x):x∈X}−min{b(x):x∈X}.(5)

ρr(b(x),b(x))=0 and 0≤ρr(b(x),b(y))≤1 are self-evidently valid.

Definition 3.6. For an HDIS (X,B,d), let b∈B, x∈X, and y∈X. Then the distance between b(x) and b(y) is defined as follows:

ρ(b(x),b(y))={0,b∈B,b(x)=∗∨b(y)=∗,d(x)=d(y)1,b∈B,b(x)=∗∨b(y)=∗,d(x)≠d(y)ρc(b(x),b(y)),b∈Bc,b(x)≠∗∧b(y)≠∗ρr(b(x),b(y)),b∈Br,b(x)≠∗∧b(y)≠∗.(6)

Example 3.7. (Continuation of Example 2.4) According to Definitions 3.1–3.6, we have

1) ρ(b1(x1),b1(x3))=(|22−12|+|02−12|+|02−02|)/3=13,

2) ρ(b1(x1),b1(x4))=(|22−13|+|02−03|+|02−23|)/3=49,

3) ρ(b1(x5),b1(x9))=1,

4) ρ(b2(x1),b2(x3))=0,

5) ρ(b3(x1),b3(x3))=|39.5−39|40−36=18.

Definition 3.8. For an HDIS (X,B,d), ∀b∈B, ∀xi,xj∈X, let Mb=(ρ(b(xi),b(xj)))n×n. Then Mb is called the distance matrix of b.

Example 3.9. (Continuation of Example 2.4) Here are the distance matrices of b1 and b3:

Mb1=(001/34/911/34/94/91001/34/911/34/94/911/31/304/9104/94/914/94/94/9014/90011111001111/31/304/9004/94/914/94/94/9014/90004/94/94/9014/9000111111000),

Mb3=(01/81/83/83/417/815/81/801/41/27/81113/41/81/401/45/813/411/23/81/21/403/811/211/43/47/85/83/8001/811/81111001117/813/41/21/81001/41111110005/813/41/21/411/400).

3.2 A New Fuzzy Relation

Definition 3.10. For an HDIS (X,B,d), ∀P⊆B, ∀θ∈[0,1], the tolerance relationship is defined as follows:

TPθ={(x,y)∈X×X:∀b∈P,ρ(b(x),b(y))≤θ}.

Let TPθ(x)={y∈X:(x,y)∈TPθ}. TPθ(x) is called the tolerance class of x.

TPθ uses distance to measure the similarity of objects. Next, we introduce a new fuzzy relationship that evaluates the similarity of objects based on the number of similar attributes.

Definition 3.11. Let (X,B,d) be an HDIS. ∀P⊆B, ∀θ∈[0,1], ∀x∈X, ∀y∈X, define

RPθ(x,y)=1|B||{b∈P:ρ(b(x),b(y))≤θ}|,Rd={(x,y)∈X×X:d(x)=d(y)}

Then RPθ and Rd are a fuzzy relation and an equivalence relation on X, respectively.

The matrix representation of RPθ is M(RPθ)=(RPθ(xixj))n×n.

Let RPθx(y)=RPθ(x,y) and Rdx={y∈X:(x,y)∈Rd}.

RPθx is the fuzzy information granule of x and Rdx is the decision class of x.

According to (1), RPθx=RPθ(x,x1)x1+RPθ(x,x2)x2+⋯+RPθ(x,xn)xn,|RPθx|=∑i=1nRPθ(x,xi)

Let X/Rd={Rdx:x∈X}={D1,D2,⋯,Dr}.

Let Pθxy={b∈P:ρ(b(x),b(y))≤θ}. Then RPθx(y)=1|B||Pθxy|.

RPθx(x)=|P||B|≤1 and RPθx(y)≤|P||B| are self-evidently valid.

Proposition 3.12. Let (X,B,d) be an HDIS. If P⊆B, P1⊆P2⊆B, and 0≤θ1≤θ2≤1, RP1θx⊆RP2θx and RPθ1x⊆RPθ2x(∀x∈X).

Proof. According to Definition 3.11, ∀y∈X, ∀θ∈[0,1], RP1θx(y)=1|B||P1θxy|, RP2θx(y)=1|B||P2θxy|.

Since P1⊆P2, P1θxy⊆P2θxy. Therefore, ∀y∈X, RP1θx(y)≤RP2θx(y).

Thus, ∀x∈X, RP1θx⊆RP2θx.

Since 0≤θ1≤θ2≤1, Pθ1xy⊆Pθ2xy. Thus RPθ1x(y)≤RPθ2x(y). Therefore, RPθ1x⊆RPθ2x.

3.3 Fuzzy Conditional Information Entropy in HDISs

This section explores the concept of fuzzy entropy measures in HDISs to measure the uncertainty of HDISs.

Definition 3.13. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. Then the fuzzy information entropy of P is defined as follows:

Hθ(P)=−∑i=1n|RPθxi|nlog2|RPθxi|n.(7)

Proposition 3.14. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. The following inequality holds:

0≤Hθ(P)≤nlog2mn|P|.

Proof. Since |P|m≤|RPθxi|≤n, |P|mn≤|RPθxi|n≤1.

Thus, 0≤−log2|RPθxi|n≤log2mn|P|. Hence, 0≤−|RPθxi|nlog2|RPθxi|n≤log2mn|P|.

The following conclusion follows directly from Definition 3.13:

0≤Hθ(P)≤nlog2mn|P|.

Definition 3.15. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. The fuzzy conditional information entropy of P concerning d is defined as follows:

Hθ(P|d)=−∑i=1n∑j=1r|RPθxi∩Dj|nlog2|RPθxi∩Dj||RPθxi|.(8)

Lemma 3.16. (X,B,d) is an HDIS. Let P⊆B, θ∈[0,1] and D∈X/d. The following conclusion is valid:

(RPθx∩D)(y)+(RPθx∩(X−D))(y)=RPθ(x,y)(∀x,y∈X).

Proof. ∀x,y∈X, we have

(RPθx∩D)(y)=RPθ(x,y)∧D(y)={RPθ(x,y),y∈D0,y∉D,

(RPθx∩(X−D))(y)=RPθ(x,y)∧(X−D)(y)={0,y∈DRPθ(x,y),y∉D.

Thus, (RPθx∩D)(y)+(RPθx∩(X−D))(y)=RPθ(x,y).

Proposition 3.17. (X,B,d) is an HDIS. Let P⊆B, θ∈[0,1] and D∈X/d. The following conclusion is valid:

|RPθx∩D|+|RPθx∩(X−D)|=|RPθx|(∀x∈X).

Proof. According to Lemma 3.16,

|RPθx∩D|+|RPθx∩(X−D)|=∑i−1n(RPθx∩D)(xi)+∑i=1n(RPθx∩(X−D))(xi)=∑i−1n[(RPθx∩D)(xi)+(RPθx∩(X−D))(xi)]=∑i=1nRPθ(x,xi)=|RPθx|.

Proposition 3. 18. (X,B,d) is an HDIS.

1) If Q⊆P⊆B, then Hθ(Q|d)≤Hθ(P|d).

2) If 0≤θ1≤θ2≤1, then Hθ1(P|d)≤Hθ2(P|d)(∀P⊆B).

Proof. 1) Let pij(1)=|RPθxi∩Dj|, pij(2)=|RPθxi∩(X−Dj)|, qij(1)=|RQθxi∩Dj|, and qij(2)=|RQθxi∩(X−Dj)|.

RQθxi⊆RPθxi according to Proposition 3.12.

Hence, 0≤qij(1)≤pij(1) and 0≤qij(2)≤pij(2).

According to Proposition 3.17, we have pij(1)+pij(2)=|RPθxi| and qij(1)+qij(2)=|RQθxi|.

Let f(x,y)=−xlog2xx+y(x>0,y>0).

Then Hθ(P|d)=−∑i=1n⁡∑j=1r⁡|RPθxi∩Dj|nlog2|RPθxi∩Dj||RPθxi|=−∑i=1n⁡∑j=1r⁡pij(1)nlog2pij(1)pij(1)+pij(2)=1n∑i=1n⁡∑j=1r⁡f(pij(1),pij(2)) and

Hθ(Q|d)=−∑i=1n⁡∑j=1r⁡|RQθxi∩Dj|nlog2|RQθxi∩Dj||RQθxi|=−∑i=1n⁡∑j=1r⁡qij(1)nlog2qij(1)qij(1)+qij(2)=1n∑i=1n⁡∑j=1r⁡f(qij(1),qij(2)).

Since the function f(x, y) exhibits monotonic increases in both x and y,

f(qij(1),qij(2))≤f(pij(1),qij(2))≤f(pij(1),pij(2)).

Hence, Hθ(Q|d)≤Hθ(P|d).

2) Let sij(1)=|RPθ1xi∩Dj|, sij(2)=|RPθ1xi∩(X−Dj)|, tij(1)=|RPθ2xi∩Dj|, and tij(2)=|RPθ2xi∩(X−Dj)|.

RPθ1xi⊆RPθ2xi according to Proposition 3.12. Hence, 0≤sij(1)≤tij(1) and 0≤sij(2)≤tij(2).

According to Proposition 3.17, we have sij(1)+sij(2)=|RPθ1xi| and tij(1)+tij(2)=|RPθ2xi|.

Hθ1(P|d)=−∑i=1n∑j=1r|RPθ1xi∩Dj|nlog2|RPθ1xi∩Dj||RPθ1xi|=−∑i=1n∑j=1rsij(1)nlog2sij(1)sij(1)+sij(2)=1n∑i=1n∑j=1rf(sij(1),sij(2))

Hθ2(P|d)=−∑i=1n∑j=1r|RPθ2xi∩Dj|nlog2|RPθ2xi∩Dj||RPθ2xi|=−∑i=1n∑j=1rtij(1)nlog2tij(1)tij(1)+tij(2)=1n∑i=1n∑j=1rf(tij(1),tij(2))

It follows from the monotonicity of f(x,y) that f(sij(1),sij(2))≤f(tij(1),sij(2))≤f(tij(1),tij(2)).

Hence Hθ1(P|d)≤Hθ2(P|d).

Definition 3.19. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. The fuzzy joint information entropy of P and d is defined as follows:

Hθ(P∪d)=−∑i=1n∑j=1r|RPθxi∩Dj|nlog2|RPθxi∩Dj|n.(9)

Lemma 3.20. (X,B,d) is an HDIS. Let P⊆B, θ∈[0,1], U∈2X, and V∈2X, then

∑y∈URPθ(x,y)+∑y∈VRPθ(x,y)≥∑y∈U∪VRPθ(x,y)(∀x∈X)

The equality holds as U∩V=Φ.

Proof. The conclusion is self-evident.

Proposition 3.21. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. Then

∑j=1r|RPθx∩Dj|=|RPθx|(∀x∈X).

Proof. Since D={D1,D2,⋯,Dr} constitutes a partition of X,

∑j=1r|RPθx∩Dj|=∑j=1r∑y∈Dj(RPθ(x,y)∧Dj(y))=∑y∈X(RPθ(x,y)∧D(y))=∑y∈XRPθ(x,y)=|RPθx|(∀x∈X)by Lemma 3.20.

Proposition 3.22. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. Then

Hθ(P|d)=Hθ(P∪d)−Hθ(P).

Proof. ∑j=1r⁡|RPθxi∩Dj|=|RPθxi| according to Proposition 3.21.

Hθ(P|d)=−∑i=1n∑j=1r|RPθxi∩Dj|nlog2|RPθxi∩Dj||RPθxi|=−∑i=1n∑j=1r|RPθxi∩Dj|n(log2|RPθxi∩Dj|n−log2|RPθxi|n)=−∑i=1n∑j=1r|RPθxi∩Dj|nlog2|RPθxi∩Dj|n+∑i=1n∑j=1r|RPθxi∩Dj|nlog2|RPθxi|n=Hθ(P∪d)+∑i=1n|RPθxi|nlog2|RPθxi|n=Hθ(P∪d)−Hθ(P).

Theorem 3.23. (X,B,d) is an HDIS. Let P⊆B and θ∈[0,1]. Then Hθ(P|d)≥0.

Proof. Hθ(P)=−∑i=1n⁡|RPθxi|nlog2|RPθxi|n according to Definition 3.13.

∑j=1r⁡|RPθxi∩Dj|=|RPθxi| according to Proposition 3.21.

Thus, Hθ(P)=−∑i=1n⁡∑j=1r⁡|RPθxi∩Dj|nlog2|RPθxi|n=−∑i=1n⁡∑j=1r⁡|RPθxi∩Dj|nlog2|RPθxi|n.