knowledge graph embedding

A knowledge graph $\backslash mathcal\{G\}\; =\; \backslash \{E,\; R,\; F\backslash \}$ is a collection of entities $E$

, relations $R$, and facts $F$.{{Cite journal|last1=Rossi|first1=Andrea|last2=Barbosa|first2=Denilson|last3=Firmani|first3=Donatella|last4=Matinata|first4=Antonio|last5=Merialdo|first5=Paolo|date=2020|title=Knowledge Graph Embedding for Link Prediction: A Comparative Analysis|url=https://dl.acm.org/doi/10.1145/3424672|journal=ACM Transactions on Knowledge Discovery from Data|language=en|volume=15|issue=2|pages=1–49|doi=10.1145/3424672|arxiv=2002.00819|hdl=11573/1638610 |s2cid=211011226|issn=1556-4681}} A fact is a triple $(h,\; r,\; t)\; \backslash in\; F$ that denotes a link $r\; \backslash in\; R$ between the head $h\; \backslash in\; E$ and the tail $t\; \backslash in\; E$ of the triple. Another notation that is often used in the literature to represent a triple (or fact) is $\backslash langle\; \backslash text\{head\},\; \backslash text\{relation\},\; \backslash text\{tail\}\; \backslash rangle$. This notation is called resource description framework (RDF). A knowledge graph represents the knowledge related to a specific domain; leveraging this structured representation, it is possible to infer a piece of new knowledge from it after some refinement steps.{{Cite journal|last=Paulheim|first=Heiko|date=2016-12-06|editor-last=Cimiano|editor-first=Philipp|title=Knowledge graph refinement: A survey of approaches and evaluation methods|url=https://www.medra.org/servlet/aliasResolver?alias=iospress&doi=10.3233/SW-160218|journal=Semantic Web|volume=8|issue=3|pages=489–508|doi=10.3233/SW-160218|s2cid=13151033 |url-access=subscription}} However, nowadays, people have to deal with the sparsity of data and the computational inefficiency to use them in a real-world application.{{Cite journal|last1=Dai|first1=Yuanfei|last2=Wang|first2=Shiping|last3=Xiong|first3=Neal N.|last4=Guo|first4=Wenzhong|date=May 2020|title=A Survey on Knowledge Graph Embedding: Approaches, Applications and Benchmarks|journal=Electronics|language=en|volume=9|issue=5|pages=750|doi=10.3390/electronics9050750|doi-access=free}}

The embedding of a knowledge graph is a function that translates each entity and each relation into a vector of a given dimension $d$, called embedding dimension. It is even possible to embed the entities and relations with different dimensions. The embedding vectors can then be used for other tasks.

A knowledge graph embedding is characterized by four aspects:

1. Representation space: The low-dimensional space in which the entities and relations are represented.
2. Scoring function: A measure of the goodness of a triple embedded representation.
3. Encoding models: The modality in which the embedded representation of the entities and relations interact with each other.
4. Additional information: Any additional information coming from the knowledge graph that can enrich the embedded representation. Usually, an ad hoc scoring function is integrated into the general scoring function for each additional information.{{Cite book|last1=Guo|first1=Shu|last2=Wang|first2=Quan|last3=Wang|first3=Bin|last4=Wang|first4=Lihong|last5=Guo|first5=Li|title=Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) |chapter=Semantically Smooth Knowledge Graph Embedding |date=2015|chapter-url=http://aclweb.org/anthology/P15-1009 |language=en |publisher=Association for Computational Linguistics|pages=84–94|doi=10.3115/v1/P15-1009|s2cid=205692|doi-access=free}}

All algorithms for creating a knowledge graph embedding follow the same approach. First, the embedding vectors are initialized to random values. Then, they are iteratively optimized using a training set of triples. In each iteration, a batch of size $b$ triples is sampled from the training set, and a triple from it is sampled and corrupted{{emdash}}i.e., a triple that does not represent a true fact in the knowledge graph. The corruption of a triple involves substituting the head or the tail (or both) of the triple with another entity that makes the fact false. The original triple and the corrupted triple are added in the training batch, and then the embeddings are updated, optimizing a scoring function. Iteration stops when a stop condition is reached. Usually, the stop condition depends on the overfitting of the training set. At the end, the learned embeddings should have extracted semantic meaning from the training triples and should correctly predict unseen true facts in the knowledge graph.

The following is the pseudocode for the general embedding procedure.

algorithm Compute entity and relation embeddings

input: The training set $S\; =\; \backslash \{(h,\; r,\; t)\backslash \}$,

entity set $E$

,

relation set $R$,

embedding dimension $k$

output: Entity and relation embeddings

initialization: the entities $e$ and relations $r$ embeddings (vectors) are randomly initialized

while stop condition do

$S\_\{batch\}\; \backslash leftarrow\; sample(S,\; b)$ // Sample a batch from the training set

for each $(h,\; r,\; t)$ in $S\_\{batch\}$ do

$(h\text{'},\; r,\; t\text{'})\; \backslash leftarrow\; sample(S\text{'})$ // Sample a corrupted fact

$T\_\{batch\}\; \backslash leftarrow\; T\_\{batch\}\; \backslash cup\; \backslash \{((h,r,\; t),\; (h\text{'},\; r,\; t\text{'}))\; \backslash \}$

end for

Update embeddings by minimizing the loss function

end while

These indexes are often used to measure the embedding quality of a model. The simplicity of the indexes makes them very suitable for evaluating the performance of an embedding algorithm even on a large scale.{{Cite journal|last1=Chen|first1=Zhe|last2=Wang|first2=Yuehan|last3=Zhao|first3=Bin|last4=Cheng|first4=Jing|last5=Zhao|first5=Xin|last6=Duan|first6=Zongtao|date=2020|title=Knowledge Graph Completion: A Review|journal=IEEE Access|volume=8|pages=192435–192456|doi=10.1109/ACCESS.2020.3030076|s2cid=226230006|issn=2169-3536|doi-access=free|bibcode=2020IEEEA...8s2435C }} Given Q as the set of all ranked predictions of a model, it is possible to define three different performance indexes: Hits@K, MR, and MRR.

Hits@K or in short, H@K, is a performance index that measures the probability to find the correct prediction in the first top K model predictions. Usually, it is used $k=10$. Hits@K reflects the accuracy of an embedding model to predict the relation between two given triples correctly.

Hits@K$=\; \backslash frac$

Larger values mean better predictive performances.

Mean rank is the average ranking position of the items predicted by the model among all the possible items.

$MR\; =\; \backslash frac\{1\}$

The smaller the value, the better the model.

Mean reciprocal rank measures the number of triples predicted correctly. If the first predicted triple is correct, then 1 is added, if the second is correct $\backslash frac\{1\}\{2\}$ is summed, and so on.

Mean reciprocal rank is generally used to quantify the effect of search algorithms.

$MRR\; =\; \backslash frac\{1\}$

The larger the index, the better the model.

Knowledge graph completion (KGC) is a collection of techniques to infer knowledge from an embedded knowledge graph representation.{{cite arXiv|last1=Cai|first1=Hongyun|last2=Zheng|first2=Vincent W.|last3=Chang|first3=Kevin Chen-Chuan|date=2018-02-02|title=A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications|class=cs.AI|eprint=1709.07604}} In particular, this technique completes a triple inferring the missing entity or relation. The corresponding sub-tasks are named link or entity prediction (i.e., guessing an entity from the embedding given the other entity of the triple and the relation), and relation prediction (i.e., forecasting the most plausible relation that connects two entities).

Triple Classification is a binary classification problem. Given a triple, the trained model evaluates the plausibility of the triple using the embedding to determine if a triple is true or false. The decision is made with the model score function and a given threshold. Clustering is another application that leverages the embedded representation of a sparse knowledge graph to condense the representation of similar semantic entities close in a 2D space.

The use of knowledge graph embedding is increasingly pervasive in many applications. In the case of recommender systems, the use of knowledge graph embedding can overcome the limitations of the usual reinforcement learning,{{cite arXiv|last1=Zhou|first1=Sijin|last2=Dai|first2=Xinyi|last3=Chen|first3=Haokun|last4=Zhang|first4=Weinan|last5=Ren|first5=Kan|last6=Tang|first6=Ruiming|last7=He|first7=Xiuqiang|last8=Yu|first8=Yong|date=2020-06-18|title=Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning|class=cs.IR|eprint=2006.10389}}{{Cite book|last1=Liu|first1=Chan|last2=Li|first2=Lun|last3=Yao|first3=Xiaolu|last4=Tang|first4=Lin|title=2019 IEEE International Conference on Computer Science and Educational Informatization (CSEI) |chapter=A Survey of Recommendation Algorithms Based on Knowledge Graph Embedding |date=August 2019|chapter-url=https://ieeexplore.ieee.org/document/8938875|pages=168–171|doi=10.1109/CSEI47661.2019.8938875|isbn=978-1-7281-2308-0|s2cid=209459928}} as well as limitations of the conventional collaborative filtering method.{{Cite web|author = Eytan, L., Bogina, V., Ben-Gal, I., & Koenigstein, N. (2025)|title = KPAR: Knowledge-aware path-based attentive recommender with interpretability |url = https://www.iradbengal.sites.tau.ac.il/_files/ugd/901879_c776b98683fb467c839173d952810e95.pdf |publisher = ACM Transactions on Recommender Systems, 3(3), 1-23.}}

Training this kind of recommender system requires a huge amount of information from the users; however, knowledge graph techniques can address this issue by using a graph already constructed over a prior knowledge of the item correlation and using the embedding to infer from it the recommendation.

Drug repurposing is the use of an already approved drug, but for a therapeutic purpose different from the one for which it was initially designed.{{Cite journal|last1=Sosa|first1=Daniel N.|last2=Derry|first2=Alexander|last3=Guo|first3=Margaret|last4=Wei|first4=Eric|last5=Brinton|first5=Connor|last6=Altman|first6=Russ B.|date=2020|title=A Literature-Based Knowledge Graph Embedding Method for Identifying Drug Repurposing Opportunities in Rare Diseases|journal=Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing|volume=25|pages=463–474|issn=2335-6936|pmc=6937428|pmid=31797619}} It is possible to use the task of link prediction to infer a new connection between an already existing drug and a disease by using a biomedical knowledge graph built leveraging the availability of massive literature and biomedical databases.

Knowledge graph embedding can also be used in the domain of social politics.

File:KG-Embedding.svg

Given a collection of triples (or facts) , the knowledge graph embedding model produces, for each entity and relation present in the knowledge graph a continuous vector representation. $(h,\; r,\; t)$ is the corresponding embedding of a triple with $h,t\; \backslash in\; \{\backslash rm\; I\backslash !R\}^\{d\}$ and $r\; \backslash in\; \{\backslash rm\; I\backslash !R\}^\{k\}$ , where $d$ is the embedding dimension for the entities, and $k$ for the relations. The score function of a given model is denoted by $\backslash mathcal\{f\}\_\{r\}(h,\; t)$

and measures the distance of the embedding of the head from the embedding of tail given the embedding of the relation. In other words, it quantifies the plausibility of the embedded representation of a given fact.

Rossi et al. propose a taxonomy of the embedding models and identifies three main families of models: tensor decomposition models, geometric models, and deep learning models.

The tensor decomposition is a family of knowledge graph embedding models that use a multi-dimensional matrix to represent a knowledge graph, that is partially knowable due to gaps of the graph describing a particular domain thoroughly. In particular, these models use a third-order (3D) tensor, which is then factorized into low-dimensional vectors that are the embeddings. A third-order tensor is suitable for representing a knowledge graph because it records only the existence or absence of a relation between entities, and so is simple, and there is no need to know a priori the network structure,{{Cite book|last1=Nickel|first1=Maximilian|last2=Tresp|first2=Volker|last3=Kriegel|first3=Hans-Peter|chapter=A three-way model for collective learning on multi-relational data |date=2011-06-28 |chapter-url=https://dl.acm.org/doi/10.5555/3104482.3104584 |title=ICML'11: Proceedings of the 28th International Conference on International Conference on Machine Learning|publisher=Omnipress|pages=809–816|doi=|isbn=978-1-4503-0619-5}} making this class of embedding models light, and easy to train even if they suffer from high-dimensionality and sparsity of data.

This family of models uses a linear equation to embed the connection between the entities through a relation. In particular, the embedded representation of the relations is a bidimensional matrix. These models, during the embedding procedure, only use the single facts to compute the embedded representation and ignore the other associations to the same entity or relation.{{Cite journal|last1=Wang|first1=Meihong|last2=Qiu|first2=Linling|last3=Wang|first3=Xiaoli|date=2021-03-16|title=A Survey on Knowledge Graph Embeddings for Link Prediction|journal=Symmetry|language=en|volume=13|issue=3|pages=485|doi=10.3390/sym13030485|bibcode=2021Symm...13..485W|issn=2073-8994|doi-access=free}}

• DistMult{{cite arXiv|last1=Yang|first1=Bishan|last2=Yih|first2=Wen-tau|last3=He|first3=Xiaodong|last4=Gao|first4=Jianfeng|last5=Deng|first5=Li|date=2015-08-29|title=Embedding Entities and Relations for Learning and Inference in Knowledge Bases|class=cs.CL|eprint=1412.6575}}: Since the embedding matrix of the relation is a diagonal matrix, the scoring function can not distinguish asymmetric facts.
• ComplEx{{cite arXiv|last1=Trouillon|first1=Théo|last2=Welbl|first2=Johannes|last3=Riedel|first3=Sebastian|last4=Gaussier|first4=Éric|last5=Bouchard|first5=Guillaume|date=2016-06-20|title=Complex Embeddings for Simple Link Prediction|class=cs.AI|eprint=1606.06357}}: As DistMult uses a diagonal matrix to represent the relations embedding but adds a representation in the complex vector space and the hermitian product, it can distinguish symmetric and asymmetric facts.{{Cite journal|last1=Alshahrani|first1=Mona|last2=Thafar|first2=Maha A.|last3=Essack|first3=Magbubah|date=2021-02-18|title=Application and evaluation of knowledge graph embeddings in biomedical data|journal=PeerJ Computer Science|language=en|volume=7|pages=e341|doi=10.7717/peerj-cs.341|issn=2376-5992|pmc=7959619|pmid=33816992 |doi-access=free }} This approach is scalable to a large knowledge graph in terms of time and space cost.
• ANALOGY{{cite arXiv|last1=Liu|first1=Hanxiao|last2=Wu|first2=Yuexin|last3=Yang|first3=Yiming|date=2017-07-06|title=Analogical Inference for Multi-Relational Embeddings|class=cs.LG|eprint=1705.02426}}: This model encodes in the embedding the analogical structure of the knowledge graph to simulate inductive reasoning. Using a differentiable objective function, ANALOGY has good theoretical generality and computational scalability. It is proven that the embedding produced by ANALOGY fully recovers the embedding of DistMult, ComplEx, and HolE.
• SimplE{{cite arXiv|last1=Kazemi|first1=Seyed Mehran|last2=Poole|first2=David|date=2018-10-25|title=SimplE Embedding for Link Prediction in Knowledge Graphs|class=stat.ML|eprint=1802.04868}}: This model is the improvement of canonical polyadic decomposition (CP), in which an embedding vector for the relation and two independent embedding vectors for each entity are learned, depending on whether it is a head or a tail in the knowledge graph fact. SimplE resolves the problem of independent learning of the two entity embeddings using an inverse relation and average the CP score of $(h,\; r,\; t)$ and $(t,\; r^\{-1\},\; h)$. In this way, SimplE collects the relation between entities while they appear in the role of subject or object inside a fact, and it is able to embed asymmetric relations.

• HolE:{{cite arXiv|last1=Nickel|first1=Maximilian|last2=Rosasco|first2=Lorenzo|last3=Poggio|first3=Tomaso|date=2015-12-07|title=Holographic Embeddings of Knowledge Graphs|class=cs.AI|eprint=1510.04935}} HolE uses circular correlation to create an embedded representation of the knowledge graph, which can be seen as a compression of the matrix product, but is more computationally efficient and scalable while keeping the capabilities to express asymmetric relation since the circular correlation is not commutative. HolE links holographic and complex embeddings since, if used together with Fourier, can be seen as a special case of ComplEx.
• TuckER:{{Cite book|last1=Balažević|first1=Ivana|last2=Allen|first2=Carl|last3=Hospedales|first3=Timothy M.|title=Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) |chapter=TuckER: Tensor Factorization for Knowledge Graph Completion |date=2019 |pages=5184–5193|doi=10.18653/v1/D19-1522|arxiv=1901.09590|s2cid=59316623}} TuckER sees the knowledge graph as a tensor that could be decomposed using the Tucker decomposition in a collection of vectors{{emdash}}i.e., the embeddings of entities and relations{{emdash}}with a shared core. The weights of the core tensor are learned together with the embeddings and represent the level of interaction of the entries.{{cite journal|last1=Ali|first1=Mehdi|last2=Berrendorf|first2=Max|last3=Hoyt|first3=Charles Tapley|last4=Vermue|first4=Laurent|last5=Galkin|first5=Mikhail|last6=Sharifzadeh|first6=Sahand|last7=Fischer|first7=Asja|last8=Tresp|first8=Volker|last9=Lehmann|first9=Jens|title=Bringing Light into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models under a Unified Framework|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|year=2021|volume=PP|issue=12 |pages=8825–8845 |doi=10.1109/TPAMI.2021.3124805|pmid=34735335|arxiv=2006.13365|s2cid=220041612}} Each entity and relation has its own embedding dimension, and the size of the core tensor is determined by the shape of the entities and relations that interact. The embedding of the subject and object of a fact are summed in the same way, making TuckER fully expressive, and other embedding models such as RESCAL, DistMult, ComplEx, and SimplE can be expressed as a special formulation of TuckER.
• MEI:{{Cite book |last1=Tran |first1=Hung Nghiep |last2=Takasu |first2=Atsuhiro |title=Proceedings of the European Conference on Artificial Intelligence (ECAI 2020) |chapter=Multi-Partition Embedding Interaction with Block Term Format for Knowledge Graph Completion |date=2020 |chapter-url=https://ebooks.iospress.nl/doi/10.3233/FAIA200173 |pages=833–840 |series=Frontiers in Artificial Intelligence and Applications |volume=325|publisher=IOS Press |doi=10.3233/FAIA200173|arxiv=2006.16365 |s2cid=220265751 }} MEI introduces the multi-partition embedding interaction technique with the block term tensor format, which is a generalization of CP decomposition and Tucker decomposition. It divides the embedding vector into multiple partitions and learns the local interaction patterns from data instead of using fixed special patterns as in ComplEx or SimplE models. This enables MEI to achieve optimal efficiency—expressiveness trade-off, not just being fully expressive. Previous models such as TuckER, RESCAL, DistMult, ComplEx, and SimplE are suboptimal restricted special cases of MEI.
• MEIM:{{Cite book |last1=Tran |first1=Hung-Nghiep |last2=Takasu |first2=Atsuhiro |title=Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence |chapter=MEIM: Multi-partition Embedding Interaction Beyond Block Term Format for Efficient and Expressive Link Prediction |date=2022-07-16 |url=https://www.ijcai.org/proceedings/2022/314 |language=en |volume=3 |pages=2262–2269 |doi=10.24963/ijcai.2022/314|isbn=978-1-956792-00-3 |s2cid=250635995 }} MEIM goes beyond the block term tensor format to introduce the independent core tensor for ensemble boosting effects and the soft orthogonality for max-rank relational mapping, in addition to multi-partition embedding interaction. MEIM generalizes several previous models such as MEI and its subsumed models, RotaE, and QuatE. MEIM improves expressiveness while still being highly efficient in practice, helping it achieve good results using fairly small model sizes.

The geometric space defined by this family of models encodes the relation as a geometric transformation between the head and tail of a fact. For this reason, to compute the embedding of the tail, it is necessary to apply a transformation $\backslash tau$ to the head embedding, and a distance function $\backslash delta$ is used to measure the goodness of the embedding or to score the reliability of a fact.

$\backslash mathcal\{f\}\_\{r\}(h,\; t)\; =\; \backslash delta(\backslash tau(h,\; r),\; t)$

Geometric models are similar to the tensor decomposition model, but the main difference between the two is that they have to preserve the applicability of the transformation $\backslash tau$ in the geometric space in which it is defined.

This class of models is inspired by the idea of translation invariance introduced in word2vec. A pure translational model relies on the fact that the embedding vector of the entities are close to each other after applying a proper relational translation in the geometric space in which they are defined. In other words, given a fact, the embedding of the head plus the embedding of the relation should equal the embedding of the tail. The closeness of the entities embedding is given by some distance measure and quantifies the reliability of a fact.File:TransE.pdf

• TransE{{Cite book|last1=Bordes|first1=Antoine|last2=Usunier|first2=Nicolas|last3=Garcia-Durán|first3=Alberto|last4=Weston|first4=Jason|last5=Yakhnenko|first5=Oksana|chapter=Translating embeddings for modeling multi-relational data |date=May 2013 |chapter-url=https://dl.acm.org/doi/10.5555/2999792.2999923 |title=NIPS'13: Proceedings of the 26th International Conference on Neural Information Processing Systems |volume=2 |publisher=Curran Associates Inc.|pages=2787–2795}}: Uses a scoring function that forces the embeddings to satisfy a simple vector sum equation in each fact in which they appear: $h\; +\; r\; =\; t$. The embedding will be exact if each entity and relation appears in only one fact, and so in practice is poor at representing one-to-many, many-to-one, and asymmetric relations.
• TransH{{Cite book|last=Wang|first=Zhen|chapter=Knowledge Graph Embedding by Translating on Hyperplanes |title=Proceedings of the AAAI Conference on Artificial Intelligence |date=2014 |chapter-url=https://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8531 |volume=28 |doi=10.1609/aaai.v28i1.8870 |s2cid=15027084 }}: A modification of TransE for representing types of relations, by using a hyperplane as a geometric space. In TransH, the relation embedding is on a different hyperplane depending on the entities it interacts with. So, to compute, for example, the score function of a fact, the embedded representation of the head and tail need to be projected using a relational projection matrix on the correct hyperplane of the relation.
• TransR{{Cite book|last1=Lin|first1=Yankai|last2=Liu|first2=Zhiyuan|last3=Sun|first3=Maosong|last4=Liu|first4=Yang|last5=Zhu|first5=Xuan|title=Learning entity and relation embeddings for knowledge graph completion |date=2015-01-25 |url=https://dl.acm.org/doi/10.5555/2886521.2886624 |publisher=AAAI Press|pages=2181–2187|isbn=978-0-262-51129-2}}: A modification of TransH that uses different spaces embedding entities versus relations, thus separating the semantic spaces of entities and relations. TransR also uses a relational projection matrix to translate the embedding of the entities to the relation space.
• TransD:{{Cite book|last1=Ji|first1=Guoliang|last2=He|first2=Shizhu|last3=Xu|first3=Liheng|last4=Liu|first4=Kang|last5=Zhao|first5=Jun|title=Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) |chapter=Knowledge Graph Embedding via Dynamic Mapping Matrix |date=July 2015|chapter-url=https://www.aclweb.org/anthology/P15-1067 |publisher=Association for Computational Linguistics|pages=687–696|doi=10.3115/v1/P15-1067|s2cid=11202498}} In TransR, the head and the tail of a given fact could belong to two different types of entities. For example, in the fact$(Obama,\; president\backslash \_of,\; USA)$, Obama is a person and USA is a country. Matrix multiplication is an expensive procedure in TransR to compute the projection. In this context, TransD uses two vectors for each entity-relation pair to compute a dynamic mapping that substitutes the projection matrix while reducing the dimensional complexity. The first vector is used to represent the semantic meaning of the entities and relations, the second to compute the mapping matrix.
• TransA:{{cite arXiv|last1=Xiao|first1=Han|last2=Huang|first2=Minlie|last3=Hao|first3=Yu|last4=Zhu|first4=Xiaoyan|date=2015-09-27|title=TransA: An Adaptive Approach for Knowledge Graph Embedding|class=cs.CL|eprint=1509.05490}} All the translational models define a score function in their representation space, but they oversimplify this metric loss. Since the vector representation of the entities and relations is not perfect, a pure translation of $h\; +\; r$ could be distant from $t$, and a spherical equipotential Euclidean distance makes it hard to distinguish which is the closest entity. TransA, instead, introduces an adaptive Mahalanobis distance to weights the embedding dimensions, together with elliptical surfaces to remove the ambiguity.

It is possible to associate additional information to each element in the knowledge graph and their common representation facts. Each entity and relation can be enriched with text descriptions, weights, constraints, and others in order to improve the overall description of the domain with a knowledge graph. During the embedding of the knowledge graph, this information can be used to learn specialized embeddings for these characteristics together with the usual embedded representation of entities and relations, with the cost of learning a more significant number of vectors.

• STransE:{{Cite book|last1=Nguyen|first1=Dat Quoc|last2=Sirts|first2=Kairit|last3=Qu|first3=Lizhen|last4=Johnson|first4=Mark|title=Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies |chapter=STransE: A novel embedding model of entities and relationships in knowledge bases |date=June 2016 |chapter-url=https://www.aclweb.org/anthology/N16-1054 |publisher=Association for Computational Linguistics|pages=460–466|doi=10.18653/v1/N16-1054|arxiv=1606.08140|s2cid=9884935}} This model is the result of the combination of TransE and of the structure embedding in such a way it is able to better represent the one-to-many, many-to-one, and many-to-many relations. To do so, the model involves two additional independent matrix $W\_\{r\}^\{h\}$ and $W\_\{r\}^\{t\}$ for each embedded relation $r$ in the KG. Each additional matrix is used based on the fact the specific relation interact with the head or the tail of the fact. In other words, given a fact $(h,\; r,\; t)$, before applying the vector translation, the head $h$ is multiplied by $W\_\{r\}^\{h\}$ and the tail is multiplied by $W\_\{r\}^\{t\}$.
• CrossE:{{Cite book|last1=Zhang|first1=Wen|last2=Paudel|first2=Bibek|last3=Zhang|first3=Wei|last4=Bernstein|first4=Abraham|last5=Chen|first5=Huajun|title=Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining |chapter=Interaction Embeddings for Prediction and Explanation in Knowledge Graphs |date=2019-01-30 |pages=96–104|doi=10.1145/3289600.3291014|arxiv=1903.04750|isbn=9781450359405|s2cid=59516071}} Crossover interactions can be used for related information selection, and could be very useful for the embedding procedure. Crossover interactions provide two distinct contributions in the information selection: interactions from relations to entities and interactions from entities to relations. This means that a relation, e.g.'president_of' automatically selects the types of entities that are connecting the subject to the object of a fact. In a similar way, the entity of a fact inderectly determine which is inference path that has to be choose to predict the object of a related triple. CrossE, to do so, learns an additional interaction matrix $C$, uses the element-wise product to compute the interaction between $h$ and $r$. Even if, CrossE, does not rely on a neural network architecture, it is shown that this methodology can be encoded in such architecture.

This family of models, in addition or in substitution of a translation they employ a rotation-like transformation.

• TorusE:{{cite arXiv|last1=Ebisu|first1=Takuma|last2=Ichise|first2=Ryutaro|date=2017-11-15|title=TorusE: Knowledge Graph Embedding on a Lie Group|class=cs.AI|eprint=1711.05435}} The regularization term of TransE makes the entity embedding to build a spheric space, and consequently loses the translation properties of the geometric space. To address this problem, TorusE leverages the use of a compact Lie group that in this specific case is n-dimensional torus space, and avoid the use of regularization. TorusE defines the distance functions to substitute the L1 and L2 norm of TransE.
• RotatE:{{cite arXiv|last1=Sun|first1=Zhiqing|last2=Deng|first2=Zhi-Hong|last3=Nie|first3=Jian-Yun|last4=Tang|first4=Jian|date=2019-02-26|title=RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space|class=cs.LG|eprint=1902.10197}} RotatE is inspired by the Euler's identity and involves the use of Hadamard product to represent a relation $r$ as a rotation from the head $h$ to the tail $t$

in the complex space. For each element of the triple, the complex part of the embedding describes a counterclockwise rotation respect to an axis, that can be describe with the Euler's identity, whereas the modulus of the relation vector is 1. It is shown that the model is capable of embedding symmetric, asymmetric, inversion, and composition relations from the knowledge graph.

This group of embedding models uses deep neural network to learn patterns from the knowledge graph that are the input data. These models have the generality to distinguish the type of entity and relation, temporal information, path information, underlay structured information, and resolve the limitations of distance-based and semantic-matching-based models in representing all the features of a knowledge graph. The use of deep learning for knowledge graph embedding has shown good predictive performance even if they are more expensive in the training phase, data-hungry, and often required a pre-trained embedding representation of knowledge graph coming from a different embedding model.

This family of models, instead of using fully connected layers, employs one or more convolutional layers that convolve the input data applying a low-dimensional filter capable of embedding complex structures with few parameters by learning nonlinear features.

• ConvE:{{cite arXiv|last1=Dettmers|first1=Tim|last2=Minervini|first2=Pasquale|last3=Stenetorp|first3=Pontus|last4=Riedel|first4=Sebastian|date=2018-07-04|title=Convolutional 2D Knowledge Graph Embeddings|class=cs.LG|eprint=1707.01476}} ConvE is an embedding model that represents a good tradeoff expressiveness of deep learning models and computational expensiveness, in fact it is shown that it used 8x less parameters, when compared to DistMult. ConvE uses a one-dimensional $d$-sized embedding to represent the entities and relations of a knowledge graph. To compute the score function of a triple, ConvE apply a simple procedure: first concatenes and merge the embeddings of the head of the triple and the relation in a single data [h; \mathcal{r}], then this matrix is used as input for the 2D convolutional layer. The result is then passed through a dense layer that apply a linear transformation parameterized by the matrix $\backslash mathcal\{W\}$ and at the end, with the inner product is linked to the tail triple. ConvE is also particularly efficient in the evaluation procedure: using a 1-N scoring, the model matches, given a head and a relation, all the tails at the same time, saving a lot of evaluation time when compared to the 1-1 evaluation program of the other models.
• ConvR:{{Cite book|last1=Jiang|first1=Xiaotian|last2=Wang|first2=Quan|last3=Wang|first3=Bin|title=Proceedings of the 2019 Conference of the North |chapter=Adaptive Convolution for Multi-Relational Learning |date=June 2019 |chapter-url=https://www.aclweb.org/anthology/N19-1103 |publisher=Association for Computational Linguistics|pages=978–987|doi=10.18653/v1/N19-1103|s2cid=174800352}} ConvR is an adaptive convolutional network aimed to deeply represent all the possible interactions between the entities and the relations. For this task, ConvR, computes convolutional filter for each relation, and, when required, applies these filters to the entity of interest to extract convoluted features. The procedure to compute the score of triple is the same as ConvE.
• ConvKB:{{Cite book|last1=Nguyen|first1=Dai Quoc|last2=Nguyen|first2=Tu Dinh|last3=Nguyen|first3=Dat Quoc|last4=Phung|first4=Dinh|title=Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) |chapter=A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network |date=2018 |pages=327–333|doi=10.18653/v1/N18-2053|arxiv=1712.02121|s2cid=3882054}} ConvKB, to compute score function of a given triple $(h,\; r,\; t)$, it produces an input [h; \mathcal{r}; t]of dimension $d\; \backslash times\; 3$ without reshaping and passes it to series of convolutional filter of size $1\; \backslash times\; 3$. This result feeds a dense layer with only one neuron that produces the final score. The single final neuron makes this architecture as a binary classifier in which the fact could be true or false. A difference with ConvE is that the dimensionality of the entities is not changed.

This family of models uses capsule neural networks to create a more stable representation that is able to recognize a feature in the input without losing spatial information. The network is composed of convolutional layers, but they are organized in capsules, and the overall result of a capsule is sent to a higher-capsule decided by a dynamic process routine.

• CapsE:{{cite arXiv|last1=Nguyen|first1=Dai Quoc|last2=Vu|first2=Thanh|last3=Nguyen|first3=Tu Dinh|last4=Nguyen|first4=Dat Quoc|last5=Phung|first5=Dinh|date=2019-03-06|title=A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization|class=cs.CL|eprint=1808.04122}} CapsE implements a capsule network to model a fact $(h,\; r,\; t)$. As in ConvKB, each triple element is concatenated to build a matrix [h; \mathcal{r}; t]and is used to feed to a convolutional layer to extract the convolutional features. These features are then redirected to a capsule to produce a continuous vector, more the vector is long, more the fact is true.

This class of models leverages the use of recurrent neural network. The advantage of this architecture is to memorize a sequence of fact, rather than just elaborate single events.

• RSN:{{cite arXiv|last1=Guo|first1=Lingbing|last2=Sun|first2=Zequn|last3=Hu|first3=Wei|date=2019-05-13|title=Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs|class=cs.AI|eprint=1905.04914}} During the embedding procedure is commonly assumed that, similar entities has similar relations. In practice, this type of information is not leveraged, because the embedding is computed just on the undergoing fact rather than a history of facts. Recurrent skipping networks (RSN) uses a recurrent neural network to learn relational path using a random walk sampling.

The machine learning task for knowledge graph embedding that is more often used to evaluate the embedding accuracy of the models is the link prediction. Rossi et al. produced an extensive benchmark of the models, but also other surveys produces similar results. The benchmark involves five datasets FB15k, WN18, FB15k-237,{{Cite book|last1=Toutanova|first1=Kristina|last2=Chen|first2=Danqi|title=Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality |chapter=Observed versus latent features for knowledge base and text inference |date=July 2015 |chapter-url=https://www.aclweb.org/anthology/W15-4007 |publisher=Association for Computational Linguistics|pages=57–66|doi=10.18653/v1/W15-4007|s2cid=5378837|doi-access=free}} WN18RR, and YAGO3-10.{{Cite journal|last1=Mahdisoltani|first1=F.|last2=Biega|first2=J.|last3=Suchanek|first3=Fabian M.|date=2015|title=YAGO3: A Knowledge Base from Multilingual Wikipedias|journal=CIDR|s2cid=6611164}} More recently, it has been discussed that these datasets are far away from real-world applications, and other datasets should be integrated as a standard benchmark.{{cite arXiv|last1=Hu|first1=Weihua|last2=Fey|first2=Matthias|last3=Zitnik|first3=Marinka|last4=Dong|first4=Yuxiao|last5=Ren|first5=Hongyu|last6=Liu|first6=Bowen|last7=Catasta|first7=Michele|last8=Leskovec|first8=Jure|date=2021-02-24|title=Open Graph Benchmark: Datasets for Machine Learning on Graphs|class=cs.LG|eprint=2005.00687}}

• {{GitHub|https://github.com/uma-pi1/kge |KGE}}
• {{GitHub|https://github.com/tranhungnghiep/MEI-KGE|MEI-KGE}}
• {{GitHub|https://github.com/Sujit-O/pykg2vec |Pykg2vec}}
• {{GitHub|https://github.com/awslabs/dgl-ke |DGL-KE}}
• {{GitHub|https://github.com/pykeen/pykeen |PyKEEN}}
• {{GitHub|https://github.com/torchkge-team/torchkge |TorchKGE}}
• {{GitHub|https://github.com/Accenture/AmpliGraph |AmpliGraph}}
• {{GitHub|https://github.com/thunlp/OpenKE |OpenKE}}
• {{GitHub|https://github.com/mnick/scikit-kge |scikit-kge}}
• {{GitHub|https://github.com/thunlp/Fast-TransX |Fast-TransX}}
• {{GitHub|https://github.com/tranhungnghiep/MEIM-KGE|MEIM-KGE}}
• {{GitHub|https://github.com/dice-group/dice-embeddings|DICEE}}

• Knowledge graph
• Embedding
• Machine learning
• Knowledge base
• Knowledge extraction
• Statistical relational learning
• Representation learning
• Graph embedding

{{Scholia|topic}}

• [https://ogb.stanford.edu Open Graph Benchmark - Stanford]
• [https://wordnet.princeton.edu/ WordNet - Princeton]

Category:Knowledge graphs

Category:Machine learning

Category:Graph algorithms

Category:Information science