On Generalization Bounds of a Family of Recurrent Neural Networks

AI helps you lot reading Science

AI Insight

AI extracts a summary of this paper

---

Weibo :

Generalization and Representational Limits of Graph Neural Networks

ICML, pp.3419-3430, (2020)

Abstruse

We accost two fundamental questions about graph neural networks (GNNs). First, we prove that several of import graph properties cannot be computed by GNNs that rely entirely on local information. Such GNNs include the standard message passing models, and more powerful spatial variants that exploit local graph structure (eastward.g., via relat... More than

0

Introduction

• Graph neural networks (Scarselli et al, 2009; Gori et al, 2005), in their diverse incarnations, have emerged as models of choice for embedding graph-structured data from a diverse gear up of domains, including molecular structures, noesis graphs, biological networks, social networks, and n-body problems (Duvenaud et al, 2015; Defferrard et al, 2016; Battaglia et al, 2016; Zhang et al, 2018b; Santoro et al, 2018; Yun et al, 2019; Jin et al, 2019).

  The working of a graph neural network (GNN) on an input graph, with a feature vector associated with each node, can be outlined equally follows.

• (one) Commencement, the authors show that there exist unproblematic graphs that cannot exist distinguished by GNNs that generate node embeddings solely based on local information.

Highlights

• Graph neural networks (Scarselli et al, 2009; Gori et al, 2005), in their various incarnations, have emerged every bit models of choice for embedding graph-structured data from a various fix of domains, including molecular structures, noesis graphs, biological networks, social networks, and n-body problems (Duvenaud et al, 2015; Defferrard et al, 2016; Battaglia et al, 2016; Zhang et al, 2018b; Santoro et al, 2018; Yun et al, 2019; Jin et al, 2019).

  The working of a graph neural network (GNN) on an input graph, with a characteristic vector associated with each node, tin be outlined as follows

• We focus on classification: (1) a graph neural networks with learnable parameters embeds the nodes of each input graph, (2) the node embeddings are combined into a single graph vector via a readout function such equally sum, boilerplate, or elementwise maximum, and (iii) a parameterized classifier makes a binary prediction on the resulting graph vector
• (ane) First, we show that in that location exist simple graphs that cannot be distinguished by graph neural networks that generate node embeddings solely based on local information
• (2) We introduce a novel graph-theoretic formalism for analyzing Consequent Port Numbering GNN, and our constructions provide insights that may facilitate the design of more than effective graph neural networks
• (three) We provide the first data dependent generalization bounds for message passing graph neural networks
• Nosotros innovate a graph-theoretic formalism for Consequent Port Numbering GNN that obviates the need for an explicit bijection and is easier to check

Results

• (4) The authors' generalization assay accounts for local permutation invariance of the GNN assemblage role.
• The embedding of node v is updated by processing the information from its neighbors equally an ordered set, ordered past the port numbering, i.eastward., the aggregation function is mostly non permutation invariant.
• At that place exist consistent port orderings such that CPNGNNs with permutation-invariant readout cannot make up one's mind several of import graph properties such as girth, circumference, diameter, radius, conjoint cycle, total number of cycles, and k-clique.
• A natural question that arises is whether the authors can obtain a more than expressive model than both CPNGNN and DimeNet. Leveraging insights from the constructions, the authors introduce one such variant, H-DCPN, that generalizes both CPNGNN and DimeNet. More than powerful GNNs. The main thought is to augment DimeNet not simply with port ordering, and additional spatial information.
• The authors' premises are comparable to the Rademacher bounds for RNNs. The authors consider locally permutation invariant GNNs, where in each layer l, the embedding hlv ∈ Rr of node five of a given input graph is updated by aggregating the embeddings of its neighbors, u ∈ North (v), via an aggregation function ρ : Rr → Rr. Unlike types of updates are possible; the authors focus on a hateful field update (Dai et al, 2016; Jin et al, 2018, 2019): hlv = φ W1xv + W2ρ( u∈N(v) g) , (ii)
• The authors demand to bound the empirical Rademacher complexity RG(Jγ) for GNNs. The authors practice this in ii steps: (1) the authors evidence that it is sufficient to bound the Rademacher complexity of local node-wise computation trees; (2) the authors jump the complication for a unmarried tree via recursive spectral bounds, taking into account permutation invariance.
• The node embedding hLv is equal to a function applied to the local computation tree of depth L, rooted at v, that the authors obtain when unrolling the L neighborhood aggregations.

Determination

• Since both the non-linear activation role and the permutation-invariant aggregation function are Lipschitz-continuous, and the feature vector at the root and the shared weights have bounded norm, the embedding at the root of the tree adapts to the embeddings from the subtrees.
• Two parts were integral to the assay: (a) bounding complexity via local computation trees, and (b) the sum decomposition belongings of permutation-invariant functions.

Related work

• GNNs keep to generate much involvement from both theoretical and practical perspectives. An of import theoretical focus has been on understanding the expressivity of existing architectures, and thereby introducing richer (invariant) models that can generate more than nuanced embeddings. Only, much less is known about the generalization ability of GNNs. We briefly review some of these works.

  Expressivity. Scarselli et al (2009) extended the universal approximation holding of feed-forrard networks (FFNs) (Scarselli and Tsoi, 1998) to GNNs using the notion of unfolding equivalence. Recurrent neural operations for graphs have been introduced with their associated kernel spaces (Lei et al, 2017). Dai et al (2016) performed a sequence of mappings inspired by mean field and belief propagation procedures from graphical models, and Gilmer et al (2017) showed that common graph neural net models models may exist studied as Message Passing Neural Networks (MPNNs). It is known (Xu et al, 2019) that GNN variants such as GCNs (Kipf and Welling, 2017) and GraphSAGE (Hamilton et al, 2017) are no more than discriminative than the Weisfeiler-Lehman (WL) exam. In order to match the power of the WL test, Xu et al (2019) also proposed GINs. Showing GNNs are not powerful enough to correspond probabilistic logic inference, Zhang et al (2020) introduced ExpressGNN. Amongst other works, Barceloet al. (2020) proved results in the context of get-go guild logic, and Dehmamy et al (2019) investigated GCNs through the lens of graph moments underscoring the importance of depth compared to width in learning higher society moments. The inability of some graph kernels to distinguish graph properties such as planarity has also been established (Kriege et al, 2018, 2020).

Reference

• Z. Allen-Zhu and Y. Li. Tin sgd acquire recurrent neural networks with provable generalization? In Neural Data Processing Systems (NeurIPS), 2019.

  

• P. Barcelo, E. V. Kostylev, Thou. Monet, J Perez, J. Reutter, and J.-P. Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations (ICLR), 2020.

  

• P. L. Bartlett, D. J. Foster, and M. Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Data Processing Systems (NIPS), pages 6240–6249, 2017.

  

• P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, and 1000. Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Neural Information Processing Systems (NIPS), pages 4502–4510, 2016.

  

• Thou. Chen, Ten. Li, and T. Zhao. On generalization bounds of a family unit of recurrent neural networks. arXiv: 1910.12947, 2019a.

  

• Z. Chen, L. Li, and J. Bruna. Supervised customs detection with line graph neural networks. In International Conference on Learning Representations (ICLR), 2019b.

  

• H. Dai, B. Dai, and 50. Song. Discriminative embeddings of latent variable models for structured information. In International Briefing on Machine Learning (ICML), page 2702–2711, 2016.

  

• M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Neural Information Processing Systems (NIPS), pages 3844–3852, 2016.

  

• Due north. Dehmamy, A.-L. Barabasi, and R. Yu. Understanding the representation power of graph neural networks in learning graph topology. In Neural Information Processing Systems (NeurIPS), pages 15387–15397, 2019.

  

• D. 1000. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Neural Information Processing Systems (NIPS), pages 2224–2232, 2015.

  

• J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and 1000. Due east. Dahl. Neural bulletin passing for quantum chemistry. In International Conference on Car Learning (ICML), pages 1263–1272, 2017.

  

• N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Briefing On Learning Theory (Colt), pages 297–299, 2018.

  

• Thou. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In IEEE International Joint Conference on Neural Networks (IJCNN), pages 729–734, 2005.

  

• W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Neural Information Processing Systems (NIPS), pages 1024–1034, 2017.

  

• B. Hammer. Generalization ability of folding networks. IEEE Transactions on Knowledge and Information Engineering science (TKDE), 13:196–206, 2001.

  

• Fifty. Hella, Grand. Jarvisalo, A. Kuusisto, J. Laurinharju, T. Lempiainen, K. Luosto, J. Suomela, and J. Virtema. Weak models of distributed computing, with connections to modal logic. Distributed Computing, 28(1): 31–53, 2015.

  

• J. Ingraham, Five. G. Garg, R. Barzilay, and T. Jaakkola. Generative models for graph-based protein design. In Neural Information Processing Systems (NeurIPS), 2019.

  

• W. Jin, R. Barzilay, and T. S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning (ICML), volume fourscore, pages 2328–2337, 2018.

  

• W. Jin, K. Yang, R. Barzilay, and T. Jaakkola. Learning multimodal graph-to-graph translation for molecule optimization. In International Briefing on Learning Representations (ICLR), 2019.

  

• North. Keriven and Chiliad. Peyre. Universal invariant and equivariant graph neural networks. In Neural Information Processing Systems (NeurIPS), pages 7090–7099, 2019.

  

• T. North. Kipf and One thousand. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.

  

• J. Klicpera, J. Groß, and S. Gunnemann. Directional message passing for molecular graphs. In International Briefing on Learning Representations (ICLR), 2020.

  

• Northward. 1000. Kriege, C. Morris, A. Rey, and C. Sohler. A property testing framework for the theoretical expressivity of graph kernels. In International Joint Conference on Bogus Intelligence, IJCAI-xviii, pages 2348–2354, 2018.

  

• N. G. Kriege, F. D. Johansson, and C. Morris. A survey on graph kernels. Applied Network Science, five(1): six, 2020.

  

• T. Lei, Westward. Jin, R. Barzilay, and T. Jaakkola. Deriving neural architectures from sequence and graph kernels. In International Conference on Motorcar Learning (ICML), pages 2024–2033, 2017.

  

• A. Loukas. What graph neural networks cannot larn: depth vs width. International Briefing on Learning Representations (ICLR), 2020.

  

• H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman. Provably powerful graph networks. In Neural Data Processing Systems (NeurIPS), pages 2153–2164, 2019a.

  

• H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks. In International Briefing on Learning Representations (ICLR), 2019b.

  

• H. Maron, E. Fetaya, North. Segol, and Y. Lipman. On the universality of invariant networks. In International Conference on Machine Learning (ICML), 2019c.

  

• M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Auto Learning. The MIT Press, 2012. ISBN 026201825X, 9780262018258.

  

• C. Morris, 1000. Ritzert, M. Fey, West. 50. Hamilton, J. E. Lenssen, Grand. Rattan, and M. Grohe. Weisfeiler and leman get neural: College-order graph neural networks. In AAAI Conference on Artificial Intelligence (AAAI), pages 4602–4609, 2019.

  

• R. Fifty. Murphy, B. Srinivasan, V. A. Rao, and B. Ribeiro. Janossy pooling: Learning deep permutationinvariant functions for variable-size inputs. In International Conference on Learning Representations (ICLR), 2019.

  

• B. Neyshabur, S. Bhojanapalli, and N. Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations (ICLR), 2018.

  

• A. Sannai and Chiliad. Imaizumi. Improved generalization jump of permutation invariant deep neural networks. arXiv: 1910.06552, 2019.

  

• A. Santoro, F. Colina, D. Barrett, A. Morcos, and T. Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Car Learning (ICML), pages 4477–4486, 2018.

  

• R. Sato, K. Yamada, and H. Kashima. Approximation ratios of graph neural networks for combinatorial bug. In Neural Information Processing Systems (NeurIPS), 2019.

  

• F. Scarselli and A. C. Tsoi. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results. Neural Networks, 11(one):xv–37, 1998.

  

• F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and M. Monfardini. Computational capabilities of graph neural networks. IEEE Transactions on Neural Networks, xx(i):81–102, 2009.

  

• F. Scarselli, A. C. Tsoi, and 1000. Hagenbuchner. The Vapnik-Chervonenkis dimension of graph and recursive neural networks. Neural Networks, 108:248–259, 2018.

  

• J. Sokolic, R. Giryes, G. Sapiro, and Grand. Rodrigues. Generalization Error of Invariant Classifiers. In International Briefing on Artificial Intelligence and Statistics (AISTATS), pages 1094–1103, 2017.

  

• P. Velickovic, M. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. In International Briefing on Learning Representations (ICLR), 2018.

  

• S. Verma and Z.-L. Zhang. Stability and generalization of graph convolutional neural networks. In International Conference on Knowledge Discovery & Data Mining (KDD), page 1539–1548, 2019.

  

• G. Xu, C. Li, Y. Tian, T. Sonobe, K. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning (ICML), pages 5453–5462, 2018.

  

• Thousand. Xu, Due west. Hu, J. Leskovec, and South. Jegelka. How powerful are graph neural networks? International Conference on Learning Representations (ICLR), 2019.

  

• R. Ying, J. You lot, C. Morris, Ten. Ren, Westward. 50. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. In Neural Information Processing Systems (NeurIPS), 2018.

  

• J. Yous, R. Ying, and J. Leskovec. Position-aware graph neural networks. In International Briefing on Machine Learning (ICML), pages 7134–7143, 2019.

  

• South. Yun, M. Jeong, R. Kim, J. Kang, and H. Kim. Graph transformer networks. In Neural Data Processing Systems (NeurIPS), pages 11960–11970, 2019.

  

• J. Zhang, Q. Lei, and I. S. Dhillon. Stabilizing gradients for deep neural networks via efficient SVD parameterization. In International Conference on Machine Learning (ICML), pages 5801–5809, 2018a.

  

• M. Zhang, Z. Cui, 1000. Neumann, and Y. Chen. An stop-to-end deep learning architecture for graph classification. In AAAI Briefing on Bogus Intelligence (AAAI), pages 4438–4445, 2018b.

  

• Y. Zhang, X. Chen, Y. Yang, A. Ramamurthy, B. Li, Y. Qi, and L. Song. Efficient probabilistic logic reasoning with graph neural networks. In International Conference on Learning Representations (ICLR), 2020.

  

数据免责声明

页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果，我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问，可以通过电子邮件方式联系我们：report@aminer.cn

lollistakinquanded1938.blogspot.com

Source: https://www.aminer.org/pub/5e4a64313a55acda1392dda3/generalization-and-representational-limits-of-graph-neural-networks

Share this post