<form id="1th7n"></form>
      <address id="1th7n"></address>

      <address id="1th7n"></address>

      學術報告

      學術報告

      您當前所在位置: 首頁 > 學術報告 > 正文
      報告時間 2023年6月13日(周二)下午3:00-5:20 報告地點 南校區網安大樓會議中心113會議室
      報告人 葉海山

      報告題目:Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums

      IMG_9998

      報告人:葉海山副教授 西安交通大學

      邀請人:趙志華

      報告時間:2023年6月13日(周二)下午3:00-5:20

      報告地點:南校區網安大樓會議中心113會議室

      報告人簡介:葉海山,西安交通大學管理學院副教授,博士生導師,入選陜西省高層次人才引進計劃青年項目。主要從事數學優化理論、算法和應用研究。在Mathematical Programming, Journal of Machine Learning Research, IEEE Transactions on Neural Networks and Learning Systems等國際期刊和會議上發表幾十余篇學術論文。

      報告摘要:Learning rate schedulers have been widely adopted in training deep neural networks. Despite their practical importance, there is a discrepancy between its practice and its theoretical analysis. For instance, it is not known what schedules of SGD achieve best convergence, even for simple problems such as optimizing quadratic objectives. In this paper, we propose Eigencurve, the first family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed. The condition is quite common in practice. Experimental results show that Eigencurve can significantly outperform step decay in image classification tasks on CIFAR-10, especially when the number of epochs is small. Moreover, the theory inspires two simple learning rate schedulers for practical applications that can approximate eigencurve.For some problems, the optimal shape of the proposed schedulers resembles that of cosine decay, which sheds light to the success of cosine decay for such situations. For other situations, the proposed schedulers are superior to cosine decay.

      上一篇:Zero-sum stochastic games under a probability criterion

      下一篇:Measurement and Analysis of Digital Economy Promoting Carbon Reduction Based on Panel Data of 278 Cities in China

      關閉

        <form id="1th7n"></form>
          <address id="1th7n"></address>

          <address id="1th7n"></address>

          h色网站