Accession Number:



Thermodynamics of Learning

Descriptive Note:

Technical Report,13 Feb 2017,01 Sep 2018

Corporate Author:

Columbia University New York United States

Personal Author(s):

Report Date:


Pagination or Media Count:



The recent exponential increase in the applications of machine learning is based on algorithms that were already well known in the second half of the 20th century. These recent successes became possible due to the increased availability of computing resources, which allowed for a new level of complexity in the algorithms, as well as the increased availability of large datasets, which allowed these algorithms to be fit in very high dimensional parameter spaces without overfitting. While these methods have been very successful, two fundamental challenges remain. The first challenge lies in evaluating how well an algorithm works a priori, and in providing bounds on the predictions emanating from the algorithm. We aim to present research directions that may address these ideas at the algorithmic level Task 1, then show how information theory can help address this constraint at the abstract learning level, independently of the algorithm Task 2. The second challenge is to overcome the energetic constraints that are currently the principal limits on the size of the computational tasks required by the training of these algorithms. We will outline how information thermodynamics may help the emerging approximate computation paradigms produce energy efficient frameworks for learning Task 3.

Subject Categories:

  • Numerical Mathematics
  • Cybernetics

Distribution Statement: