Probabilistic programming is the idea of developing a programming language for writing and reasoning about probabilistic models from machine learning and statistics. Such a language comes with the implementation of several generic inference algorithms that answer various queries about the models written in the language, such as posterior inference and marginalisation. By providing these algorithms, a probabilistic programming language enables scientists to focus on designing good models based on their domain knowledge, instead of building effective inference engines for their models, a task that typically requires expertise in machine learning, statistics, and systems. Even experts in machine learning and statistics may get benefited from such a probabilistic programming system because using the system they can easily explore highly advanced models.

In the past five years, with colleagues from programming languages, machine learning, and probability theory, I have worked on developing the semantic foundations, efficient inference algorithms, and static program analysis for probabilistic programming languages, especially those that support expressive language features such as higher-order functions, continuous distributions and general recursion. In this talk, I will describe a few lessons that I have learnt so far.