We present GoGP, a library for probabilistic programming around Gaussian processes. Kernels, beliefs about hyperparameters and about observation inputs and outputs are programmatically expressed using the same differentiable probabilistic programming framework. On one hand, a basic usage with maximum-likelihood hyperparameter estimates and homoscedastic Gaussian noise requires as little coding as with any Gaussian process library. On the other hand, imposing prior beliefs on hyperparameters, handling input uncertainty, heteroscedastic and non-Gaussian noise, or change point detection can be organically added on top of a Gaussian process kernel. Just like distributions in probabilistic programs, kernels can be flexibly combined using general code flow rather than through predefined combination operators (usually addition and multiplication). Implemented in Go, a general-purpose programming language popular for server-side programming, our library is both well suited for integration into data processing/machine learning pipelines and exploits language features such as efficient light-weight parallelism for computation efficiency. We demonstrate different uses of the library on a variety of case studies inspired by applications.