Balancing exploitation (doing what has worked in the past) with exploration (choosing a novel course of action for information gain) is central for adaptive behavior in dynamic environments. First, I will show that human learners typically employ a directed exploration strategy when solving restless multi-armed bandit problems, that is, they strategically explore options that promise high information gain. I will show that exploration is at least in part under catecholaminergic control, and that it is affected in disorders linked to the dopamine system. Second, I will show that incorporating similar exploration mechanisms into existing models for the popular two-step reinforcement learning task improves model fit across different task variants. Finally, I will outline our recent work on exploration behavior in recurrent neural network models. These models have recently gained attention in cognitive and systems neuroscience, and might recapitulate some of the mechanisms observed in empirical work. Specifically, I will show that these networks can be trained to solve restless bandit problems with human level accuracy. A closer inspection of the computational mechanisms employed by these networks as well as the representations that emerge in their hidden unit activity dynamics then reveals both similarities and differences compared to human and animal neuroscience work.