The town is small and can be visited in one hour. As you already guessed, due to the specific color, you need to come only in the morning or in the evening, at other times coloring will be lost. The color of the houses is due to ochre (iron oxide with clay), which lies in this area in large quantities and was extracted industrially for centuries.
Right next to the car park, begins the Ocher Trail (Sentier des Ocres). This is an easy route through the former quarry.
Getting there: Parking coordinates N 43 54.075 E 5 17.672. The nearest railway station is Cavaillon, 27 km from Rousillon. Then take bus 17 of TransVaucluse company (timetable and map).