For each adjoint variety not of type A or C, we study the irreducible component of the Hilbert scheme which parametrizes all smooth conics. We prove that its normalization is a spherical variety by using contact geometry, and then compute the colored fan of the normalization. As a corollary, we describe the conjugacy classes of conics in the adjoint variety and show smoothness of the normalization. Similar results on the Chow scheme of the adjoint variety are also presented.