Solitons and cavitons (the latters are localized solutions with singularities) for the nonlocal Whitham equations are studied. The fourth order differential equation for traveling waves with a parameter in front of the fourth derivative is reduced to a reversible Hamiltonian system defined on a two-sheeted four-dimensional space. Solutions of the system which stay on one sheet represent smooth solutions of the equation but those which perform transitions through the branching plane represent solutions with jumps. They correspond to solutions with singularities of the fourth order differential equation -- breaks of the first and third derivatives but continuous even derivatives. The Hamiltonian system can have two types of equilibria on different sheets, they can be saddle-centers or saddle-foci. Using analytic and numerical methods we found many types of homoclinic orbits to these equilibria both with a monotone asymptotics and oscillating ones. They correspond to solitons and cavitons of the initial equation. When we deal with homoclinic orbits to a saddle-center, the values of the second parameter (physical wave speed) are discrete but for the case of a saddle-focus they are continuous. The presence of multiplicity of such solutions displays the very complicated dynamics of the system.