Southward interplanetary magnetic field (IMF) in the geocentric solar magnetospheric (GSM) reference frame is the key element that controls the level of space weather disturbance in Earth's magnetosphere, ionosphere, and thermosphere. We discuss the relation of this geoeffective IMF component to the IMF in the geocentric solar ecliptic (GSE) frame, and using the almost continuous interplanetary data for 1996–2015 (inclusive), we show that large geomagnetic storms are always associated with strong southward, out‐of‐ecliptic field in the GSE frame: Dipole tilt effects, which cause the difference between the southward field in the GSM and GSE frames, generally make only a minor contribution to these strongest storms. The time‐of‐day/time‐of‐year response patterns of geomagnetic indices and the optimum solar wind coupling function are both influenced by the timescale of the index response. We also study the occurrence spectrum of large out‐of‐ecliptic field and show that for 1 h averages it is, surprisingly, almost identical in ICMEs (interplanetary coronal mass ejections), around CIRs/SIRs (corotating and stream interaction regions) and in the “quiet” solar wind (which is shown to be consistent with the effect of weak SIRs). However, differences emerge when the timescale over which the field remains southward is considered: for longer averaging timescales the spectrum is broader inside ICMEs, showing that these events generate longer intervals of strongly southward average IMF and consequently stronger geomagnetic storms. The behavior of out‐of‐ecliptic field with timescale is shown to be very similar to that of deviations from the predicted Parker spiral orientation, suggesting the two share common origins.