BNL-RHIC. We present direct photon-hadron correlations in 200 GeV/A Au+Au, d+Au and p+p collisions, for direct photon $p_T$ from 5-12 GeV/$c$, collected by the PHENIX Collaboration in the years from 2006 to 2011. We observe no significant modification of jet fragmentation in d+Au collisions, indicating that cold nuclear matter effects are small or absent. Hadrons carrying a large fraction of the quark's momentum are suppressed in Au+Au compared to p+p and d+Au. As the momentum fraction decreases, the yield of hadrons in Au+Au increases to an excess over the yield in p+p collisions. The excess is at large angles and at low hadron $p_T$ and is most pronounced for hadrons associated with lower momentum direct photons. Comparison to theoretical calculations suggests that the hadron excess arises from medium response to energy deposited by jets.