Charging the phone itself uses very small amounts of energy. The real energy impact lies in the data centers and networks that process data for your smart phone. According to a recent study your smart phone requires more energy to run than your home's refrigerator.
Currently, it is estimated that around two percent of the world's energy supply goes to the Internet and it contributes almost two percent of total carbon emissions. These numbers are only expected to rise as Internet and smart phone usage expands.