I learned Spanish and English at practically the same time (Spanish slightly before, but before pre-k I somehow spoke English as well as all the other kids from watching TV). Looking at both languages objectively, I'd say English is considerably more difficult than Spanish. Spanish has fewer sounds, and the alphabet can very accurately represent what the words sound like: it's 100% phonetic. English writing looks rather silly by comparison. English verbs also seem to make less sense than Spanish verbs: irregulars don't seem to be anymore prevalent in Spanish than in English. I'd say English has more irregulars.
The most "difficult language to learn" is all relative to what language you're learning from, anyway. A Chinese person would obviously find another sino language easier than English, whereas a German would find it considerably more difficult. But I would also like to add, for what it's worth, that Spanish-speaking 3 year olds are considerably more eloquent than English-speaking 3 year olds. I'd say that's the best way to objectively measure how all languages compare, regardless of base language.