I did?
Firstly, if you want the most bang-for-your-buck, get all three games (Spyro's Adventure, Giants, and Swap Force) on the console of your choice. Then, if you have a 3DS, get the 3DS versions of all three. The 3DS version is a completely different game (and, in my opinion, more enjoyable than the Console versions). For the first two titles, you can get the games used (game-only) for under $10. Sometimes, you can find them new that cheap.
All Spyro's Adventure (SSA) figures work in Giants.
All SSA and Giants figures work in SWAP Force.
Some SWAP Force figures work in SSA/Giants and some Giants figures work in SSA.
The basic thing is - if the character existed in some form when the game was released, the figure will work in the game.
As has been said, to access everything in SSA, you need one of every element - 8 figures total.
To access everything in Giants, you need each of those 8 elements and 1 Giant character.
To access everything in SWAP Force, you need the 8 elements, the 1 Giant, and the 8 movement types of SWAPpable Characters. Your SWAPpable characters have elemental types as well, so you can overlap those with your elements, but you can't use the SWAP characters in the older games anyway - so if you have all 3 (6, if you count the 3DS) games, you'll need a total of 17 characters in all.
Now, aside from that, there are four Adventure Packs that were released for SSA. These packs include a figure that unlocks a new level in both SSA and Giants. The level unlocked is virtually the same in both games (with a few things tweaked in Giants to accommodate the new characters and such). There are two Adventure Pack items released with SWAP Force that also unlock two new levels, only in SWAP Force. The nice thing about the Adventure Pack items is that if you only want to unlock the level, you just have to put it on the portal once and it's unlocked - this means you can usually find the location piece (the part of the adventure pack that unlocks the level) second hand pretty cheap.
All that aside - you're not missing a whole lot by not having one of each element. Really. The side paths are nice, but the main reason to get the other figures is to play with their various movesets and battle mode. And, of course, to have a crazy collection of 200+ figures.
So, basically, don't feel like you need to go out and buy everything RIGHT NOW BECAUSE IT'S THERE just to play the game. Unless you plan on getting all 200+ figures, just take your time, get new figures as they go on sale or you find them second-hand cheap.