What makes a Zelda game a Zelda game? The fusion of both the adventure and puzzle genres, with a dash of fighting.
The first, most important thing in a Zelda game is the overworld. It must be grand, it must be interesting, and it must be a little confusing at first. You're meant to feel insignificant, lost, and yet hope to see everything and find all the secrets. You should probably, but not necessarily, be able to see a map of where you've been, but it shouldn't show fine details.
The dungeons should contain the puzzles, clearly. Whether it's how you clear a room of enemies easily with the terrain given, how you open a door or find a key, or how you use your new item, each room should contain a puzzle. If it doesn't it's useless and a waste of time. A boss, itself, can be a puzzle, requiring you to learn what tool to use at what time, based on the boss's movement, the stage's design, and what has been acquired in the dungeon preceding the boss.
To add to the exploration, dungeons in Zelda games should rarely have a necessary order, though there should definitely be an intended order. If any actual order is given by plot, the dungeon should be the first or last dungeon, or a batch of dungeons should be opened up. It's also typical for one dungeon to be available after specific tools have been gained, but the entrance into one dungeon should rarely require that a previous dungeon be beaten, unless dungeons are opened in batches.
Four Swords games are not traditional Zelda games, and do not follow traditional "rules" that other Zelda games meet.
Now, when you look at Zelda games, it becomes clear that, IMO, a few games miss the "Zelda" mark, yet are in the series. For example, in the first game, much of the basic enemy combat is meant only to slow down Link's progression, making the overworld seem a little larger than it is. Link's Adventure branched far off the typical gameplay, and many hate it for that. Personally, of all the Zelda games I've either played or witnessed much of, I've found that the The Legend of Zelda and A Link to the Past are the two games that best embody Zelda.
Yes, I know that many people loved OoT, and it and Majora's Mask are great Zelda titles, but OoT just seems out of place with earlier titles. For example, as much as Epona has become part of Zelda lore now, rides weren't really present in other Zelda titles. Ok, sure, there were Pegasus Boots in LttP, you got me. So why, if you have Epona, would you need the ability to teleport everywhere? Epona and half of the Ocarina's functions contradict each other, IMO. Now, I know Epona is used to jump fences, yes, I realize that, However, aside from the initial "Wow!" factor of riding Epona, wouldn't the game have been more fun with it's own item that can get you to the fenced areas that also could have some kind of special effect on enemies, as well? Then, there's the whole torch thing. What happened to the fire rod? The ice rod? You never gain access to an "anywhere" fire weapon, nor do you gain a tool to put out fires, either. The combat in several areas can impede your progress, but never feels like a puzzle, but rather, a chore. And of course, Navi really doesn't help the game.
I can't comment so much on any of the other games, since OoT, I've felt a very disinterested in the series. From all the impressions, pictures, videos, and everything I've seen from all the games, the dungeons and overworlds feel unimaginative, gray, and much less like an adventure than they should. OoT's puzzles were a little too simple and bland, so I've feared the same in later games, and I don't want to invest myself in a game that I'll feel is a disappointment. Maybe I made my mind up too soon, but I feel like I was the only one who didn't feel like OoT had a "Wow!" to any of it, and perhaps it's because any time I touch it, I compare it to LttP with disregard to the 3D-ness that others were shocked at in '98.
Honestly, I just want to be able to explore anywhere I want, see things I can't yet reach, and have an environment that's full of questions and mysteries. I don't feel like many games in the Zelda series give that feeling, but I'm pretty sure they're supposed to. Oh, and I want to be puzzled in dungeons. It doesn't matter how the game looks as long as it meets that criteria. If it does, it's a Zelda game. If it doesn't, it's an imitation or something else. If it's okay being something else, I'm okay with it, like the Fourswords titles, which are co-op action/puzzle hybrids, and intentionally leave out the exploration. However, I feel like a lot of the newer Zelda titles are missing the grand type of exploration, missing out on why we liked the combat, and are missing the intuitive puzzles that the earlier games did.
I would say that there's a limit to what you can do with simple block puzzles, but the Adventures of Lolo and its sequels prove that idea wrong, so I honestly think that Miyamoto & anyone that wants to be involved in Zelda need to establish the series true identity before they make another one, because with each iteration, the games seem to be losing what makes them Zelda.
Also, alternative worlds, dimensions, and gameplay, like Wolf Link, and things like that have precedence. LttP had the Dark World and Bunny Link, OoT had Young Link's time and Old Link's time. Really, I've felt like the purpose of the two worlds is to add an element of exploration. Like solving a 3D jigsaw puzzle from two separate angles or something. That's how I've felt, at least.
Edit: As far as taking Zelda and putting the next game in a new environment, I'm actually all up for it. I think it would inspire the developers to consider the base gameplay we like about Zelda, and re-establish that much stronger than any 3D Zelda has, since fans would probably need the gameplay more than ever to accept the game. I would be excited to see something new that can retain the sense of adventure that older titles held. And no, sailing doesn't have that same sense of Adventure to me.