What do you mean by vertical elements? Marathon made use of height, and System Shock had sloped surfaces and 3D models you could walk on. Dark Forces had deep pits but that's really the Star Wars aesthetic rather than anything specific they were trying to achieve.
Well, heres the thing:
LucasArts only ever made Star Wars themed clones of whatever happened to be popular at the time. I agree, this was fine as long as the clones were good, but it's been a very long time since they have been.
Basically the problem is that Lucas couldn't retain talent, whether internally or through their third party business relationships. We'll probably never know exactly why (non-disparagement contracts, woo) but I've heard stories that personally blame George Lucas for being a big indecisive baby, ordering rolling changes to products in late development. Not just stuff like the characters, which apparently changed a lot, but questions like "what popular game will we clone?" changed practically monthly. But, you know, just rumors. Regardless of why, we know the entire JK team quit, and the one person I know who worked on TFU lasted a couple of months.
Most of the games you've listed were made by third parties:
KotOR series - Bioware/Obsidian
Rogue Squadron series - Factor 5
X-Wing series - Peregrine Games (later Totally Games)
Jedi Knight series - last two by Raven
And all of these business relationships fell apart while **** was still profitable. We'll never know exactly why, but Bioware, at least, felt they were sacrificing more for the Star Wars IP than it was worth. (TOR wasn't made by Bioware, it was made by a different EA subsidiary they re-branded 'Bioware').