Jedi Knight Quadranscentennial Edition

[Quib Mask]

i.e. Jedi Knight 25th Anniversary Edition

(starting to get this post created. I'll flesh this out as I'm getting files organized; there's a lot that needs to be documented)

The main JK executable, based on the official 1.01 patch, includes the now-standard assortment of JKUP patches, plus the following features:

• -
• (list to be further populated)
• -
• rotated particles by 45 degrees
• enhanced z-buffer accuracy
• CPU relief (JK should no longer peg a core to 100%)
• toggle enhanced 3D audio without JK restart
• fixed a (big) memory leak in A3D audio code
• fixed a jkSetForceSpeed crash
• added ZeqMacaw's increased synced thing limit
• enabled unused COG extension features:
  
  • hotkey message
  • discrete hotkey filtering
  
  
• added some OpenJKDF2 QOL_IMPROVEMENTS:
  
  • smooth singleplayer aerial physics
  • improved behavior above 100 fps
  • waggle animation scaled to framerate
  • keyboard turning scaled to framerate
  • mouse sensitivity numeric display
  
  
• fixed mouse sensitivity setting drift
• increased mouse sensitivity maximum
• added an adjustable internal framerate (frametime arg and cmd)
• added JkGfxMod detection and compatibility enhancements
• reset debug mode flags at main menu
• tweaked help command to not list itself
• devmode arg skips intro cutscene (can view in cutscene menu)
• added some (all?) OpenJKDF2 memory leak fixes
• added OpenJKDF2 AI awareness fix
• added dual wield weapon mesh rendering
• removed JKUP's increased COG symbol limit:
  
  • it was not actually usable since variables 256 and above disabled COG verbs and rendered a COG functionally useless
  
  
• added mouse wheel support
• made the free look hotkey invert state (since "enable free look" is typical)
• fixed missing F1-F8 hotkeys when player data load fails
• adjustments to default controls (e.g. space = jump, auto run enabled, etc.)
• fixed crash when alt-tabbing during joystick "identify" menu
• added limited controller trigger support (both triggers can't simultaneously cause an action due to limitations in winmm/dinput controller support, but actions that don't really need to occur simultaneously such as "Use Force Power" and "Fire 1" work adequately)
• added left and right stick click-in support (aka L3/R3 or LSB/RSB); these have no defaults bound as I hate them, but they're now available for use
• added rudimentary controller support to the game menus (left stick = move cursor, A = click, B = escape)
• enabled using mouse buttons and joystick (controller) buttons while in-game chat is active

Additionally, external files provide the following features:

• enhanced 3D audio (a3d.dll)
• traditional rendering pipeline (ddraw.dll, wndmode.dll)
• install-free DirectPlay support (dplaysvr.dll, dplayx.dll, dpnhpast.dll, dpnhupnp.dll, dpwsockx.dll)
• CD audio (winmm.dll, libvorbisfile-3.dll, 00_soundtrack.cog, 00_playambient2.cog)
• automatic FOV, mipmap and LOD scaling (item_wrenchyellow.cog)

Where modified COGs are used to provide features, they're designed to match the unmodified vanilla COG in behavior and checksum so as to not cause multiplayer incompatibility.

QM

Edit: I've written down what I've done in the past month, but all prior features are yet to be documented.

[Jon`C]

Feel free to bundle the executable with JkGfxMod if you want. It uses Microsoft Detours to hook the executable. Microsoft provides a tool to patch an executable offline, which in your case would let you skip the extra inject.exe stuff.

[Quib Mask]

I actually want to keep JkGfxMod dynamically patched, both so that if it gets further updates a user can simply drop in the new file(s), and because other stuff I've either done personally or adapted from others' (ZeqMacaw et al.) work has the vanilla rendering pipeline functional (including with 32-bit color modes and glow-in-the-dark colors like the 8-bit software renderer) on modern Windows (and still backwards compatible through at least Win98SE; I never tested on 95/98, nor on 11 yet, but I'm optimistic), so having both options seems better than forcing one. The injector, in spite of the anti-virus false positives, also seems to prove handy for launching via Steam.

The glowing Lightsabers and projectiles possible with JkGfxMod are so damn amazing though, forcing that as a standard sure is tempting. I'm experimenting with the displacement map capability as I'm writing this, but not sure if anything will come of it.

For reference, for JkGfxMod compatibility, I did the following:
- detect if JkGfxMod is loaded (via GetModuleHandleA on renderer.dll, just after JK calls CreateWindowExA); I don't like how hacky this is, but it's simple and reliable
- if detected:
-- skip QueryInterface for Viewport3 (JkGfxMod kills itself otherwise, I wish it just returned E_NOINTERFACE)
-- restore the ZBufferMode for rendering layers 3 (transparent level geometry) and 4 (pov weapon model) which are otherwise modified from vanilla
-- modify some window centering code I added to instead snap the client area to 0, 0
-- modify setting load code for
--- 3DDeviceGUID to return JkGfxMod's GUID
--- displayMode to always return 1
--- b3DAccel to always return true (1)
-- modify setting save code for the above three, to skip saving them

The first two are critical for compatibility.

QM

Edit: Playing around with displacement maps. This is the vertical water shaft in Canyon Oasis:


And here's what the mouse config screen looks like now:

[Quib Mask]

I've been kinda going nuts with JkGfxMod. There wasn't a sane way to resolve a compatibility issue between my new code to let the controller drive the cursor in menus and JkGfxMod, so I went ahead and got a build environment functional for JkGfxMod (not difficult, thank you for making a tidy, self-contained project and sharing it with the world) and started making additions.

So far, I've done the following:
- hook SetCursorPos to translate stretched window coordinates
- tweaked the GetCursorPos hook so that it also affects in-game (for 640x480 software mode mouse use)
- hook CreateFileA to load texture replacements during level load to reduce in-game hitching without random pre-caching (I hated writing this hook in C++, I could've made it hyper efficient in assembly)
- level-load-loading the textures necessitated populating a filename-based dictionary during program load to be used during level load (from ["replaces"][0]["name"] entries, accomodating multi-cel textures)
- added a boolean to jkgm.json to control level-load-loading ("per_level_texture_loader")
- avoid crashes for missing albedo_map, emissive_map and displacement_map files
- automatically use current screen resolution if "resolution" isn't defined in jkgm.json (instead of 640x480)
- added "hud_texture_num_slabs" to jkgm.json to specify number of frames per HUD layer refresh (on my computers, at my preferred screen resolutions, updating every frame doesn't clobber framerate, ymmv)
- added additional non-displaced texture passthrough from displacement mapping to screen space shaders
- rewrote SSAO (including aspects in the DLL)
- modified displacement mapping
- made "enable_vsync": true be the default (there were non-zero complaints about videocard heat)

QM

[Brian]

Wow I don't know how I missed that screenshot but it looks amazing!

[Quib Mask]

More JkGfxMod tweaks:
- altered framebuffer clears to restore vanilla JK HOMing; this is more necessary than is obvious to avoid rare-but-consistent partial screen blackouts as the camera passes sector adjoins
- added relaxed cone mapping as an alternative to standard displacement mapping

Relaxed Cone Mapping and fine tuning the new "Fast" Ambient Occlusion have been what I've been working on the past month. Not sure yet how to (or if it's worth) getting screenshots that show off the differences.

[SMLiberator]

super happy to see this is still going on, plus with jkgm integration
any hints as to when we can see a demo?

- altered framebuffer clears to restore vanilla JK HOMing; this is more necessary than is obvious to avoid rare-but-consistent partial screen blackouts as the camera passes sector adjoins

this is actually super noticeable in levels with 3do skyboxes. also there are some levels and mods that rely on HOM to create certain visual effects, so it's good that consistency with base JK is maintained in this aspect.

[Brian]

I need this!

[SMLiberator]

also, can anything be done to fix this silly wee seaming issue caused by JkGfxMod mipmapping?

[Quib Mask]

I've been working with someone on an all-in-one tool for precedurally generating depth maps (and their matching cone maps). It's technically going to be generically useful and not JkGfxMod-specific. As a baseline, it can be fed a basic (albedo) texture, and will then:
- produce an approximate normal map
- use the normal map to produce a depth map
- apply filters to the depth map (normalizing, changing distribution, blurring, etc.)
- use the (filtered) depth map to produce a cone map
- reconstruct a normal map based on the filtered depth map (not used by JKGM)
It can be fed one of those intermediate data textures, for example, a properly created normal map, or a hand-made depth map, and just start from there to produce the output further down the chain.

While the normal map creation is low cost (computationally) and basically instant (and also really simplistic), calculating a depth map from a normal map is very CPU intensive and can take multiple minutes per texture (once 512x512 or larger), and then generating a cone map from a depth map is very GPU intensive and can also take multiple minutes.

Once this tool is done, and I've processed existing JKGM assets into cone maps, I think I'll be ready to release. All of this already works, but wasn't in a state that could be shared, so it's just been a matter of getting it to that point. No point releasing modified JKGM + shaders if no one can produce the necessary data to feed the shaders.

---

Regarding the mipmapped texture seams, I've definitely witnessed that as well on some of the JK Enhanced models. I'll at least be taking a look to see if there's a practical solution. I assume it's due to the neutral color painted in the "unused" gaps of the UV layout getting blended in during the mipmap generation and affecting visible pixels, so that's where I'll be starting my investigation.

QM

[Jon`C]

Since you're going to so much effort, it might be worthwhile adding your generated normal maps to the material pack anyway. JKGM could ignore them for now, but normal maps would be useful for adding screen space reflections in the future (or for modernizing the lighting in OpenJKDF2).

Also, if by chance your approach could be applied to it, the same goes for a roughness-metallic texture. It is not useful today, but in any situation where you would be able to use the normal maps you would want to have roughness maps. Mipmapping is destructive to fine detail in normal and displacement maps, and roughness maps lets you treat that lost detail as microfacet scattering (which it technically is). There's a siggraph paper from either valve or square-enix that discusses their approach to computing [an]isotropic roughness from normal mip levels. It's pretty straightforward iirc.

[SMLiberator]

also envmaps when

[Quib Mask]

Merry Christmas! Barely made it before the new year when it would be too late to call it a 25th Anniversary version...

https://drive.google.com/drive/folde...GC?usp=sharing
JK_2002.ZIP password is: JK_2022

The password is to keep gdrive from mangling or vaporizing DLL and EXE files now or in the future.

---

This is just JK 2022, not the modified JkGfxMod stuff yet.

Extract the _setup folder into your Jedi Knight folder...

...then go into the _setup folder and run _setup.cmd.

For now, do this in a copy of your JK folder. The setup script tries to take a backup of important files, and running the setup script again will offer to uninstall back to how it was, but I only did enough testing to make sure it worked enough to release.

---

The DirectDraw proxy has special accommodations for the Steam version so that things will work (and even automatically launch JkGfxMod's inject.exe if present) without renaming files, etc. In other words, you can do a fresh Steam install, copy in and run JK 2022's setup script, then launch via Steam and it should just work.

The setup script also undoes some damage Steam's first-run setup does (namely it uninstalls the app compat database for JK.exe that Steam installs).

I've only tested the setup script with the original 1997 JK and the Steam version, but it gets the main executable out of the 1.01 patch installer, so it will probably work under most circumstances.

While you can delete the _setup folder after patching, if you leave it, you can re-run the setup script to verify the patch is still applied, and uninstall the patch.

QM

[SMLiberator]

nice
I couldn't figure a way to apply the patch via Linux, so I resorted to an old windows install to do that, but it was super easy then.

I'm mostly familiar with this patch as I'm (supposedly) the one person to have tested its early 2018 build back then on JKHub (and I also applied some of the OpenJKDF2 stuff via manual hex editing hehe), but neat to see this brings some new stuff.
The first, most obvious and important one is JkGfxMod compatibility - not only this works nicely (unlike the 2018 build which simply crashed when trying to run with JKGM) but it also seems to have solved the super annoying keyboard input lag that happened when framerates dropped when using JKGM coupled with JK13. Speaking of which, JKGM framerates do seem a tad bit better (haven't really compared, but it's better than I remember) on my 7yo PC with integrated Intel graphics.

Hopefully the memory leak fixes will solve the seemingly random crashes that happen in JKGR after some time spent on most maps. I do wonder what "OpenJKDF2 AI awareness fix" is about though, haven't tested thoroughly enough to see a difference. The only negative side effect I've noticed so far is that controlling JKGR cars via arrow keys seems to have been messed up somehow, I believe due to keyboard turning being scaled to framerate. Hopefully I can fix that via some simple cog tweaking.*** [Edit: jesus christ the possibilities with the new hotkey message are scary]

Glad this is out, and hope to see the JKGM tweaks coming out sooner rather than never

[Edit 2: sadly I don't see any way that can be fixed I don't see a way to tell apart keyboard turning input from mouse turning input, so I'll have to make it be either one or the other. Oh well. It's still worth it for everything else.]

[Edit 3: the keyboard (and only keyboard, doesn't affect mouse) input lag wasn't caused by JKGM, but rather by your JK2018 patch. sadly, it is present on this version.]

[Quib Mask]

Okay, probably have time to pick this up again, so hopefully you'll see this SMLiberator.

As a first test to try and resolve the keyboard input delay, would you try hex editing the patched JK 2022 executable as follows:
- at offset 0x10E016 you should see
6A 01 00 00
- change to
E6 4D EF FF
This should remove a Sleep(1) from the main render loop that might be the problem.

Obviously keep a backup of the executable since this may not help and I'll want to try other things. My current hypothesis is it's a Linux-specific issue either with the Sleep added to the rendering loop to relax CPU load or with how Linux is handling keyboard event messages. Windows 10 (or maybe 8, not sure when window key event messages got so much more computationally expensive) also had some issues in that regard compared to 7, and especially compared to XP and older.

QM

[SMLiberator]

My bad, I just noticed I made a mistake on my previous message - the input lag wasn't being caused by the 2018 patch, it was fixed with the 2018 patch, and was present in JK13 and in the 2022 patch. Will give the hex edit a try ASAP.

[edit] nah, I don't think that fixed it

[Quib Mask]

Well, if you're positive JK13 and 2022 have the input delay and 2018 absolutely doesn't, then I'm not surprised; that hypothesis was based on 2018 and 2022 both having input delay. I'll look through the differences between those three with that in mind, but having you double check that 2018 does not would be appreciated (so I'm not chasing ghosts).

QM

[SMLiberator]

JK22 definitely has it, JK18 definitely does not. For some reason JK13 isn't launching here but I'm 100% positive it also does. Note that it only happens (or is only noticeable?) on low framerates and it can't be replicated by setting frametime to a high value.

[Quib Mask]

Alright, while I'm refamiliarizing myself with JK's guts and carefully analyzing changes I'd made between the 2018 and 2022 versions, here's a bulk archive of all ~50 snapshots between the two in case you're up for some brute force work:
JK_2023_troubleshooting.7z
https://www.sendspace.com/file/mv4bll
Password: SMLiberator
Be careful that you get the correct download link, it'll be ~397 KB and contain over 50 "bak" files. Each is just a prepatched JK executable where you rename it to use it. No patching, etc.

My apologies for the junk file host, but I don't really want this file to persist or be easy to access as it's a confusing mess and I normally never distribute patched executables.

Inside the archive, rev 129 is the 2018 release (with advapi32.dll redirected to portable.dll as in JK2022) so start by making sure it in place of JK2022 works as expected. Rev 180 is the version just before the JK2022 release. All testing should be done in a JK2022 setup.

I'd suggest doing a binary search and see if you can pinpoint which version introduces the input delay. So something like:
Start in the middle and...

- try 155
-- if delay try 145
--- if delay try 140
--- if no delay try 150

- try 155
-- if no delay try 165
--- if delay try 160
--- if no delay try 170

...and so on.

Just if you're up for it. I'm picking away at things on my end, trying to guess which changes might have made sense to affect keyboard input. It could be COG hotkey stuff, or high framerate stuff, various things involving controller support... I'm really not sure yet.

If rev 129 in an otherwise JK2022 setup still has input delay, that would imply that it's not changes to the JK executable itself giving you trouble, but one of the supporting files. So it makes sense to check that first.

And a reminder, keep a backup of your patched JK2022 executable.

QM

[SMLiberator]

Apparently 150 (win10fix) is the last version with no keyboard input lag. 151 (rendersleep) and I'm guessing every version after that has it, but I haven't tested all of them.

[Quib Mask]

Yeah, no need to literally test them all. Having pinpointed 150 -> 151 is likely conclusive, and should be related to my previous theory of it being associated with the sleep calls. Now to figure out why what I had you try before was inadequate (or if there was another change between those versions that I'm forgetting).

For a quick and dirty test while I dig, in the JK2022 executable:
- at offset 0x10E1AA you should see
FF 25 A4 04 8F 00
- change to
C2 04 00 90 90 90

That should disable all sleeps in any main logic loops that I added.

QM

[SMLiberator]

I don't think that did anything I tried changing both 0x10E016 and 0x10E1AA offsets to the values you said but neither worked alone nor together.

Just out of curiosity, what are those sleeps for?

[Quib Mask]

They keep the game from pointlessly pegging a CPU core to 100%; on a CPU from this millennium, JK can use less than 1% at the pause menu if allowed to idle. A 1ms sleep is inserted in 3 places: in the pause menu message pump (both when in a level, or when not) and in the full screen rendering loop. If that is what's giving you trouble, my current hypothesis is that Linux is either pickier about minimum sleep time (JK asks for 1ms accuracy but maybe Linux ignores that?) or is handling Windows' window message emulation in a way that stuffs up the sleep logic. Buuut... the second change I suggested, the one at 0x10E1AA, just disables all 3 sleeps, so behavior should be back like vanilla, so I need to identify what else changed between rev 150 and 151 that might be the cause; it's entirely possible it has nothing to do with the sleeps.

QM

Edit: There isn't a ton changed between 150 and 151; I just reorganized how the sleeps are patched in. At first glance I'm confused how it makes any difference. I'll set up a patch for you to apply tomorrow or something. Are you fine with (i.e. able to use in Linux) the bspatch utility I use in the patch distribution, or would some other patch format be your preference (e.g. IPS)? Or heck, if you want to keep patching these by hand, they are small, the next thing to try will be 20 bytes, so, up to you.

[Jon`C]

my current hypothesis is that Linux is either pickier about minimum sleep time (JK asks for 1ms accuracy but maybe Linux ignores that?) or is handling Windows' window message emulation in a way that stuffs up the sleep logic.

TL;DR version:

On Windows, the "timer" to which timeBeginPeriod/NtSetTimerResolution referred isn't just some "system clock", but rather the programmable interval timer (the clock on the motherboard that generates hardware interrupts, periodically switching control from applications back to the operating system kernel). Calling these functions has a direct effect on latency:

- USB devices are polled by kernel code, which now runs more often.
- Processes end up with slightly less total CPU time, but they wake up more often.

Sleep() of any length blocks the calling process and returns control to the kernel, which will schedule other processes to run. That is the real minimum waiting period, and that's what timeBeginPeriod limits by way of the second point.

That's how things work on Windows. Architecturally, however, Linux is a totally different operating system. There's no way to actually implement this version of timeBeginPeriod/NtSetTimerResolution on Linux, so Wine... doesn't:

https://github.com/wine-mirror/wine/...mm/time.c#L372

But this is fine, because literally everything I said above is Windows implementation details. You aren't supposed to rely upon any of this stuff, and for the most part, programs don't. The implementation details only start leaking in situations like this, where someone is trying to do something cheesy because doing it the right way is too hard (windows events, swapchain+vsync).

You aren't even supposed to know this stuff. It's forbidden knowledge. I command you to forget it the instant you finish reading this post.

[SMLiberator]

It's odd to me that I also suffer from keyboard input lag with JK13, though. Also odd that it only seems to affect keyboard input but not mouse input.
IIRC last time I had to use an old Windows install for patching the executable, so I think hex editing will do fine.

[Quib Mask]

While sleeps aren't useful for precise timing, yielding for a hopeful 1ms when you know you're not doing anything for 10+ms makes a world of difference for CPU usage. As far as I recall, Sleep(0) behavior changed between the Win9x lineage and WinNT, where on the former it actually yields, while on the latter it basically never yields (there are theoretical circumstances where it will, but they aren't typical real world circumstances), so to give any sort of benefit on modern Windows, when patching sleeps into old software to try and get it to behave better, all you can (realistically) do is Sleep(1) and make sure something called timeBeginPeriod (or equivalent).

To make matters stupider, after all these decades, Windows wedged in a change to timeBeginPeriod maybe to try and have something like program specific accuracy, or maybe some misguided battery saving measure?
https://randomascii.wordpress.com/20...t-rule-change/
It only outright breaks things when you relied on, say, a helper program that set the timer accuracy for the benefit of another program, but timeBeginPeriod was documented as having a global effect, so this wasn't some change to implementation details, it was just a breaking change.

offset 0x10E015 before / after
E8 6A 01 00 00 33 C0 39 2D C4 5E 85 00
E8 E6 4D EF FF 33 C0 E8 63 01 00 00 90

offset 0x10E184 before / after
E8 77 4C EF FF E9 78 FF FF FF 90 90
E8 7D FF FF FF 39 2D C4 5E 85 00 C3

Try that both on JK2022 and on rev 151 and test the in-game behavior, if you would please.

QM

[SMLiberator]

Oh yeah, that absolutely fixed it on JK22.

[Jon`C]

While sleeps aren't useful for precise timing, yielding for a hopeful 1ms when you know you're not doing anything for 10+ms makes a world of difference for CPU usage.

I guess I kind of see it like this: if it's this specific sort of situation, where you're trying to hack it into an old codebase, it's probably fine. I can't imagine spending the effort to do it any other way. If you're starting from scratch, though, Windows already offers a ton of answers for this problem, tailored for specific use cases, and you should definitely use one of those instead of trying to do it yourself. Like presentation intervals in your graphics API, WM_TIMER (these messages coalesce), etc. If nothing else, it makes it less likely that the skeleton crew of interns maintaining the NT kernel will break your app.

[Quib Mask]

Certainly, sleeping and hoping you don't miss a good timing window is a terrible approach, it's just often what's viable when the original code wasn't designed with any yielding. It literally couldn't take certain approaches because they didn't exist yet (like presentation intervals; as far as I know that started with DX9), and if I had to guess, they weren't going to waste any logic on yielding when the target framerate wasn't met on typical hardware of the time anyway. If I had infinite time and interest, I'm not even sure what ideal approach I'd take to retrofit JK.

SMLiberator, It's mildly baffling why that improves things; it's actually more sleep heavy that way. I need to do some testing to make sure it's not one of the variations that hurts framerate on some versions of Windows when holding keypresses. Also, are there still keyboard / mouse sensitivity issues for your driving stuff? I'd like to at least pin down the cause of that (framerate turn speed normalizing from OpenJKDF2?) so it can be toggled with a setting, or something.

QM

[SMLiberator]

I did not manage to fix it (or kind of did by treating any rotational input as keyboard input). Previously, I differentiated keyboard left/right arrow keys from mouse movement by their value, as keyboard input is discrete (either 0 or 90 or -90) and mouse input is somewhat analog. However with keyboard turning speed being linked to FPS (therefore variable) I can no longer differentiate keyboard from mouse input.

A toggle would work well enough I suppose

[Quib Mask]

Okay, let's see if this is all it takes to restore vanilla keyboard turning:
offset 0xD7D15 before / after
E8 26 B6 F6 FF 90
D8 0D 04 2F 52 00

If so, I'll get some sort of toggle hooked up to that. I'm leaning towards a launch argument + in-game console command. I could do it via some UI checkbox or some unused COG accessible setting bit. The UI route is a boatload of work and not something I want to do. The COG route is tempting, but would need to figure out which global setting bit is available. Interested in what you'd actually want to use.

QM

[SMLiberator]

Will test ASAP.

On a completely different note, are you willing to try and fix some issues and limits present in base JK, like actors only rotating using turnright (submode 31) as a rotate anim regardless of which way they're turning? Also the FOV issue (and 32 iterations limit) with FirstThingInView.

There's also a problem with 0x10 physics flags for actors (Object will align itself with the floor surface it's attached to) but I assume that'd be much much harder to fix