Academy for Modding Excellence

Academy of Modding Excellence Public Forums => General Discussion => Topic started by: peachykeen on April 28, 2010, 05:52:27 PM



Title: NWShader
Post by: peachykeen on April 28, 2010, 05:52:27 PM
Heya,

Bit new here, wanted to pop in and say hi, among other things. ;)

I've been working on the project NWShader for a little while now, and the recent versions have added some scripting functions and content-creation options NWN previously hasn't had. Among other things, these include fullscreen and material shaders (seen in NWN2, and The Witcher, Unreal 08, STALKER, Crysis, and other current-gen games) and script function to manipulate these effects during the game.
http://sourceforge.net/projects/nwshader/

For those of you unfamiliar with shaders, they're mini-programs in a simplified language that are run by the graphics card. They have the power to change how things are drawn on screen, from simple scrolling textures to shifting and twisting models and even multi-texturing (base texture, detail, lightmaps, reflections, etc).

They give the artist full control over how their content looks, no longer limited by what texture sizes or formats NWN can use and what special effects are built-in.

NWShader allows shaders to be attached to textures in NWN, appearing whereve the texture is used. Fullscreen shaders can be used for general effects, including some of the usual ones in newer games (bloom, depth of field, SSAO) and a few that work great with NWN (ghostly effects in your death area, alternate planes with totally different looks).

The script functions also allow for area shaders (turn an effect on or off when a player enters/leaves), event-based effects (focal length for cinematic-style cutscenes) and other effects.

Now, for those of you not text-oriented, I bring visual aids: :P

http://img718.imageshack.us/i/nwshader.jpg/ (not taken by me)
Screenshot Thread (http://sourceforge.net/apps/phpbb/nwshader/viewtopic.php?f=5&t=4&sid=293c655500bcfac7bc0bd145be38133d)
My shot album (http://s4.photobucket.com/albums/y145/peachykeen000/nwshader/)

Area shaders: http://www.youtube.com/watch?v=h8FRa5Y75RA
Interactive shaders: http://www.youtube.com/watch?v=9SbZ7Vl2-KM
Neither of those videos are edited. That’s what the game looked like.

Edit: I also have a module built that will be going online shortly, after a bit more testing. It's a showcase of what NWShader can do from the module/scripting side, and of course needs you to be running it on your end. I'll post a link to that when it's stable.

A few examples of what you can do (not shown above):
Ultravision goggles or see-in-the-dark helm that don't use light effects
Realtime visual effects during cutscenes or scripted events, including things like fading to/from any color and movie-like focus, blur, and overlay effects
Scripted texture changes (attach material shaders to a forest tileset and use scripts to dynamically change the season)

NWShader can be used by players to improve the graphics in single player or on any server without any extra work or risk on the admin's end. With a few tweaks, the server (or even single-player modules) can send special commands to the player (using SendMessageToPC), giving authors, admins and DMs unprecedented control over what the player sees.

NWShader, by its design, runs client-side and acts much like an override. It is compatible with the CEP, CTP, NWNCQ, Project Q, and (so far) every known module, hakpak, or override. It works with singleplayer and multiplayer, on any server, and poses no security risk. It also works with programs like Fraps and XFire, and the camera hack or any server-specific engine tweaks (so long as NWShader is installed after the tweaks). It runs on Windows 2000, XP, Vista and 7 (32 and 64-bit), and I'm working on a Linux port (I'm afraid I don't have access to a Mac for development there). The system requirements are a bit above NWN and a video card capable of using shaders is obviously needed ;), but with simple configurations NWShader has almost no slowdown and can even occasionally help with some driver problems on newer systems.



Now, I'm looking for content creators and scripters willing to try the new stuff. This isn't a call for people to program for me or make me a module, I'm looking for folks willing and able to incorporate the new features I've provided into their modules or projects and occasionally give feedback. This will greatly help with testing and development, and I have an open feature request thread on the project forums (http://sourceforge.net/apps/phpbb/nwshader/).

In case you're still wondering "why should I": :P
 - You get visual effects never seen before in NWN.
 - You get complete control over how your content is seen ingame.
 - You get simple script commands to manipulate the visuals.
 - You get documentation and any help you need (that I can provide) on using these and getting them set up.
 - You can always request features and improvements (or even contribute code or plugins if you're a programmer)
 - I get feedback on how to improve all the things you're getting. ;)


If you're interested in working with the new stuff, trying out NWShader, if you have tried it out and have feedback, or just have a few questions or comments, please contact me (a reply here, PM to pkpeachykeen at the BioWare boards, or email to cx029a@live.com all work fine). I'll get back to you as soon as I can.

Cheers, Sean :)


Title: Re: NWShader
Post by: Andarian on April 28, 2010, 07:47:41 PM
This looks VERY promising and interesting -- I'll take a look at it when I have some time. I actually have a background in computer graphics, and the idea of writing GPU code to attach to textures sounds very cool.  8)


Title: Re: NWShader
Post by: Invisig0th on April 29, 2010, 10:43:03 AM
I think this is a very, very interesting project. As a Microsoft .NET developer with a CIS degree, this is the kind of thing generates waves of geeky excitement through me. My mind races thinking about the kinds of cool and exciting things that would be done with this.

However, as a NWN builder, I am unfortunately not comfortable with insisting that people install auxiliary DLLs on their client machine in order to fully experience my module, even if this does result in substantially improved graphics for those who do so. That could drastically limit the audience for my module.  Non-programmers have enough difficulty with getting the right module-related files in the right directories, much less messing with installing third party DLLs that intercept instructions to the graphics card. So even in a best-case scenario, this would only be used by a fraction of my players. At that point I must decide between the benefits of taking the time to leverage NWShader for a fraction of my players versus the benefits of spending that same valuable time doing things that would benefit all of my players. Faced with that choice, I'm going to choose the latter every time. Spending time on anything that only a small number of my players may end up using will always be a questionable use of my time. As such, I personally don't really see any practical way to incorporate this into my upcoming work. Other builders may disagree, but that's my own viewpoint on it.

It's clear this is a "chicken and egg" situation. I appreciate the fact that players probably wont' install NWShader if builders haven't made modules requiring it, but I also realize that most builders won't use NWShader if players don't already have it installed. And as a builder, it just doesn't make sense for me to make the first move on this one.

I certainly don't mean to discourage you, because I think what you're doing here is awesome. But as awesome of an achievement as this is, I can't see at this point how you will be able to bridge the gap between an interesting technical demo and widespread acceptance among those playing NWN. Until use of NWShader is ubiquitous among NWN players (which is to say, until it has at least the saturation that CEP has now), I simply won't be able to justify spending any significant time leveraging this technology in my own modules.

As people discuss this further, it may turn out that there are other options here that would facilitate widespread adoption. I hope so. In the meantime, I'm eagerly anticipating playing that demo module to see first-hand exactly what this technology can do.



Title: Re: NWShader
Post by: peachykeen on April 29, 2010, 11:29:47 AM
Quote
However, as a NWN builder, I am unfortunately not comfortable with insisting that people install auxiliary DLLs on their client machine in order to fully experience my module, even if this does result in substantially improved graphics for those who do so. That could drastically limit the audience for my module. -snip- Other builders may disagree, but that's my own viewpoint on it.

I fully understand that, and pretty much expected it at the beginning. Because of this, I have a couple things to mention that may help your opinion.
First, a module built using NWShader commands will be usable by all players, whether they have it installed or not. All the commands are sent from module/server to client using SendMessageToPC, so there are a few things you can do. I have a partially finished module, for a PW, that asks the player in the beginning if they have NWShader installed and stores their response as a local int. Whenever the module might want to send a NWShader command, it checks that and can send or skip (for the area shaders, it amounts to an extra 2 lines of code each in the OnEnter and OnExit scripts). If you send a script command to a player without NWShader, it will show up in the message window (no harm, but kinda annoying), if they have it, the script commands will be hidden.

Edit: An example of turning a shader on when a player enters (uses an abyssal plane shader, just for example):
Code:
if ( GetLocalInt(entering_PC, "options_nwshader") == 1 ) { SendMessageToPC(entering_PC, "|nwshader fs_on abyssal"); }
That's it. Even more complicated things use a similar syntax, so hopefully it won't be hard to support. I've found adding NWShader commands to scripts is pretty easy so far, but I am biased toward it, of course.  :D

Second, I'm willing (and offering) to put my time into adding NWShader support to modules. I obviously can't go over every module and add it, but if you want to use a few features, I'll be happy to write the scripts and shaders for you, either to get you started or add simple support. I'm not asking anyone to take huge amounts of their own time, I'm completely willing to do a lot of the work. I see it as a pretty fair tradeoff. :)
Third, I can't quote exact usage stats, but I do have download counts for NWShader. The last 3 updates, in order, were downloaded 159, 206, and the latest 303 times so far. So there are a few people out there using it. Not many yet, but that's what this is all about. ;)

Quote
Non-programmers have enough difficulty with getting the right module-related files in the right directories, much less messing with installing third party DLLs that intercept instructions to the graphics card.
I've worked on projects like this for other games, and trust me, I know how running tech support for this is. To help out non-programmers, I have a regular old GUI with a button to attach NWShader (done some technical overhauls since the last update that make it a lot easier). You just grab the archive, extract, run the GUI, click the button, and you're good to go.  Hopefully everyone can manage that. ;) Everything else has tooltips in the GUI and is documented on the project wiki.

Quote
It's clear this is a "chicken and egg" situation. I appreciate the fact that players probably wont' install NWShader if builders haven't made modules requiring it, but I also realize that most builders won't use NWShader if players don't already have it installed. And as a builder, it just doesn't make sense for me to make the first move on this one.
I understand completely. Defeating the "chicken and egg" situation takes something of capital, and I'll put in as much time as I'm able to get this out there and in use.

Quote
In the meantime, I'm eagerly anticipating playing that demo module to see first-hand exactly what this technology can do.
I sent a test copy to my testers last night and am getting some feedback already. Might have a few bugs to iron out, then I'll be releasing. The module is online and appears to be working fine.


Title: Re: NWShader
Post by: Invisig0th on April 29, 2010, 01:38:30 PM
First, a module built using NWShader commands will be usable by all players, whether they have it installed or not.
Oh yes, I was aware of that. The issue is not whether or not the new features degrades gracefully. I've seen your work, and I assumed that you would not be doing this if this was not the case.

Second, I'm willing (and offering) to put my time into adding NWShader support to modules.
That is a very generous offer, and I hope that builders who will be using NWShader do collaborate with you in that way. Even so, doing what you describe will unavoidably tie up some of the original author's time -- not just with regard to initial implementation, but also in troubleshooting subsequent problems that may or may not be related to NWShader. If something doesn't work properly, people will inevitably complain to the builder about it. In my own personal experience, technical people almost always underestimate the considerable time and effort required to work the kinks out of a system like this.  No matter how much of this time burden you graciously offer to bear yourself, the builders that you work with (at least initially) will still need to devote a significant amount of time and energy to building their module in a way that works properly with your tool. That is time that could instead be used for building things that would benefit every single player, including the ones that choose not to use NWShader. It's up to the builder to decide which way they want to go with that, and my decision at this time would be to do things that are useful to everyone whenever possible. Please don't interpret this as any sort of negative critique of your work here -- it is not.

It's also important to note that you and any builders that you work with will be asking players to install something that makes a fundamental change to the way the game executable works, regardless of what module is being played. In many ways it is similar to requiring people to put files in their OVERRIDE folder, because it is a change that potentially impacts every module played with this game executable. That fact is going to make it harder for ALL builders to debug problems that are reported with their modules, because not only will they have to ask people with problems to empty out their override folder, now they will also have to also ask people to remove NWShader for debugging purposes (which those players may or may not even know how to do). So what you are doing here will most certainly directly impact all the builder who have no interest in using your tool in their existing or upcoming modules. This is something that cannot be said for the other comparable projects like CEP, ProjectQ, etc. And some folks will not be happy about it.

No matter how excellent you are at doing what you are doing (and I have no doubt that your skills here are exceptional, and your motivations good), the fundamental idea of what you are doing brings up lots of concerns and drawbacks. These things will unfortunately remain a barrier to widespread use of your tool. It seems to me that this is just a fundamental characteristic of the type of thing you are trying to do here. The people who use your tool effectively will definitely love it -- I have absolutely no doubt of that. But there will also be lots of people who either don't want to use it, don't bother making the effort to figure out what it is or how to use it, or don't even know about it. As a builder, I need to focus my energies on making modules that *all* those people can play.

I'll say it again -- I'm very excited to see where this project leads. I'll definitely be trying any demos you upload, and I may even take some time and test it for fun. But this is likely to be something that I never actually implement with my own projects due to the various concerns I've mentioned above. For the foreseeable future, my efforts must remain focused on games that can be fully enjoyed with a vanilla NWN installation and no third party tools. I'm sure you can appreciate the reasons behind that. That's not meant to detract from the exciting work you are doing here or to discourage others from using NWShader in their projects. But I do think it is important to have an open discussion about some of the reasons why someone might *not* want to take this route. Like everything else in software development, NWShader has pros and cons, and knowing about them helps people to make better decisions.


Title: Re: NWShader
Post by: Invisig0th on April 30, 2010, 04:26:05 AM
Also, please don't misquote me or paraphrase me in other forums. The following is from your post on the Bioware forums.
Quote
Posted 04/29/10 19:32 (GMT) by pkpeachykeen
I'd also like to address a concern someone brought up at another forum. They mentioned that they'd rather not put their time in if they won't have anyone benefiting from it.
That's not what I said. I said that I cannot use your NWN add-on in my own published module because only a portion of my players would see the improvements that come with NWShader. My concern is that I don't want the rest of the people thinking they are getting a sub-standard version of my module simply because they are not using NWShader. It's a reasonable concern. I'm also not willing to insist that my players install something that changes the underlying way the game executable functions in order to play my module. That is also a reasonable concern.

I'm not saying your tool is not interesting or useful. I'm just telling you that your tool is not the right thing for what I am doing, and why it may never be a suitable fit for my own projects.

I may very well "put my time in" helping you to test this. I did say I was willing to help out. But I can't use this in my module, which is exactly what you asked for in the first post. I thought you deserved to know why I am not willing to do that.
I'm looking for folks willing and able to incorporate the new features I've provided into their modules or projects


Title: Re: NWShader
Post by: Invisig0th on April 30, 2010, 05:34:45 AM
The chances are also very high that you will have at least a few problems with certain video cards. Even if you do everything absolutely right on your end, there definitely will be hardware and/or drivers out there that just don't properly implement the features you are leveraging. Those will be problems that you literally cannot fix by improving NWShader. They would require players to either upgrade their drivers or their video card in order to use NWShader.

As it turns out, there are *lots* of people running NWN using crappy on-board video cards, most of them successfully. I do realize that the ones that don't support shaders at all will not be a problem. It's the ones that have implemented shaders and implemented them badly or in a buggy manner that I'm worried about. And unfortunately, the serious gamers that will be testing NWShader are extremely unlikely to experience problems like that, because they are extremely unlikely to be using a crappy video card.

As a module builder, I certainly don't want to be fielding questions about upgrading video cards and drivers in addition to addressing the normals bug reports associated with a module.  Most players don't even know what a video card driver is. And they won't be complaining to you as the author of NWShader -- they will be complaining to me, as the author of the module that doesn't work properly. And, honestly, it's my job to address any such issues.

You are not yet at the point where you are aware of those problems (or apparently even aware of the likelihood of such problems). It's very easy to be optimistic regarding things like this at the beginning of a project. But personal experience tells me that you are in for some harsh realities further down the road on this one.

The answer to all this, of course, is a lot more testing. It's clear that you are serious about doing lots and lots of testing, which is great. But that kind of testing has to happen *before* most serious module developers will consider integrating NWShader into their beloved projects. Asking people to help you test your tools is one thing, and it is a good thing. Asking people to use your tool in their personal projects without knowing the potential drawbacks is another matter entirely. Integrating a tool like this into serious development projects the last step in the process, not the first.


Title: Re: NWShader
Post by: peachykeen on April 30, 2010, 01:33:25 PM
That is a very generous offer, and I hope that builders who will be using NWShader do collaborate with you in that way. Even so, doing what you describe will unavoidably tie up some of the original author's time -- not just with regard to initial implementation, but also in troubleshooting subsequent problems that may or may not be related to NWShader. If something doesn't work properly, people will inevitably complain to the builder about it. In my own personal experience, technical people almost always underestimate the considerable time and effort required to work the kinks out of a system like this.
Having run tech support on other projects for years, I know full well what kind of time and effort goes into troubleshooting and debugging. I've seen the exact same questions asked hundreds of times by users, handled wiki updates and release distribution for user-bases of over 10,000. It's a pain, I'm not going to lie. But it's something I do have experience with, and with NWShader I'm trying to avoid a lot of the other issues I've seen. Stability and ease-of-use are top priority, and I've implemented every user suggestion so far to improve on both of those. The code is as stable as I can make it, and for the last couple versions, I have not seen a crash caused by NWShader code. The options and features are well labeled and I spend time between every update documenting as many things as possible.

Quote
It's also important to note that you and any builders that you work with will be asking players to install something that makes a fundamental change to the way the game executable works, regardless of what module is being played. In many ways it is similar to requiring people to put files in their OVERRIDE folder, because it is a change that potentially impacts every module played with this game executable. That fact is going to make it harder for ALL builders to debug problems that are reported with their modules, because not only will they have to ask people with problems to empty out their override folder, now they will also have to also ask people to remove NWShader for debugging purposes (which those players may or may not even know how to do). So what you are doing here will most certainly directly impact all the builder who have no interest in using your tool in their existing or upcoming modules. This is something that cannot be said for the other comparable projects like CEP, ProjectQ, etc. And some folks will not be happy about it.
This is true, and is, of course, based on how NWShader functions. It's impossible, with the current knowledge of the engine, to prevent that. However, I can (and have) avoided the issue as much as possible from my end. Disabling NWShader is as simple as enabling it, a single click. NWShader makes a backup of the single file it modifies, and is compatible with all other known modifications to the engine. In addition, all of NWShader's code sits beneath the engine (with the exception of the 4 hooks, which use Microsoft-designed libraries to safely connect to the game).

Also, please don't misquote me or paraphrase me in other forums. The following is from your post on the Bioware forums.
Quote
Posted 04/29/10 19:32 (GMT) by pkpeachykeen
I'd also like to address a concern someone brought up at another forum. They mentioned that they'd rather not put their time in if they won't have anyone benefiting from it.
That's not what I said. I said that I cannot use your NWN add-on in my own published module because only a portion of my players would see the improvements that come with NWShader. My concern is that I don't want the rest of the people thinking they are getting a sub-standard version of my module simply because they are not using NWShader. It's a reasonable concern. I'm also not willing to insist that my players install something that changes the underlying way the game executable functions in order to play my module. That is also a reasonable concern.
I had no intention to misquote you, apologies if it seemed I was. You did state that:
Quote from: Invisig0th
But this is likely to be something that I never actually implement with my own projects due to the various concerns I've mentioned above. For the foreseeable future, my efforts must remain focused on games that can be fully enjoyed with a vanilla NWN installation and no third party tools.
While I may have paraphrased (to avoid a direct, identifiable quote, which I thought would certainly be rude), they have the same meaning. You're not planning on using it, at least not for now. I understand your reasoning, and obviously, you're not in the crowd I'm trying to sell to. ;) If you choose not to use the features or incorporate it into your work, so be it. I'm still looking for anyone who is interested in doing so, and I've found at least a few people so far.

Quote
The chances are also very high that you will have at least a few problems with certain video cards. Even if you do everything absolutely right on your end, there definitely will be hardware and/or drivers out there that just don't properly implement the features you are leveraging. Those will be problems that you literally cannot fix by improving NWShader. They would require players to either upgrade their drivers or their video card in order to use NWShader.

As it turns out, there are *lots* of people running NWN using crappy on-board video cards, most of them successfully. I do realize that the ones that don't support shaders at all will not be a problem. It's the ones that have implemented shaders and implemented them badly or in a buggy manner that I'm worried about. And unfortunately, the serious gamers that will be testing NWShader are extremely unlikely to experience problems like that, because they are extremely unlikely to be using a crappy video card.
There are some issues with certain cards. I've avoided as many as physically possible, but using native drivers and standard Windows linkage. Of course, there is a lower limit for hardware. There are very few (if any) chipsets and drivers that misimplement the extensions NWShader uses, for the most part they are either implemented or completely absent. In the case of some issue, all NWShader code is surrounded by try-catch blocks (which allow it to internally handle issues), with error-recovery and logging in the catch blocks. For example, I had a few issues early on with bad or corrupt shaders crashing the game. I added code allowing NWShader to dynamically disable any shader it finds might have a problem on that particular system, at runtime. In other words, it's designed to adjust to the game as it runs and recover from any errors to the fullest extent possible with the C++ language.
As for testing, I actually have a range of video cards and systems with people helping test. NWShader has been tested on Windows 2000, XP, Vista and 7, on everything from a Radeon x800 to HD 5000-series cards, as well as a similar variety of nVidia chipsets. The only incompatible setup we've identified so far was an Intel GMA550, if I remember right, which is renowned for having poor 3d support. The wiki has a page listing all known compatible and incompatible cards, as well as links to other pages with feature summaries of various generations and chipsets. All possible care is being taken to make NWShader as universally compatible as possible.

Quote
You are not yet at the point where you are aware of those problems (or apparently even aware of the likelihood of such problems). It's very easy to be optimistic regarding things like this at the beginning of a project. But personal experience tells me that you are in for some harsh realities further down the road on this one.

The answer to all this, of course, is a lot more testing. It's clear that you are serious about doing lots and lots of testing, which is great. But that kind of testing has to happen *before* most serious module developers will consider integrating NWShader into their beloved projects. Asking people to help you test your tools is one thing, and it is a good thing. Asking people to use your tool in their personal projects without knowing the potential drawbacks is another matter entirely. Integrating a tool like this into serious development projects the last step in the process, not the first.
Like I mentioned above, I'm fully aware of both ends of a project like this. I've been doing development on them for a while, and running tech support for some time before that. If anything, I'm overly pessimistic about such things. I know what it's like to answer a question that's been answered 3 posts up, 10 before that, and is in the contents list of the opening post of the stickied thread. :P This isn't my first time down this path, and I'm taking what I've learned before into heavy consideration.
In regards to testing, I am running as much testing as possible. This project is still in development, obviously, with new features being added all the time. Before a release ever sees the main download page, I run it through a group of testers and fix as many issues as possible. A few have slipped through, but were handled as soon as I was able to (mostly configuration issues and some driver problems in the first few revisions). What is implemented is highly stable, especially for something that's commonly considered impossible. I've delayed releases to fix a single crash before, and the current version (with script functions) took quite a while to get to a state where I was even remotely confident with it. Even still, it's in internal testing before it will see a public release, because of how much has been changed.
In the opposite end of your comment, though, I have seen a similar addition take off in a game community. Once it become stable and known, usage went through the roof and it became an established part of the game, with not using it the exception. At this point, I have no idea if NWShader will reach that level of acceptance, but I will do anything I can to create as high-quality a product as is possible.

In summary, NWShader is a (for NWN) advanced addition. It does things not done before, and is somewhat complex. That's necessary, because it does things none of the developers anticipated or prepared for. Some things have to be worked around, some handled as problems appear.
Everything possible is being done to make NWShader stable, reliable, useful and easy to use. I keep a bug tracker online at all times, as well as searching Google for any issues people run across that I haven't seen or weren't reported. I'll be continuing active development and troubleshooting indefinitely, or until such time as NWShader is completely stable and has every feature anyone wants.
I understand that, at this stage in the game, not many people will be jumping on board the new ship that's showed up. I completely understand your points, and (with all due respect), you're not in the crowd I'm selling to. If you want to use NWShader, if it becomes an established part of the community, I'll help you develop for it then. Until then, I hope you enjoy any playing with it, and any problems you spot, just send em' over. :)

If there are any authors willing to give it a shot or who want to try it out now, early adopters per-say, I'll certainly help with the development of any module taking advantage of the additions.


Title: Re: NWShader
Post by: Invisig0th on April 30, 2010, 02:44:31 PM
Alrighty then. I guess I'll just suggest that you might want to be a bit more forthcoming up front regarding the known limitations and problems of NWShader, as well as being more open to the possibility that other limitations and problems will be found. Presenting your software as if it is completely risk-free and bug-free is always guaranteed to arouse suspicion from people who know better.  It does appear that you honestly believe that there is no possible way that what you're doing could cause serious problems for some people. I sincerely hope you're right about that.

However, I still don't think you really appreciate how much of a burden you would potentially be putting on the entire NWN community by encouraging hundreds of people to replace a critical game DLL for the sake of better eye candy. Replacing a Bioware DLL with an altered version is a very drastic change, and doing that can easily cause all sorts of nightmares for the end users. Anyone who doesn't see that as a cause for at least mild concern is simply not being realistic. I realize you are completely comfortable with that risk, but not everyone will be. And this won't just impact people who install NWShader -- it will impact a lot of other folks. You yourself have said that what you are doing here is very similar to putting files in the override, since the changes your tool makes are global in scope. As we all know, files in the override are *notorious* for making troubleshooting other problems significantly more difficult. Builders are going to be extremely unhappy if it turns out that NWShader breaks any of their (non-NWShader) modules, even if only a small number of modules and a small number of players are impacted. Mark my words -- you will definitely get complaints if and when that happens.

I'm sorry if you feel I'm picking on you here - that is not my intention. But these concerns are very real and very relevant. You will need to wrestle with these things in one way or another.

As for misquoting me -- you did misquote me. You said "They mentioned that they'd rather not put their time in if they won't have anyone benefiting from it." I never said that I wouldn't "have anyone benefiting from it". In fact, I openly admitted that some portion of my players would definitely benefit from me using NWShader. My concern was that not everybody would benefit, which is a very different (and entirely accurate) statement.

Anyway, best of luck with your project.  Believe it or not, I am interested to see where it goes. I just wish you had been a little more open up front about the potential drawbacks, and a little more understanding of why knowledgeable people might reasonably be reluctant to adopt this kind of fundamental change to the game's code. Your apparent reluctance to discuss these very real risks gives me some cause for concern regarding this project.


Title: Re: NWShader
Post by: peachykeen on April 30, 2010, 04:24:48 PM
Alrighty then. I guess I'll just suggest that you might want to be a bit more forthcoming up front regarding the known limitations and problems of NWShader, as well as being more open to the possibility that other limitations and problems will be found. Presenting your software in a way that could be read as saying that it is completely bug-free is certain to arouse suspicion from those people who know better.

There have been bugs, in fact there are a couple minor visual ones I have yet to iron out. It is not completely bug-free, but I'm certainly trying to get it there. There are, obviously, limits, and I mentioned those. Both hardware-wise and how much I can get into the engine, I've run into a few limits already.

Quote
It does appear that you honestly believe that there is no possible way that what you're doing can cause big problems for some people. I sincerely hope you're right about that. However, I still don't think you really appreciate how much of a burden you will be putting on the entire NWN community by encouraging hundreds of people to replace a critical game DLL for the sake of better eye candy. Replacing a Bioware DLL with one you wrote is a very serious change, and it can obviously cause all sorts of nightmares for the end users. Anyone who doesn't see that as a cause for at least mild concern is simply not being realistic. I realize you are comfortable with that risk, but not everyone will be. You yourself have said that what you are doing here is very similar to putting files in the override, since the changes your tool makes are global in scope. As we all know, files in the override are absolutely *notorious* for causing confusion and often make troubleshooting other problems significantly more difficult. Builders are going to be extremely unhappy if it turns out that NWShader breaks any of their (non-NWShader) modules, even if only a small number of modules and a small number of players are impacted. Mark my words -- you WILL get complaints if and when that happens.
I think you misunderstand how it works there. It doesn't overwrite anything BioWare, or even anything else. I haven't replaced any of BioWare's DLLs, only added a new one. Any code sits between BioWare's original interface and the existing system interface.
Imagine a cable, I simple added a box halfway down the line. It takes NWN's commands, modifies them as needed, and passes them onto the system. If anything goes wrong, it falls back to a simple pass-through method that does no modification.

Quote
As for misquoting me -- you did misquote me. You said "They mentioned that they'd rather not put their time in if they won't have anyone benefiting from it." I never said that I wouldn't "have anyone benefiting from it". In fact, I openly admitted that some portion of my players would definitely benefit from me using NWShader. My concern was that not everybody would benefit, which is a very different (and entirely accurate) statement.
Again, apologies. It does appear my turn of phrase misrepresented you, and that is my mistake.

Quote
Anyway, best of luck with your project.  Believe it or not, I am interested to see where it goes. I just wish you had been a little more open up front about the potential drawbacks, and a little more understanding of why knowledgeable people might reasonably be reluctant to adopt this kind of fundamental change to the game's code. I personally don't believe that this is worth the risk. On that point we'll have to agree to disagree.
Thank you for your interest, and I'm sorry for the misquote and any (accidental) misleading. There are a few issues, I'm just trying to get rid of them. Nothing in the game code or system code changes, I simply add a layer in-between.


Title: Re: NWShader
Post by: Invisig0th on April 30, 2010, 04:31:56 PM
My mistake. You're absolutely correct, of course -- this is adding a link to the chain, not replacing an existing link. However, adding another potential point of failure inevitably results in increasing the total number of failures. hopefully that number will be very low in this case.

Thanks very much for the cordial and reasonable discussion. We may not agree on some of these points, but hopefully at least we can both see where the other person is coming from.


Title: Re: NWShader
Post by: peachykeen on April 30, 2010, 07:07:24 PM
My mistake. You're absolutely correct, of course -- this is adding a link to the chain, not replacing an existing link. However, adding another potential point of failure inevitably results in increasing the total number of failures. hopefully that number will be very low in this case.
Unfortunately true. :)
I have had plenty of crashes in the tests I've done, especially in the engine hook. The scripting functions alone took weeks of solid work to get working right, and it was a lucky find on Google that helped me simplify them, otherwise it would have been quite a while longer. I've found Microsoft to be most helpful in this project, at least their Detours library (it has some helpful tutorials and manages all the engine hooks for me).
All in all, the fallback code I added removed almost 90% of the crashes I'd seen or heard of, which was a very nice result for very little code. There is one I know happens occasionally at the moment (sometimes the game crashes when no shaders are enabled and you close the game, due to something in nVidia's code or how I call it), but not too many pop up anymore. There's still a few visual bugs, but nothing game-breaking. For a project still technically in alpha, I'd like to think that's a good thing.

Quote
Thanks very much for the cordial and reasonable discussion. We may not agree on some of these points, but hopefully at least we can both see where the other person is coming from.
I do understand your reasoning, and you do make valid points. I, of course, am terribly biased towards my project and wish everyone instantly loved it. :P I know that's not going to happen, so I'm trying to get as close as I can.

I'm also starting to wonder if there's anyone else on these forums. :P


Title: Re: NWShader
Post by: QSW on May 01, 2010, 10:32:47 AM


I'm also starting to wonder if there's anyone else on these forums. :P

Aha, yep, more of us than you might guess from the absence of loud banging and chatter.  Of course, I only have a very basic layman's idea of what you're talking about, but don't let that stop you....it hasn't stopped me yet ;)


Title: Re: NWShader
Post by: peachykeen on May 01, 2010, 08:17:20 PM
Well, for anyone interested, I've released the update and have the demo online.

The download is here:
https://sourceforge.net/projects/nwshader/files/

And the address for the server thus:
http://cx029a.dnsdojo.com
It's also listed as "NWShader Server" under the action category in GameSpy. It's a simple server, just showing off some of the new script functions. The default setup NWShader comes with is basically what the server uses, although you can enable the shader "color.cgfx" (check it off under the GUI's shaders tab) for a few more fancy functions.

I've had a few bug reports so far and am trying to track them down at the moment, but I've gotten far more success reports with this release. Even the bug reports have had details and complete logs (thank the gods for on-crash logging).

The program that attaches NWShader creates a backup of the executable, so if you want to remove it, just delete the new one and rename that. I haven't seen or had any reports of module or content problems with the default settings (some of the WIP advanced stuff can break things if you turn it on, though).


Title: Re: NWShader
Post by: Invisig0th on May 01, 2010, 09:16:18 PM
While looking into installing this latest version of NWShader, I see that installing NWShader requires literally replacing the main EXE for the game. Are you out of your mind?

I honestly think I understand why you don't think this is a big deal. But it's pretty clear that you do not fully appreciate why a whole lot of people are going to be unwilling to literally replace the game's executable file with something they downloaded off the internet. It does not matter how minor the changes to the main executable are, and it does not matter that you made it easy to revert to the "real" game executable -- you are asking people to do something very risky and very drastic here in order to use your tool, and providing no warning at all about the potential risk in doing so. Replacing the main executable of the game could break ***ANYTHING***. It's clear that this serious risk simply does not show up on your own personal radar, but the fact that you don't seem to understand why doing something like this will be a problem for a lot of people is severely troubling.  And the fact that you don't see how this could cause widespread problems throughout the community if a problem somehow crops up AFTER your tool is installed by hundreds of people is, quite honestly, frightening.

Sorry, for my part I'm uninstalling and deleting NWShader entirely. No thank you. It's terribly foolish in general to install any program that changes the core game files like this. Only people who know exactly what they are doing, and who are willing to reinstall the game if anything gets messed up, should even *consider* such a thing.

I'm sure you're ready to respond by telling me about all the extra effort you have put in to prevent such problems. And while all that may be true, you and I both know that this does not mean you can guarantee that no serious problems will occur. Everyone who replaces their core game files by installing this tool is taking a very serious risk, and you currently do not appear to be taking ANY steps to notify people about that. At the VERY least, you should have a disclaimer on your download page clearly explaining that the NWShader installation changes the core game files, that you cannot guarantee that serious problems will not occur due to those changes, and that anyone who installs NWShader must do so at their own risk. That is exactly what any responsible developer would do here.

With all sincerity, I truly mean no disrespect to you personally or to what you are trying to do here.You definitely have a high level of knowledge of the subject matter, and NWShader will be an interesting experiment for those people who are familiar with these types of things.  And I do think what you're doing is interesting on a very esoteric, academic level.  But the way you're going about this is terribly irresponsible.  If you honestly think that encouraging people not familiar with these things to alter their core game files without warning them of the potential risks is a good idea, then clearly there are some very important things that you don't understand at all.

Someone will definitely have to warn your users about the serious potential risks of what you are doing here, and it would be best if those warnings came from you as the developer. I would encourage you to take a good, hard look at all this. And this is exactly what I have been saying since your first post. The way in which you are going about this project sends up all sorts of red flags. I realize this is your baby and that it's natural to be overly optimistic, but please consider showing more concern for your users and for the NWN community as a whole. People will have a lot more respect for what you are doing here if you err on the side of caution. And the more radical the nature of what you are doing, the more reasonable it is to be overly cautious. If something does go seriously wrong, at least people cannot say they weren't warned of that possibility. Please do the right thing.


Title: Re: NWShader
Post by: Andarian on May 02, 2010, 04:36:06 AM
While looking into installing this latest version of NWShader, I see that installing NWShader requires literally replacing the main EXE for the game.

Preface: I haven't had time to look at this at all or to personally check any of these claims. But if it's true that NWShader replaces the main game executable, then isn't it in violation of the NWN EULA? Perhaps I'm misremembering (and someone correct me if I am), but as I recall it the EULA prohibits altering the game's executable code without permission. Adding new executable code to run on the GPU is one thing, but I would think that replacing a Bioware executable would be something else altogether.

I still think this is an very intriguing idea. But I also tend to agree that Invisig0th has a point that installing something like is a bigger deal than I'd first thought.


Title: Re: NWShader
Post by: peachykeen on May 02, 2010, 07:47:24 AM
While looking into installing this latest version of NWShader, I see that installing NWShader requires literally replacing the main EXE for the game. Are you out of your mind?

Actually, I am. Otherwise I wouldn't have tried to add shaders to a game made before they were invented. :P
But it doesn't replace the engine. It hooks into it, similar to how NWNCX does, or the camera hack. I think you're going by the final appearance of having a backup of the engine. The method NWShader uses has been developed and distributed as a tool by Microsoft, the same company that made the executable format. Not only does it make an internal backup (the modified EXE actually has a backup of the original stored in it), but I make an extra backup for safety's sake. If the patch can't make a backup, it stops itself and doesn't even touch anything. I honestly can't think of more safety measures to add.

Now, it is (in rare cases) a risky thing to do. Using Microsoft's tools to do it makes it easier and safer, but it is something of a drastic change. To get NWShader to work how it should, it's the best way (I've tried 3 or 4 different methods, and this one is all-around the most stable, safest and simplest, believe it or not).

To explain, I may get a bit technical, but here's how NWShader works:
You run the hook program to attach NWShader. Now, almost all programs on Windows use DLLs. DLLs have code that is separate from the program, but the program can use. DLLs can be upgraded easily and by other companies (a video card company can make driver DLLs and upgrade them without updating every game using them).
By default, Neverwinter Nights asks Windows to load the file OpenGL32.dll, which is the video drivers. The hook program in NWShader asks Detours ( http://research.microsoft.com/en-us/projects/detours/ ) to change that to NWShader.dll, which loads OpenGL32.dll itself.
So now, once Detours does its thing, NWN loads NWShader, which loads OpenGL. It's just a basic go-between layer, the only way it modifies the executable is changing the name of the file the game asks for.

Now, for those of you familiar with how a program is set up, all the names are stored in something called an import table (described here http://en.wikipedia.org/wiki/Portable_Executable#Import_Table ). The program is made of sections; first comes code, imports, exports, then resources and text. What NWShader does, through Detours, is change one entry in the import table. Detours, by design, copies the original table to the end of the file (so it can be restored later) and modifies the selected entry.

None of the game code is changed or touched, at all.

The source code for the patch is publically available here:
http://nwshader.svn.sourceforge.net/viewvc/nwshader/tools/hooker/src/hooker.cpp?revision=21&view=markup
So anyone worried about what's going on inside can see exactly what happens (or have their programmer friends take a look). Nothing NWShader does is hidden, the full source is available for anyone interested or concerned.

Quote
Only people who know exactly what they are doing, and who are willing to reinstall the game if anything gets messed up, should even *consider* such a thing.
Well, since NWShader only changes the NWMain file (one file in the game) and makes a backup first, hopefully any reinstalls can be prevented.
That's the only file it touches, so nothing else can get broken, even if things go horribly wrong.
In case things do go wrong, it makes a backup and the install page tells users to keep the backup safe.

Quote
I'm sure you're ready to respond by telling me about all the extra effort you have put in to prevent such problems. And while all that may be true, you and I both know that this does not mean you can guarantee that no serious problems will occur. Everyone who replaces their core game files by installing this tool is taking a very serious risk, and you currently do not appear to be taking ANY steps to notify people about that. At the VERY least, you should have a disclaimer on your download page clearly explaining that the NWShader installation changes the core game files, that you cannot guarantee that serious problems will not occur due to those changes, and that anyone who installs NWShader must do so at their own risk. That is exactly what any responsible developer would do here.
Oh, certainly. I've been working yesterday and today to update the wiki with info on the new version, and I'm working on a more detailed page of how NWShader works. In the install instructions, I already added a line mentioning that NWShader modifies that file and warning users to keep a backup safe incase anything does go wrong.
Going with the GPL's usual disclaimer line, NWShader itself warns the user every time it's run that it has no warranty and all that, as does the license included with it. I don't think I can pack that disclaimer into too many more pages. :P

Quote
With all sincerity, I truly mean no disrespect to you personally or to what you are trying to do here.You definitely have a high level of knowledge of the subject matter, and NWShader will be an interesting experiment for those people who are familiar with these types of things.  And I do think what you're doing is interesting on a very esoteric, academic level.  But the way you're going about this is terribly irresponsible.  If you honestly think that encouraging people not familiar with these things to alter their core game files without warning them of the potential risks is a good idea, then clearly there are some very important things that you don't understand at all.
Well, can you think of another way to go about it? Not to be smart, but when tweaking graphics code after-market, there are very few methods that make it possible.

You can add an OpenGL32.dll to the game directory, which I tried (in two flavors).
The first try required copying your system drivers into the NWN dir also, which was a serious pain and caused problems for me in development and users.
The second adds that file and dynamically linked to the drivers, which would have worked (and would have been the ideal solution), except NWN refused to see the OpenGL start function and so didn't run, at all.
I'm limited by both what the system can do and what NWN can do/wants to do, so my options are pretty narrow. If you can come up with another way to do it, and not touch NWmain, I'll definitely try to use that, but I'm out of ideas at this point, and Microsoft is handing out a neat, clean solution.

Quote
Someone will definitely have to warn your users about the serious potential risks of what you are doing here, and it would be best if those warnings came from you as the developer. I would encourage you to take a good, hard look at all this. And this is exactly what I have been saying since your first post. The way in which you are going about this project sends up all sorts of red flags. I realize this is your baby and that it's natural to be overly optimistic, but please consider showing more concern for your users and for the NWN community as a whole. People will have a lot more respect for what you are doing here if you err on the side of caution. And the more radical the nature of what you are doing, the more reasonable it is to be overly cautious. If something does go seriously wrong, at least people cannot say they weren't warned of that possibility. Please do the right thing.
I'll add extra notes to the install page and the thread.
As far as erring toward caution, honestly, everything has been done to make this as safe as possible. I understand the risks and what could happen (being familiar with the PE format, I may have a better idea of what can really happen than most). I think you may be over-reacting a tad, but hopefully the links above and the code example will help you understand what's really happening and how it works.

Quote from: Andarian
Preface: I haven't had time to look at this at all or to personally check any of these claims. But if it's true that NWShader replaces the main game executable, then isn't it in violation of the NWN EULA? Perhaps I'm misremembering (and someone correct me if I am), but as I recall it the EULA prohibits altering the game's executable code without permission. Adding new executable code to run on the GPU is one thing, but I would think that replacing a Bioware executable would be something else altogether.
NWShader doesn't use a new copy of the file, or replace it, or anything like that. It uses the same sort of thing as the camera hack and NWNCX to modify a small chunk of the file.
As far as I know, from the mods that've done it for NWN and a half-dozen other, similar projects for different games (using very similar or identical EULAs), and a few people checking with their lawyers (actually, Bethesda Softworks ran it by their lawyers at one point), it's allowed on a few conditions.
The main thing is that it be made as a patch. If I were sending out a copy of the executable, that would obviously open all sorts of copyright issues and break EULA. Using a byte-patch to change a file on the user's system is allowed, as long as it doesn't have the code in it. NWShader uses a patching system developed by Microsoft, which meets that limit.
If anyone is really worried that this particular method for this, in NWN, isn't allowed, I can try to get in touch with the developers. I've sent them a couple messages and haven't gotten replies yet. Seeing as I wasn't able to get official word, I went with the patching method that's been allowed previously.



Now, in summary: this is a radical change. There's no other way to go about it.
I've made it as safe as possible, and there is more than one warning. The wiki mentions it, the log file mentions it, the readme mentions it, and the license file mentions it. If it can be made safer, I'll certainly do so.
Once you understand what's really happening, it's not nearly as scary as it looks at first. It's also not nearly as drastic. I haven't seen anyone scared of the camera hack, and that's built using a cracking tool (the byte-patcher used in it is more commonly used for piracy and injecting viruses).

Edit: Actually, I just had a though. Would you all be more comfortable with using this if I made a copy of the executable and changed the copy instead? So you have the old one, plus a new NWShader-enabled one. I could whip up a quick launcher to make it simple for players, and even make it optional (make a modified copy for "safe mode" or backup and modify for "in-place" mode). Opinions on that?


Title: Re: NWShader
Post by: Andarian on May 02, 2010, 09:01:32 AM
NWShader doesn't use a new copy of the file, or replace it, or anything like that. It uses the same sort of thing as the camera hack and NWNCX to modify a small chunk of the file.

As far as I know, from the mods that've done it for NWN and a half-dozen other, similar projects for different games (using very similar or identical EULAs), and a few people checking with their lawyers (actually, Bethesda Softworks ran it by their lawyers at one point), it's allowed on a few conditions.

The main thing is that it be made as a patch. If I were sending out a copy of the executable, that would obviously open all sorts of copyright issues and break EULA. Using a byte-patch to change a file on the user's system is allowed, as long as it doesn't have the code in it. NWShader uses a patching system developed by Microsoft, which meets that limit.

Here's a link to the NWN Eula (http://nwn.bioware.com/forums/viewtopic.html?forum=58&topic=46097) for reference. This is the part that I was remembering:

Quote
3. End-User Variations. So long as you fully comply, at all times, with this License, Infogrames grants to you a limited, personal, revocable, non-exclusive right to: (i) use the Software's toolset feature to create your own NEVERWINTER NIGHTS modules ("Modules"), and (ii) create your own modifications to work with the Software (e.g., custom data files not created using the toolset) (the "User Conversions", and together with the Modules, the "Variations"). Your rights to create Variations are subject to the following restrictions: (1) your Variations must only work with the full commercial version of the software game NEVERWINTER NIGHTS; (2) your Variations must not contain modifications to any executable file; [emphasis added]

As I read this, mods that involve modifications to an executable file are expressly disallowed by the EULA. Whether Bioware cares or intends to enforce the terms of the original EULA I don't know. I would actually tend to doubt it at this stage of the game (pun intended). It's probably also worth noting in that connection that point (3) immediately following goes on to prohibit "pornographic" mods as well -- a provision that's hardly been enforced during the entire history of NWN.


Title: Re: NWShader
Post by: peachykeen on May 02, 2010, 09:19:53 AM
Here's a link to the NWN Eula (http://nwn.bioware.com/forums/viewtopic.html?forum=58&topic=46097) for reference. This is the part that I was remembering:

Quote
3. End-User Variations. So long as you fully comply, at all times, with this License, Infogrames grants to you a limited, personal, revocable, non-exclusive right to: (i) use the Software's toolset feature to create your own NEVERWINTER NIGHTS modules ("Modules"), and (ii) create your own modifications to work with the Software (e.g., custom data files not created using the toolset) (the "User Conversions", and together with the Modules, the "Variations"). Your rights to create Variations are subject to the following restrictions: (1) your Variations must only work with the full commercial version of the software game NEVERWINTER NIGHTS; (2) your Variations must not contain modifications to any executable file; [emphasis added]

As I read this, mods that involve modifications to an executable file are expressly disallowed by the EULA. Whether Bioware cares or intends to enforce the terms of the original EULA I don't know. I would actually tend to doubt it at this stage of the game (pun intended). It's probably also worth noting in that connection that point (3) immediately following goes on to prohibit "pornographic" mods as well -- a provision that's hardly been enforced during the entire history of NWN.

I've seen that before (aren't all game EULAs basically the same?). If I remember right (it's been a while since I last heard any official standpoint on this), the reason such things are allowed plays off a technicality. My patcher doesn't contain a modified executable file. It contains code to modify the file. You still have to have the file in the first place, so there's no chance of helping pirates by handing out copies. That's my understanding of this, at least. Not being a lawyer myself, I can only go by what they've told me, so... Hopefully it is allowed. If BioWare complains, I'll change things around somehow. :)


Title: Re: NWShader
Post by: Invisig0th on May 02, 2010, 10:09:13 AM
There's no doubt it is a direct violation of the EULA. As such, discussing NWShader on nwn.bioware.com is probably forbidden according to forum rules, and the upload of NWShader to NWVault is also probably not allowed. Whether or not they will want to enforce those rules is an open question. But hacking the game code, for whatever reason, is definitely against the rules. In fact, I'm not even sure at this point if is appropriate to discuss this here. I'm not sure if we have a policy regarding game hacks/cheats. Not my call, of course, but it is disturbing that this question even needs to be asked.

You're welcome to discuss with Bioware whether or not you have violated your EULA by distributing this tool. However, it is undeniable that you are indeed making "modifications to (an) executable file". Your tool literally change the code of the game executable, and is therefore a hack by definition. That obviously is something you are responsible for disclosing BEFORE people install your tool. Once again, the fact that you are comfortable with the risks here does NOT mean that it is okay for you to assume that the people who download your tool will agree and will want to do the same thing on their own machines.

Sorry, peachykeen, but it's clear you have some unrealistic assumptions about the concerns that end users are going to have about what you are doing here and how you are doing it, and you also have some unrealistic assumptions about the range of skills among the people in the NWN community.  I would guess that a large number of the people who have already installed this tool "just to try it out" are not programmers, and are people who will not understand how to undo the changes you are making, and *certainly* don't realize that you have replaced their game executable.  Those people will not be visiting your web site to read a disclaimer or read through the source code. In some cases you will literally be dealing with people who would need assistance just to change a file's name.  As a long time member of the NWN community, I can assure you with absolute certainty that even if you did attempt in good faith to properly warn people beforehand about the inherent risks related to your tool, many people would be installing your tool without any clue that you are altering their game files. To not even attempt to provide those warnings (for the people who would bother reading them) is amazingly reckless.

I do understand that you have gone to extreme lengths to address any concerns you could possibly think of. Unfortunately, it is is the concerns that you could NOT possibly think of that often cause the biggest problems. And it won't be just you that pays if that happens. It could be a healthy chunk of the NWN Community. ALL software is flawed, sometimes deeply flawed, despite the best efforts of the developers. That is a well established fact in the field of software development. The best developers accept the fact that their software may fail, perhaps catastrophically, and attempt to manage that risk in ways besides simply fixing known bugs. I don't see you doing that here.

You, from an highly informed point of view, are clearly willing to accept the risks related to your tool. That is perfectly reasonable. Mess with your game files however you like -- that's your business. However, that doesn't mean you can assume that other people much less informed about these topics are just as willing to accept those risks. Allowing someone you do not know to change executable files on your computer is always a Very Bad Thing (or at least a Very Risky Thing). No matter how awesome and thorough that person may be, it can end badly.
 
As you move forward on this, I would encourage you to be extremely careful about what you do, how you do it, and how thoroughly you inform people about the risks related your tool. I really am surprised that I have to explain this to you. So far you've raised a lot of red flags from my point of view as a Microsoft certified .NET Architect with 15 years of experience in the corporate world. Based on my background and this discussion, I regret to say that I would not recommend that anyone unfamiliar with these things install your tool. It's far too risky, and it's being handled in a way that does not properly address those very real risks, which only makes things worse. I suspect you're going to find that many tech-savvy people will be happy to accept the risks and work with you to test and improve this tool. But amateurs (which make up the majority of the NWN community) should seriously avoid the kinds of things you are doing here.

It really is a shame, though. The underlying thing that you are trying to here is actually pretty cool. But even an excellent idea can be ruined if it is handled badly. And I'm certain that you are an *excellent* programmer, but you seem to be falling short on the 'people' side of this equation. If you decided to approach all this in a *much* more responsible manner, I do think you could address almost all of the concerns mentioned here. There are still going to be a fair number of unavoidable risks here no matter what you do, but taking appropriate steps to reduce the danger of those risks (and thereby doing whatever you can to protect your end users from bad results) is the very best way to ensure that your project is successful.


Title: Re: NWShader
Post by: peachykeen on May 02, 2010, 10:51:04 AM
Sorry, peachykeen, but it's clear you have some unrealistic assumptions about the concerns that end users are going to have about what you are doing here and how you are doing it, and you also have some unrealistic assumptions about the range of skills among the people in the NWN community.  -snip-, I can assure you with absolute certainty that even if you did attempt in good faith to properly warn people beforehand about the inherent risks related to your tool, many people would be installing your tool without any clue that you are altering their game files. To not even attempt to provide those warnings (for the people who would bother reading them) is amazingly reckless.
I have provided the warning. I can't force users to read anything, but I can provide the text. It has been available since the release of this new version yesterday, and I'm currently expanding it and making it more obvious. The warnings have been provided.

Quote
I do understand that you have gone to extreme lengths to address any concerns you could possibly think of. Unfortunately, it is is the concerns that you could NOT possibly think of that often cause the biggest problems. -snip- The best developers accept the fact that their software may fail, perhaps catastrophically, and attempt to manage that risk in ways besides simply fixing known bugs. I don't see you doing that here.
You haven't read the code. It has general try-catch blocks linked to the disable portion, designed to gracefully exit and log any errors I haven't added specific handling for or haven't seen before.

Quote
You, from an highly informed point of view, are clearly willing to accept the risks related to your tool. -snip- Allowing someone you do not know to change executable files on your computer is always a Very Bad Thing (or at least a Very Risky Thing). No matter how awesome and thorough that person may be, it can end badly.
That's one of the reasons all the code is open source. If anyone is worried about security, they can look at it. If anyone has any suggestions, they can try them out and send them to me to be added (or ask to be given subversion access).
 
Quote
It really is a shame, though. The underlying thing that you are trying to here is actually pretty cool. But even an excellent idea can be ruined if it is handled badly. If you decided to approach all this in a *much* more responsible manner, I do think you could address almost all of the concerns mentioned here. There are still going to be a fair number of unavoidable risks here no matter what you do, but taking appropriate steps to reduce the danger of those risks (and thereby doing whatever you can to protect your end users from bad results) is the very best way to ensure that your project is successful.
You've harped on this point for most of your posts, but what steps do you suggest be taken? I've pointed out what steps I have taken, up to (and including) the best general error-catching available in C++. If you have a better method, please let me know.
Hell, if you feel so strongly about it, I'll add to you to the project (as long as you promise not to edit every page to start with "don't use this!" :P) or we can go over the code and discuss each problem you see. I'm completely willing to try and improve the project as much as possible.
Another thing you need to consider is this is still alpha. It's only a few months old at this point, and I just recently finished implementing the basic blocks. This latest release was the first one with a major focus on anything but feature-adding, and is the first one that's not just a proof of concept. If you see so many issues, help me fix them.

Edit: Apologies if the above seems rude, I was on the way out to mow the lawn before the rain started.
However, while mowing I got a spark of inspiration that may, if it works out, let me avoid modifying any executables, which will lay all EULA questions to rest. It'll also render a few of your concerns null, and won't cause me any extra trouble. I'll be testing that out tonight.
Now, another thing, I'm not entirely familiar with the NWN community yet. I'm pretty new here, this is my first project for NWN. So I may be misjudging. In the community/game I'm from, and where most of my experience is from, it's a common question to ask "are you running MGE?" when someone mentions a crash or visual error. That mod (essentially the same as NWShader) has become an established part of the community. If I can work out the bugs in NWShader and make it both stable and safe, I can imagine it doing the same (of course, it'll probably take a similar timespan of 4 or 5 years before it fully catches on). This is all still alpha. I'm still getting it working right. At this point is it best suited for people who understand what's going on, I'm only starting to make it more user-friendly (and obviously not doing so great :P).
Because of that, feedback at this stage, from both the "yay I like!!!" and "wait, it does what?!" sides is important. As you mentioned, I can't think of everything, so I need as many suggestions and contributions as possible to get everything I haven't considered. You seem to have a lot of concerns, many of which are valid. If you have the time or are willing to help fix them, it'd be much appreciated. If not, if you could at least suggest a better way, I'll see if it can be done. :)


Title: Re: NWShader
Post by: Invisig0th on May 02, 2010, 12:13:34 PM
I have provided the warning. I can't force users to read anything, but I can provide the text. It has been available since the release of this new version yesterday, and I'm currently expanding it and making it more obvious. The warnings have been provided.
Yeah? Where?

As I write this, on the download page for NWShader on NWVault (http://nwvault.ign.com/View.php?view=Other.Detail&id=1443), there is no indication that this tool is an alpha or prerelease. There is absolutely no mention of the fact that installing the tool will change your game executable. In fact, there is no sort of warning or disclaimer *at all* on that page regarding the various potential risks or the assumed skill level of the user. At this point, the information that is currently provided on the NWVault download page would lead one to believe that this product is finished, stable, and safe.  Even the people who do take the extra time to read the download description you have up there now will be completely unaware of any potential risks. The closest thing you have to a warning is that "NWShader is under rapid development". That doesn't exactly cover it. That could simply mean that you're working on version 2.0.

So if you honestly think you're taking the appropriate steps to provide people with good information about the pros and cons of what they are downloading, I'm sorry to say that you really haven't. If you have posted any warnings on that page, I certainly can't see them -- and if I can't see them when I'm specifically looking for them, then it's safe to assume that most people won't see them.  As a general rule, people simply aren't going to go to your web site or surf your Wiki before installing a game add-on, no matter how much you beg them to.  You need to provide that information on the NWVault download page in a very obvious manner -- probably highlighted in the download description and also included in the documentation for your tool included in the download. It's your responsibility as the developer, particularly with experimental tools, to make sure the people considering using your tool have easy access to all the information they need to make an informed decision regarding whether or not to proceed.  Saying that it is the user's responsibility to dig around for that information is just ignoring the reality of how such things work in the real world. You will save yourself a lot of headaches by simply accepting the fact that it will be up to you as the developer to provide this information as explicitly as possible.

I appreciate the deep and sincere dedication you have to improving the code of your tool in order to reduce problems. I really, really do. But you are letting a lot of other things slip through the cracks, things that are decidedly not code related. Those things can cause lots of problems, too. Those are the things that will make or break your project. As of this writing, you simply have not addressed these issues, not even at a bare minimum.


Title: Re: NWShader
Post by: peachykeen on May 02, 2010, 01:22:20 PM
Alpha label (next to the project name):
http://sourceforge.net/projects/nwshader/
That label actually shows up on all the base SourceForge pages, including the download page.

Note on how the new method works (added yesterday at release-time, added onto today):
http://sourceforge.net/apps/mediawiki/nwshader/index.php?title=Installation

There's also a mention that the file will be changed and a warning to keep the backup safe in the install section of the readme.

Quote
I appreciate the deep and sincere dedication you have to improving the code of your tool in order to reduce problems. I really, really do. But you are letting a lot of other things slip through the cracks, things that are decidedly not code related. Those things can cause lots of problems, too. Those are the things that will make or break your project. As of this writing, you simply have not addressed these issues, not even at a bare minimum.
You keep mentioning this point and referring to the basics of how NWShader works. I've already said I'm updating the docs to add these warnings (which weren't an issue before the latest release, roughly 24 hours ago).
Past that, what are these things that are such a problem in your view?
I can't fix them unless you point to specifics (other than warning people to hang onto a backup file). In addition, the entire project is open source. If you feel so strongly that it is poorly done and needs fixing, then help fix it! :)

Edit: I've added a note to the Vault page. That means that the main download page clearly states the project is alpha and the readme, instructions, and Vault link (which sends you to the SF download page or wiki) all have notices.
I also noticed that you skipped over the potential fix I suggested in one of my posts, mentioning making a copy then modifying that. You point out flaws with the project, but have yet to discuss real solutions. I'd like to find a way to fix these issues, instead of just going over the fact they're there.


Title: Re: NWShader
Post by: Invisig0th on May 02, 2010, 05:30:44 PM
Well, you just got done claiming that you couldn't think of any possible additional way to notify people that you are changing the nwmain.exe file, even though you had literally mentioned nothing at all about any of this on the NWVault download page. I understand that you are frustrated, but I think we can also agree upon reflection that you had hardly exhausted all options.

In all seriousness, do you honestly not understand the value of putting important information like this on the NWVault download page? Or do you see some benefit to not posting that information? Do you not believe that some people would get very upset with you if you did not take reasonable steps to inform them that their critical game files will be altered by your tool? These things seem completely obvious and self-evident to me. 

Regardless, thank you very much for mentioning on the download page that the NWShader replaces the nwmain.exe file. That's definitely a step in the right direction. It will continue to puzzle me that you were reluctant to do this. That tells me that there's still a huge gap between what you consider necessary and important (as a programmer) and what your end users will consider necessary and important (as average NWN players, who may in fact not even be all that computer literate). Writing solid code is frequently the easiest part of a project -- it's dealing with the needs of the end users that is generally the most difficult. Based on this discussion, it's likely that this is going to continue to challenge you throughout this project. But at the end, I guarantee you that you will be glad you made that journey. It is astounding difficult for a programmer to see things from the point of view of an end user, but the most successful programmers are the ones who never stop trying to do so.

As for implementing alternative approaches -- when you upload a different solution, I'll be happy to give your tool another look. I've already taken time out of my schedule to give you exactly what you had asked for here -- detailed feedback, based on my considerable knowledge of the NWN community and past NWN projects. I'm under no obligation to solve fundamental design problems for you, but I promise you I will do my best to try to offer useful suggestions as development progresses.  You're clearly a smart guy and open to learning new things, so I have no doubt this project will be successful.


Title: Re: NWShader
Post by: peachykeen on May 02, 2010, 06:01:55 PM
Well, you just got done claiming that you couldn't think of any possible additional way to notify people that you are changing the nwmain.exe file, even though you had literally mentioned nothing at all about any of this on the NWVault download page.-snip-
In all seriousness, do you honestly not understand the value of putting important information like this on the NWVault download page? -snip-
Actually, I'd just forgotten to add it to that page. Between being the documents-reading sort myself and knowing exactly what's going on here, I missed there. I'd like to have someone helping handle documentation, but so far I'm the only one on this (I did have some help with getting the wiki setup), so I have to keep track of everything and split my time. Especially around release times, I'm sure you can understand how that goes.

Quote
Regardless, thank you very much for mentioning on the download page that the NWShader replaces the nwmain.exe file. -snip- Based on this discussion, it's likely that this is going to continue to challenge you throughout this project. But at the end, I guarantee you that you will be glad you made that journey. It is astounding difficult for a programmer to see things from the point of view of an end user, but the most successful programmers are the ones who never stop trying to do so.
Not reluctant, just forgot to (the Vault page is mainly a link page, so it's the last one I update after the project pages). The whole reason I started this topic was to try and get in touch with more users, so I guess it's working (albeit in a somewhat different manner than I was going for :P).

Quote
As for implementing alternative approaches -- when you upload a different solution, I'll be happy to give your tool another look. I've already taken time out of my schedule to give you exactly what you had asked for here -- detailed feedback, based on my considerable knowledge of the NWN community and past NWN projects. I'm under no obligation to solve fundamental design problems for you, but I promise you I will do my best to try to offer useful suggestions as development progresses.  You're clearly a smart guy and open to learning new things, so I have no doubt this project will be successful.
Just to be clear, I'm not asking for you to solve the problems, just point out any you see that I miss and if an idea comes to you, let me know (if it's no trouble). Any outside views are helpful, and I'm trying to get as many as possible. You understand how programming goes, and those times when something looks so clearly like the best solution (but you missed something important). This particular way of hooking was one of those, it's wicked simple code-side and guaranteed to be stable. I didn't anticipate players having a complaint with the whole patching bit. :facepalm:
I've done some work on a new method, which looks like it might work, if the engine cooperates. It's a bit more unstable (more prone to infinite loops, in particular), but doesn't modify anything (just drop the files in). I'm going to have to try and get it working and iron out the new issues if I can.
Thanks for your offer of help. I'll let you know when/if I have a different way working. :)


Title: Re: NWShader
Post by: Invisig0th on May 02, 2010, 07:03:44 PM
Sounds good. Will do.

If it turns out you can avoid replacing the main program executable, I think that will clearly be a better solution for lots of reasons.

Also, I probably shouldn't have used the word "reluctant" there, because overlooking something is very different than avoiding it. My apologies for the poor wording.


Title: Re: NWShader
Post by: peachykeen on May 03, 2010, 06:59:06 PM
Aright. After trying a couple different way (preprocessor definitions for standard calls are a great thing, easy to change), I think I may have pinned down a working one. I also need to even out my hair to cover the missing spots... :P

I tried dropping an OpenGL32.dll into the NWN directory, which failed. NWN refuses to see the wglCreateContext from my DLL (creates the window and starts OpenGL in it). So you end up with a black screen. The function can be exactly the same as the Windows one, it still won't use it. I have absolutely no idea why, and a thorough run-through with IDA Pro (which I have loaded with the full debug symbols for the client, so it's the next best thing to having the source) couldn't tell me a thing. I had to scrap that. It was also prone to (easy to avoid) stack overflows and needed a huge number of dynamic functions (very time-consuming to code).

I was experimenting with other methods, and eventually took another look at NWN's import table. Now, it generally uses OpenGL, but for the mouse/keyboard, Direct Input is used. It just so happens that the Dinput.dll has 1 function imported. Only one, compared to the 80+ in OpenGL32.dll (over 170, when you consider Cg and GLEW, which NWShader needs). Not only that, but by dropping a DInput in, there's no chance of NWShader causing any infinite loops and it doesn't sit directly between the game and the drivers (better for a quick exit).

So, at this point, I have a solution that appears to work, and involves extracting a couple DLLs into your NWN folder.
No install, no hook, no patch, nothin' sketchy. You extract and play. Don't want it, delete them (or just rename them to test something).

The only similarity it has to the other method is using Detours to attach to a few functions in memory (NWShader tells Detours to hook them when the DLL starts, and whenever NWN calls a couple OpenGL functions, Detours sends them to NWShader and then on to the system). No files are modified, just a few DLLs added.

Not only that, but this method allows NWShader to completely detach itself at runtime (even a script could tell it to shut down) and I've already noticed better error prevention (a couple mistakes I made while coding were logged and NWShader removed itself, which honestly took me completely by surprise and was pretty awesome).

I can provide an example build if you want to see it in action, but (if this solution pans out to be as stable as the other), does that take care of your concerns? (both legal and file-wise)


Title: Re: NWShader
Post by: Tybae on May 04, 2010, 08:22:57 AM
First off, Welcome to our forums.  It is nice to see new people popping in who aren't spam bots.  ;)

Second, the forums are not very active for a few reasons.  First being we are in the middle of a cycle and when we are judging work, playtesting, etc., we don't often get the time to post in the public forums.  Second being I have been out of the loop for the last month or so due to RL issues that will be over soon, so I'll be able to stoke the forum fires from time to time.

Third, as far as the EULA goes, each potential violation of Bioware's EULA lies solely with Bioware's legal team and we do not want them getting involved.  There's a reason the employees call Legal "The Sleeping Dragon".  Also, the EULA is a very touchy subject which can, and has, resulted in emotions running high.  While this has not happened in this thread, I just want to make sure that we continue to be civil to everyone (I know it seems obvious, but it needs to be said).  On a personal note, I really like the healthy conversation going on here and the fact that no one has taken anything personal up to this point. 

Also on a personal note, I really like the idea behind NWShader and wish you success. 


Title: Re: NWShader
Post by: peachykeen on May 13, 2010, 03:35:52 PM
Thanks for the welcome. As far as EULA issues go, I think those have all been settled in the latest revisions. I'm testing a new version now with an install-free system, seems to be working fine. I added partial support for translations while doing that, so hopefully I can get this localized for all NWN's languages pretty soon. Between optimizing some things, I've also overhauled the material system, which makes for some nice looking screenshots and videos. Here's a brief example:
http://www.youtube.com/watch?v=8pC7IgFsti8

Testing is going on for the new version and I'll have a complete guide on how to update from all previous versions. Hopefully any future issues can be avoided, it looks like this solution is both stable and simple. I've done a bit to simplify things this time around, and will keep working on speed and simplicity with future updates.