Auto-extracting files from AP archives for use in re-skinning

[mikesimpson]

Dave Dewhurst has posted a great tutorial for re-skinners showing how to automatically extract various files from .ap archives for use when you distribute your new skins.

With his kind permission, I have converted the .html file into a .pdf file and placed it on my site at http://www.rstools.info/tutorials/Auto_ ... _files.pdf

Mike

[OldProf]

Thanks, Mike, at the least for giving me another reason not to attempt re skinning.

Why did RSC introduce this AP business, anyway? It obviously causes a lot of headaches, but is there an up side to it?

[Chacal]

Technically, it makes it easier for Steam to validate the local files. Validating thousand of small individual files probably brought lots of problems, not the least, as I remember, the necessity to validate again a few times until all files passed the test.

Conceptually, it makes it easier to apply mods and repaints to the game and to revert back to original files.
If correctly implemented, that is.

The idea was this:
1- When the game engine tries to open a file,
- if the file exists only in unpacked form (as before TS2014), the game uses this;
- if the file exists only in packed form (inside a .ap file), the game uses this;
- if the file exists in both packed and unpacked form, the game uses the unpacked one.

2- Reverting back to original files is easy: delete unpacked files.
3- Distributing mods is easier and safer because you never have to modify original files.

Experienced game hackers are familiar with this mechanism,used by countless games since the 1990s. For example games using Python scripts instead of lua: this mechanism is built right into the Python interpreter.

That is the theory behind it. Sadly there seems to have been implementation problems for TS2014. I don't know if this has been fixed or not.

[Chacal]

Now implementing such a change to an already existing program (TS2013) can lead to problems.

The best way of doing this is known as "shimming", i.e. inserting a logic module (shim) between the game engine and the file system.
In a well-built program using modular architecture, this shim should already exist anyway because game logic should never talk directly to the file system (1).

file logic.jpg

WIth this architecture, the game engine tells the file logic shim to fetch a file, wherever it is. The file logic determines which file to fetch and returns it to the game engine. This way, the game engine doesn't need to know about .ap files, and doesn't need to be modified at all.

I would encourage Mike Simpson to use the same concept for RW-Tools, getting rid of all the special .ap file functions in the menu and making all of this transparent to the user.



(1) Whether or not this is compiled into the same executable is not important

[mikesimpson]

Unfortunately, Chacal, I only program in Visual Basic 6 and the time taken to extract any files from an .ap file, e.g. in the case of checking every tile in a route to search for missing .bin files, would take for ever (sometimes the check route option has to check some hundred thousand files) and the users would no longer use my program. Writing a complete unzip program to extract required files is quite complex and as I am now in my 70s, I have neither the time nor inclination to do so.

What has not been pointed about with regard to the .ap files, is that just to make a simple bug fix to a route, Steam then redownloads the whole route, in some cases 1.5 gigs of .ap file. Maybe OK for U.S. users, but there are still a lot of people out there who pay for their downloads by the gigabyte.

My personal opinion is that the real reason RSC adopted the .ap files (and the .out files) is to completely lock down the program so that if a user did not have particular rolling stock, then they had to buy it from RSC.

Mike

[Chacal]

Unfortunately, Chacal, I only program in Visual Basic 6 and the time taken to extract any files from an .ap file, e.g. in the case of checking every tile in a route to search for missing .bin files, would take for ever (sometimes the check route option has to check some hundred thousand files) and the users would no longer use my program. Writing a complete unzip program to extract required files is quite complex and as I am now in my 70s, I have neither the time nor inclination to do so.

Hi Mike,
Yes I understand, performance would indeed be an issue with VB6.
Writing an unzip program would indeed be out of the question, I was thinking more about integrating with a free zip library such as Mark Nelson's OCX for ZLib, but I don't know if that is feasible with VB6. On the other hand, as you say it may not be worth the effort.
RW-Tools is already tremendously useful without this.

[mikesimpson]

Hi Mike,
Yes I understand, performance would indeed be an issue with VB6.
Writing an unzip program would indeed be out of the question, I was thinking more about integrating with a free zip library such as Mark Nelson's OCX for ZLib, but I don't know if that is feasible with VB6. On the other hand, as you say it may not be worth the effort.
RW-Tools is already tremendously useful without this.

Yes, I already have a set of routines for extracting / adding to zips which I use in RW_Tools in some options such as the .ap file manager routine, I might see if I can do something with this when I have time. Currently I am spending more time with my radio utilities and having a rest from trains...

Mike

[Ericmopar]

What I miss, is the former ease in which I could use RW Tools to clone any scenario and modify it with different weather, rolling stock etc. Not being able to freely do this, has actually greatly reduced my purchases from DTG. I used to love taking new rolling stock and customizing scenarios with it.

I've seen your radio utilities Mike, are you a Ham by chance? KM6YW.

[mikesimpson]

What I miss, is the former ease in which I could use RW Tools to clone any scenario and modify it with different weather, rolling stock etc. Not being able to freely do this, has actually greatly reduced my purchases from DTG. I used to love taking new rolling stock and customizing scenarios with it.

I've seen your radio utilities Mike, are you a Ham by chance? KM6YW.

Hi,
Well you can still do it if you unpack the scenario and clone it. You can then edit the cloned scenario (of course you might also have to unpack the rolling stock as well).

No, I am not a radio ham, just a short wave listener (I worked in telecommunications for around 40 years before retiring).

Mike