Altoona & bust: RSC carelessness

[Chacal]

Warning:

Perhaps we should align our expectations with reality.

I work with development teams year-round. This for critical software that, if broken, can lead to loss of life, loss of reputation, loss of public confidence, loss of business. Project budgets measured in millions, time to delivery measured in years.

Professional teams, comprised of architects, analysts, programmers, project managers. Separate teams for development and for QA. Elaborate processes for code promotion and deployment. Unit tests, functional tests, integration tests, system tests, acceptance tests, performance tests, load tests, penetration tests. Using integrated development environment, with automated tools for performing thousands of tests as part of the build process.

And yet, with all that, they produce buggy software. Ridden with incredibly obvious bugs or even design flaws.

And as I trace those to their origin, I realize again and again that software is hard to do. And hard to test.
In particular, writing test cases is incedibly hard to do. WIth complex software, there is an infinity of permutations of all possible factors that can lead to unexpected behavior. Not easy to predict.

Now imagine you write a train simulator game. Most of it is actually making content, there's not much software programming that can be tested with automated tools. You have almost none of the above incentives, teams, budget, time, or tools. You have a QA team and beta testers, about the only thing they can do is sit down and play the game. You fix whatever bug they report, if it is fixable within the limits of the game engine.

Now imagine you're a 3rd-party content builder and you've just built an add-on for a train simulator game. You do that as an aside, a hobby that may bring home a few extra bucks. You have nothing but time. You're a builder and don't know a lot about scenarios, but the game company wants some. You do what you can and send the product to the game company. QA teams and beta testers will sit down and try the add-on with your scenarios. The game company will sell that for 20 bucks (most people will buy it on sale for 12 bucks), which will be split between the builder, the game company, Steam and the credit card company.

Just how much QA are we expecting for that kind of product?

[buzz456]

Please refrain from introducing any logic into this conversation.

[mrennie]

Warning:

Perhaps we should align our expectations with reality.

I work with development teams year-round. This for critical software that, if broken, can lead to loss of life, loss of reputation, loss of public confidence, loss of business. Project budgets measured in millions, time to delivery measured in years.

Professional teams, comprised of architects, analysts, programmers, project managers. Separate teams for development and for QA. Elaborate processes for code promotion and deployment. Unit tests, functional tests, integration tests, system tests, acceptance tests, performance tests, load tests, penetration tests. Using integrated development environment, with automated tools for performing thousands of tests as part of the build process.

And yet, with all that, they produce buggy software. Ridden with incredibly obvious bugs or even design flaws.

And as I trace those to their origin, I realize again and again that software is hard to do. And hard to test.
In particular, writing test cases is incedibly hard to do. WIth complex software, there is an infinity of permutations of all possible factors that can lead to unexpected behavior. Not easy to predict.

Now imagine you write a train simulator game. Most of it is actually making content, there's not much software programming that can be tested with automated tools. You have almost none of the above incentives, teams, budget, time, or tools. You have a QA team and beta testers, about the only thing they can do is sit down and play the game. You fix whatever bug they report, if it is fixable within the limits of the game engine.

Now imagine you're a 3rd-party content builder and you've just built an add-on for a train simulator game. You do that as an aside, a hobby that may bring home a few extra bucks. You have nothing but time. You're a builder and don't know a lot about scenarios, but the game company wants some. You do what you can and send the product to the game company. QA teams and beta testers will sit down and try the add-on with your scenarios. The game company will sell that for 20 bucks (most people will buy it on sale for 12 bucks), which will be split between the builder, the game company, Steam and the credit card company.

Just how much QA are we expecting for that kind of product?

This post should be made compulsory reading.

Thanks Chacal.

[Toripony]

Good points made. I saw plenty of amazing bugs in medical software, too. It mostly has to do with testing and only the end users will eventually test *everything*. To answer the last question posed, I expect more than one fast run-through (5-15 mins using Async x5) for the $12 scenario that they're going to sell a few hundred copies of. In this particular case, start the scenario, go get lunch and come back to see where all the trains ended up. Some of this stuff is not that hard if "artists" had a mind for "systematic development"... which most don't. Personally, I feel blessed and cursed with half of each mind but not a full mind at all.

HOWEVER.....

Please refrain from introducing any logic into this conversation.

[artimrj]

I made the scenarios for the California Zephyr and the Big Boy/ Challenger. I ran each one at least 30 hours myself. That's 10 scenarios. Then they went through the beta team. 1 Big Boy Scenario and 1 Zephyr scenario did not make it out. One of the ones that did had a AI crashing into a siding consist that it did not do if the player train was passed a certain spot at a certain time. Either going too fast or too slow made it happen. It never happened to me in my testing. Or any of the beta testers.

What can I say?

[Toripony]

A confounding-ly funny example there, Bob! I think one of the premises of testing is to *try* to break something. I guess I got a lot of practice at this managing a computer network among (in the 90's) mostly computer-illiterate healthcare professionals. I spent a lot of time trying to think of what the users could do wrong and then designing a way to prevent that from happening. It was all about *not* using it the way I would. Doctors seemed to be able to break anything, though.

Maybe it's just me, but it seems like there are even fewer markers in RSC's routes than ever. I put markers in my route at every possible "route" that AI could choose and then I instruct them exactly which route to take at almost every junction. It's overkill, and probably limits my scenarios but I tend to like to control precisely where AI is going. I don't trust computers... can you blame me?

[CrimsonKing]

Maybe it's just me, but it seems like there are even fewer markers in RSC's routes than ever. I put markers in my route at every possible "route" that AI could choose and then I instruct them exactly which route to take at almost every junction. It's overkill, and probably limits my scenarios but I tend to like to control precisely where AI is going. I don't trust computers... can you blame me?

There definitely appears to be fewer markers on RSC routes but that doesn't stop scenario creators from adding their own. Few creators add their own markers because the overall content creation documentation is definitely lacking even though it is one of the biggest selling points for the game.

Unfortunately, even when one adds markers, there is no guarantee that the dispatcher will route the trains correctly if the player is too fast or too slow.

[OldProf]

Despite all of these comments, I'm sticking to my guns. In the scenario I cited originally, that AI train was programed to run through a siding where a rack of freight cars was parked. No testing was actually needed: just common sense. Whoever wrote the scenario both placed the freight cars on the siding and ordered the AI train to drive into them. The player train never got anywhere near the siding in question, so the player's rate of speed could not possibly cause the crash except in the sense that faster driving on the player's part would lead to completing the scenario before the crash occurred.

I am not and would not call for perfection in anything, much less a train simulator. I'd just like to see some competence and common sense. Although that is probably too much to ask, not asking will only lead to more of the sloppy work that guarantees incompetence and lack of common sense.

[buzz456]

Definitely getting a grumpster award.

[Ericmopar]

Some people have figured it out, because Elphaba makes superb scenarios. It's getting to the point, where I only want to download certain authors scenarios from the Workshop, because so many of them are so bad. One of the interesting things about Workshop scenarios, is this. Even if the person gets the AI scheduling etc right, they frequently have other things broken, like the light for the time of day, or trees in the middle of the engine at Essex owned by the hotel.

[Toripony]

I've been thinking... if we take a simulation view of Tom's experience, wouldn't he be getting some lashes from the roadmaster for completing his assignment so slowly? Taking it further, if it could be proven that his poor job performance indirectly caused the other train to crash, he's getting the boot! The scenario-writer could claim that this was his thinking when he plotted that AI's course.

[Chacal]

As long as we can't schedule AI trains based on events, scenario writing will be very difficult.
Presently we're trying to fit the player's unpredictable behavior into time-tabled behavior. This makes no sense.

I'm trying to complete the "Run this town" scenario that came with the GG1 DLC. It is impossible to complete from the start, because the author didn't take into account the time needed for the nifty camera animations at the start, so even when flooring it I can't make the first time-tabled instruction on time. "Flooring it" being an euphemism as the GG1 isn't known for its heart-stopping acceleration.

So I started lowering the performance percentage for timetabled instructions, and at the end of the scenario I end up with a switch locked by an AI train waiting miles away. Am I arriving too early, or too late, or is the AI stuck because of a bug? The case is so complex I can't start figuring it out. How can I possibly test this, let alone fix it? If I change anything, all the AI trains' timings get out of whack.

[Toripony]

I've gotten stuck needing to adjust that performance percentage and couldn't for no apparent reason. Caused me to completely change my scenario. I agree that an event based trigger would help a lot. I wonder if anybody put that in DG/RSC's "suggestion box" thread? I read that thread just before they closed it but don't remember seeing that idea.

[Chacal]

I may have a hint for you. I don't know if you already know this.
At first I just changed the percentage value in the instruction form in the timetable editor, and waited as the editor seemed to recalculate the times.
I failed to notice that the times did not change, they were NOT really recalculated, and they were still displayed in red, which means the arrival time is impossible to attain.
When I noticed this, I unchecked and then checked again the "timetabled" box, and the times were correctly recalculated, and no longer displayed in red.

[Toripony]

Good tip! I'll give it a try.