That's some good testing there, Lou



  • So we're on the final push for a Release Candidate, and testing is going fast and furious! One of the bugs was assigned to me, as it had to do with the proper sensing of fired bursts at targets. In this case, oddly enough, it was one specific target in one specific training exercise. This seemed odd, since the code was generic and knew nothing about the circumstance in which it had found itself. The fix actually turned out to be quite easy, the problem was that the targets were uphill and any missed rounds were flying out into oblivion, resulting in the code waiting for them to expire before telling you if you missed or not, and by how much. Like I said, an easy fix once we figured out the circumstances behind it.

    However, this was the third group of targets in the training exercise. The second group of targets consisted of two moving trucks. The first of which, which is closest to the player vehicle, comes flying over a hill fast enough to actually get airborne, then makes a hard left. This results in said truck doing one of two things:

    • An epic barrel roll.
    • Driving on two wheels for a bit in a surprising show of skill.

    All of us (engineers) had a good laugh at this, but the more sobering fact is that there is no issue written up for this behavior. How they have missed it up to this point, I do not know, especially considering it happens like 30 meters right in front of the training crew. This is mildly concerning to us.

    Things we also noticed that they've missed, so far, which we caught in the first 15 minutes:
    • Dead vehicles still make their engine sounds.
    • The previously mentioned truck target.
    • Running any of the laser boresight training with the grenade launcher results in a hard crash at startup.
    • The management application shows the wrong user instructions for the laser boresight training on all weapons.
    • Selecting light amplification, then selecting binoculars and deactivating the light amplification puts the view in the wrong mode.
    • It's possible to laser the sky.
    • The loading screen for all exercises says, 'This is a placeholder'.

    Things they've written up today, after 8 hours of testing (there may be more, but this is what we heard about):
    • The (visible) laser is visible in all views, by all stations, at night time (Dafuq? It's a 'visible' laser!).
    • There is a tooltip in the management app that is missing a period at the end.

  • 🚽 Regular

    @CodeNinja said:

    The loading screen for all exercises says, 'This is a placeholder'.

    In my experience, those ones are the worst, because in my last few jobs, getting people to write copy tends to take forever no matter how short it is. We had a release go over by a week because there were days of debate over certain copy in a FAQ page. I wish you luck and hope your experience is different from mine.



  • @RHuckster said:

    @CodeNinja said:
    The loading screen for all exercises says, 'This is a placeholder'.

    In my experience, those ones are the worst, because in my last few jobs, getting people to write copy tends to take forever no matter how short it is. We had a release go over by a week because there were days of debate over certain copy in a FAQ page. I wish you luck and hope your experience is different from mine.

    Well, the problem in this case isn't so much that we're waiting on someone to make the loading screen, or even approve it. The reason it wasn't yet written up is that the test team, "Thought that was fine". They never took the time to look at the requirements, or even bring it to a manager's attention. They just outright made a decision about what was OK and what wasn't.

    We've got an artist working on it now, from home. She says it'll be ready tomorrow. I'm looking forward to it, the one they did for one of our other projects is actually quite nice.


  • @CodeNinja said:

    They just outright made a decision about what was OK and what wasn't.
     

    If they hadn't spotted the ones you did, would you still be blamed for their misses? Just curious where accountability lies.

    But yeah, it sounds like your QA team isn't of high quality.



  • @CodeNinja said:

    airborne-epic-barrel-rolling-two-wheel-driving truck
    Maybe they thought this was intentional? It sounds like something I'd like to see.



  • @CodeNinja said:

    The second group of targets consisted of two moving trucks. The first of which, which is closest to the player vehicle, comes flying over a hill fast enough to actually get airborne, then makes a hard left. This results in said truck doing one of two things:

    • An epic barrel roll.
    • Driving on two wheels for a bit in a surprising show of skill.
    Are you by any chance working on Arma 3? I remember that in Operation Flashpoint and Arma 2 the physics were kind of funny and paper-model-like, and enabled this kind of behavior almost regularly.


  • @Zecc said:

    It sounds like something I'd like to see.

    go play around in Operation Flashpoint, you can also see/make an M1A1 tank basically ski off a hillside in there. It's positively fun for a while.



  • @Cassidy said:

    @CodeNinja said:

    They just outright made a decision about what was OK and what wasn't.
     

    If they hadn't spotted the ones you did, would you still be blamed for their misses? Just curious where accountability lies.

    But yeah, it sounds like your QA team isn't of high quality.

    Yep, we'd be on the hook for it. In fact, we got a stern talking too when, for some reason, they decided to completely re-install the software on all the computers in one of the mobile units even though it had auto-deployed the following evening. Apparently we didn't tell them they didn't need to do that, even though they've not had to do that for something like a year now...


    @SEMI-HYBRID code said:

    Are you by any chance working on Arma 3? I remember that in Operation Flashpoint and Arma 2 the physics were kind of funny and paper-model-like, and enabled this kind of behavior almost regularly.

    Nope, but we are using PhysX, and I think Arma uses it too. They have a newer version than we do, I think we're still on API 2.something, which has some rather interesting oddities when you try and simulate vehicles. Talk around the office is we may transition to Havok, but I'm not holding my breath.


    @Zecc said:

    @CodeNinja said:

    airborne-epic-barrel-rolling-two-wheel-driving truck
    Maybe they thought this was intentional? It sounds like something I'd like to see.


    It is pretty awesome. :) Especially since it's not something we'll be fixing in software, instead we'll just tell the exercise developers to reduce the vehicle speed or the angle of the turn. Toyota pickups with machine guns on them don't usually turn 90 degrees at 50mph.



  • @CodeNinja said:

    Yep, we'd be on the hook for it.
     

    Hence why there's no pressure for them to do well - not held accountable for their failures.

    @CodeNinja said:

    Apparently we didn't tell them they didn't need to do that, even though they've not had to do that for something like a year now...
     

    Were they told they DID need to? Or is this one of those "assume they're aware of the exception".



  • @CodeNinja said:

    Toyota pickups with machine guns on them don't usually turn 90 degrees at 50mph.

    This is not what I expected



  • @Cassidy said:

    Were they told they DID need to? Or is this one of those "assume they're aware of the exception".


    I'm not actually sure of what they were thinking. They'd been testing the system for almost a week at that point, and every night it had automatically installed. For some reason, they just decided to do it again manually that day, and when they got called on it they said it was because we never told them they didn't need to do it. The only reason they got caught is because our manager caught them hanging out in the high bay playing ping-pong all morning.



  • @CodeNinja said:

    For some reason, they just decided to do it again manually that day, and when they got called on it they said it was because we never told them they didn't need to do it.
     

    .. and someone believed them, so blame was shifted.

    There's a big cultural cockup going on there: misunderstanding of their responsibilities (by them and others) leading to easy blame divert.

    OTOH, it's a great source of more WTFs to report.

    Doesn't make your job any easier, though. Oddly, it's usually the other way around - testers are blamed (not thanked) for finding failures, holding the deployment back (i.e. villified for actually performing their responsibilities well).

    I can understand not being effective at testing because people don't want to be told that defects exist, but to actually perform weak testing and then blame coders for QA signing off bug-ridden products is a major WTF.



  • @CodeNinja said:

    Nope, but we are using PhysX, and I think Arma uses it too. They have a newer version than we do, I think we're still on API 2.something, which has some rather interesting oddities when you try and simulate vehicles.

    do tell. btw, shouldn't stuff like that be mainly an issue of wrong settings of global gravity and/or object weight? Or just scale of the models/world?



  • Maybe they lack suspensions.



  • @SEMI-HYBRID code said:

    @CodeNinja said:
    Nope, but we are using PhysX, and I think Arma uses it too. They have a newer version than we do, I think we're still on API 2.something, which has some rather interesting oddities when you try and simulate vehicles.

    do tell. btw, shouldn't stuff like that be mainly an issue of wrong settings of global gravity and/or object weight? Or just scale of the models/world?




    It's just, odd... I'd love to go back and rework our implementation of PhsyX vehicles because I think it's wrong. For example, engine speed is a function of the vehicle velocity, not the other way around. This was, however, how the PhysX documentation said to do it, which just seems strange to me. I think the main issue we have is that there are quite a few variables that are poorly documented as to what they actually do. The fact that we haven't spent a lot of time fine-tuning the vehicle settings doesn't help a lot. It also seems like there is something functionally broken with how the transmissions, engines, and suspension work. Supposedly the newer versions of PhysX has better vehicle simulation.



  • @CodeNinja said:

    It's just, odd... I'd love to go back and rework our implementation of PhsyX vehicles because I think it's wrong. For example, engine speed is a function of the vehicle velocity, not the other way around. This was, however, how the PhysX documentation said to do it, which just seems strange to me.

    Excuse my ignorance, but why should PhysX care about engine speed? That's an implementation detail. PhysX shouldn't care whether your truck is powered by an internal combustion engine or by unicorn farts, so long as it obeys the laws of physics!



  • @ekolis said:

    @CodeNinja said:
    It's just, odd... I'd love to go back and rework our implementation of PhsyX vehicles because I think it's wrong. For example, engine speed is a function of the vehicle velocity, not the other way around. This was, however, how the PhysX documentation said to do it, which just seems strange to me.

    Excuse my ignorance, but why should PhysX care about engine speed? That's an implementation detail. PhysX shouldn't care whether your truck is powered by an internal combustion engine or by unicorn farts, so long as it obeys the laws of physics!


    Perhaps to provide information to the audio engine about what kinds of sounds to make? Only thing I can think of for why they'd need the engine speed.



  • Engine speed is most likely used as input to the sound engine to vary the pitch of the engine noise of the vehicle. Also, obeying the laws of physics is really a matter of degree... the more detailed and accurate the simulation, the longer the simulation takes to run, and you have a hard limit of 1/30 seconds at the very most to do everything you need to do before you start it all over again for the next frame. Most everything in most games is just faked to "look right" anyway, because that's usually what really matters in the end. Also, 3-D physics is surprisingly hard to make mathematically stable. Letting the velocity of the vehicle drive the engine speed may be more stable than the other way around.



  • @ekolis said:

    @CodeNinja said:
    It's just, odd... I'd love to go back and rework our implementation of PhsyX vehicles because I think it's wrong. For example, engine speed is a function of the vehicle velocity, not the other way around. This was, however, how the PhysX documentation said to do it, which just seems strange to me.

    Excuse my ignorance, but why should PhysX care about engine speed? That's an implementation detail. PhysX shouldn't care whether your truck is powered by an internal combustion engine or by unicorn farts, so long as it obeys the laws of physics!





    Well, the engine contains data that defines it's power (torque) at varying RPMs, which is fed through the transmission to determine how much force is applied by the wheels to move the vehicle forward. We also use that data to vary the engine sound. The biggest problem we have right now is that they want the PhysX representation of the vehicle to behave exactly as the real vehicle, so we're actually sitting here with stopwatches and charts (Vehicle X should reach Y mph in N gear after S seconds), if it's not within a half second of that we have to tweak it. Problem is that it's cumulative, so if you tweak one to match the data point, the others shift around.

    It's mostly an implementation thing on our end, but I do know that PhsyX 3 added a bunch of new stuff for making vehicle simulation easier, and I wish we had that. PhysX wheels in 2.x tend to launch themselves into the air when driving over curbs (M1114s doing backflips is funny, but not desired).

    Sadly, since we're on 6-9month product dev cycles, we haven't had any time to really fix the problem and instead just cludge it as best we can. Rumor has it that we're going to switch to a different physics engine at some point, so maybe we'll get time to implement that correctly (not going to hold my breath).


  • Considered Harmful

    @CodeNinja said:

    The biggest problem we have right now is that they want the PhysX representation of the vehicle to behave exactly as the real vehicle, so we're actually sitting here with stopwatches and charts (Vehicle X should reach Y mph in N gear after S seconds), if it's not within a half second of that we have to tweak it.

    I'm actually impressed that you bothered. I thought the only games that cared about that were simulators like Gran Turismo.



  • @CodeNinja said:

    The biggest problem we have right now is that they want the PhysX representation of the vehicle to behave exactly as the real vehicle, so we're actually sitting here with stopwatches and charts (Vehicle X should reach Y mph in N gear after S seconds), if it's not within a half second of that we have to tweak it. Problem is that it's cumulative, so if you tweak one to match the data point, the others shift around.

    ...shouldn't something like this "just work" if you define the parameters (such as engine power, gear ratios, etc.) correctly/according to official vehicle specs?



  • @joe.edwards said:

    @CodeNinja said:
    The biggest problem we have right now is that they want the PhysX representation of the vehicle to behave exactly as the real vehicle, so we're actually sitting here with stopwatches and charts (Vehicle X should reach Y mph in N gear after S seconds), if it's not within a half second of that we have to tweak it.

    I'm actually impressed that you bothered. I thought the only games that cared about that were simulators like Gran Turismo.

    If we were making a game, then we would be able to get away with it, but this is a simulator. Also, I wished it looked as good as Gran Turismo! That'd be epic. Our single Graphics guy tries hard, but the first feature to get cut is always the 'pretty'.


    @SEMI-HYBRID code said:
    @CodeNinja said:
    The biggest problem we have right now is that they want the PhysX representation of the vehicle to behave exactly as the real vehicle, so we're actually sitting here with stopwatches and charts (Vehicle X should reach Y mph in N gear after S seconds), if it's not within a half second of that we have to tweak it. Problem is that it's cumulative, so if you tweak one to match the data point, the others shift around.

    ...shouldn't something like this "just work" if you define the parameters (such as engine power, gear ratios, etc.) correctly/according to official vehicle specs?


    I wish! Plugging in 'realistic' values will get you a starting point, but that's about it. For example, and this is even in the API documentation we have for the version of PhysX we're using, it's recommended to place the vehicle Center Of Mass under the ground to keep it from flipping over at low speed during turns. We really need to hire someone just to do vehicle physics, I thought we had, but apparently he's an ex-Kennedy Space Center guy and he's apparently stated that PhysX can't properly simulate a vehicle, so he can't do it. I get the impression his background in simulation would result in a perfectly simulated vehicle that took 12 hours to simulate, but would accurately account for the wind resistance of all the bolts and various pieces of bodywork. Since we need to maintain at least a 30fps minimum on 3+ generation old PC hardware, that's not going to happen.


Log in to reply