Bank of Australia
-
Can any of our upside down friends give us a bit more insight about this story?
The wait goes on in NAB pay glitch
NAB says it is making "good progress" on fixing the computer glitch that has left thousands without access to their money.
@Sydney Morning Herald said:
Disruption for millions of people without cash is continuing, with the National Australia Bank (NAB) still unable to say how long it will take to fix its computer glitch.
A number of branches have stayed open throughout the weekend to help people access money while "good progress" is made to solve the problem, NAB spokesman George Wright said.
The crisis began on Wednesday due to errors in NAB's data processing.
-
˙spɹɐʍʞɔɐq uı ʇxǝʇ ǝɯos pǝɹǝʇuǝ ǝuoǝɯos
-
No solid info, but I can link to a press release and provide some context. NAB (which is a listed company, not a government institution) is one of the Big Four banks in Australia, and does a lot of payment processing on behalf of clients (electronic retail transactions, payroll disbursement, etc.) and smaller banks and non-bank financial institutions (e.g. credit unions). At the moment, we only know that they had some sort of problem with overnight processing (i.e. transaction files not being processed) on Wednesday where a whole lot of payments weren't processed correctly, and that they're still trying to correct/catch up on processing. Because it happened in one of the big four (unlike, say, the US banking environment where there are thousands of small banks rather than a small number dominating the retail banking sector), if there is a problem then it tends to have very wide impact.
(Australian) ABC News reports that it was a corrupted file of some sort - and, more interestingly, a suggestion that they couldn't roll back data. Presumably any forum members who actually work at NAB and know what's happening aren't allowed to comment - but I wonder if anyone from other institutions has more information and can provide some insight?
-
@Paddles said:
(Australian) ABC News reports that it was a corrupted file of some sort - and, more interestingly, a suggestion that they couldn't roll back data.
To be honest, my first thought was the bank had collapsed and they're feeding the public a bogus story (Dutch DSB bank did something like that), but they're opening branches on sunday to help people get to their money, so I guess that's not it.
Also: @http://www.theaustralian.com.au/business/industry-sectors/nab-chaos-infects-other-banks/story-e6frg96f-1225962382944 said:
But Mr Wright denied that a computer virus or sabotage had triggered the computer glitch, despite a notice posted on the NAB website yesterday warning customers against opening hoax emails. "It wasn't a virus and it wasn't sabotage," he said. "It was a corrupt file that caused the system not to run in the normal way."
-
Ah, SQL injection then? ;)
I don't know any details of this particular issue, but I did hear on the radio this morning that other banks are now experiencing knock-on effects because they can't get transactions to or from NAB processed.
I'd expect it to be most likely a malformed EFT file combined with a lack of input-checking on their end. You might think that a major bank would validate files before feeding them in, but apparently you would be wrong. You might also expect them to be able to restore data from backups, and to be able to continue processing other transfers while the damaged one is being investigated. Of course, if they'd been able to do those things the story wouldn't have made much of an impact in the media. But there should be some interesting facts behind it - not that we're ever likely to hear them.
My company used to use NAB for our corporate accounts and payments processing, but switched to CBA a year or two ago. I think a lot of people in management are feeling rather happy about that decision at this point in time. I don't bank with NAB either, so fortunately it hasn't been a problem for me.
-
Well, to be completely honest, you wouldn't suspect anything fishy if a plain-looking citizen walked in and wanted to open an account. The name also looked normal enough: Robert Tables.
-
@Scarlet Manuka said:
I don't know any details of this particular issue,
@Scarlet Manuka said:
I'd expect it to be most likely a malformed EFT file combined with a lack of input-checking on their end. You might think that a major bank would validate files before feeding them in, but apparently you would be wrong.
Wow, you just leaped, in two sentences, from a wild-ass guess, to chiding the bank for your wild-ass guess.
-
@blakeyrat said:
@Scarlet Manuka said:
You might think that a major bank would validate files before feeding them in, but apparently you would be wrong.
Wow, you just leaped, in two sentences, from a wild-ass guess, to chiding the bank for your wild-ass guess.Or he based his wild-ass guess on what the spokesperson said:
@George Wright said:
It was a corrupt file that caused the system not to run in the normal way.
-
@b-redeker said:
@blakeyrat said:
@Scarlet Manuka said:
You might think that a major bank would validate files before feeding them in, but apparently you would be wrong.
Wow, you just leaped, in two sentences, from a wild-ass guess, to chiding the bank for your wild-ass guess.Or he based his wild-ass guess on what the spokesperson said:
@George Wright said:
It was a corrupt file that caused the system not to run in the normal way.
Ok? Was that intended to address my post in some way? It was a reply to it, I guess.
Did you have a point, or...?
-
@blakeyrat said:
Did you have a point, or...?
That you chided him for a guess that wasn't wild-assed.
-
As b-redeker pointed out, it wasn't a "wild-ass guess", it was a guess based on what the spokesperson said, and the fact that the issue affected payment transfers (so it would make sense for it to be an issue with a payment transfer file, right? I think some of the news reports I've heard did say so specifically, but I can't swear to that now) - and also on my knowledge of the EFT file formats used for bank payments processing in Australia.
For background, if anyone cares, while I don't work at a bank, I do work in the financial services industry, and one of the pieces of code I'm responsible for maintaining is the one that generates a backup EFT file from our nightly transaction processing run in case our Financials database falls over when we need to do a payment run. So, if nothing else, I at least do know what an EFT file looks like. I can imagine a few scenarios that might cause problems if the bank did not validate the input on their end before loading it. More likely, they do validate it, but the file that caused the problem had an issue that wasn't caught in the validation. So they don't validate it [b]enough[/b], I guess. ;)
I have also heard stories about the quality of software and processes used in the banking industry, the general net effect of which has been to leave me with the impression that I never want to work at a bank.
In any case the inability to recover gracefully from the problem even after a number of days surely implies some WTF-worthy failures. For one, in a sane system a malformed EFT file ought only to cause trouble for the payments within that file. It also ought to be possible to restore from backups, roll forward to just before the problem occurred, and then not load the file that caused the problem until you have corrected the issue.
If you want a wild-ass guess, here's mine. My top two candidates for the cause of the problem would be (a) a special character in the input file (bonus points for an mswin1252 em dash, en dash, or "smart quote" character), or (b) someone edited the file in Excel at some point. Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
-
@Scarlet Manuka said:
You might think that a major bank would validate files before feeding them in, but apparently you would be wrong. You might also expect them to be able to restore data from backups, and to be able to continue processing other transfers while the damaged one is being investigated. Of course, if they'd been able to do those things the story wouldn't have made much of an impact in the media. But there should be some interesting facts behind it - not that we're ever likely to hear them.
That's funny. There's another large Australian institution that is still having problems with a payment system: Queensland Health's payroll system. I know several people who were either way under- or over- paid by that system. And they've all heard stories like people being paid their payroll number, or some random amount. Issues still are ongoing.
-
@Zemm said:
problems with a payment system: Queensland Health's payroll system.
I'm not sure how that story is in any way related but it's a fantastic WTF:
Warnings by SAP, IBM and others to properly test the system were not heeded because it was considered too great a task.
Health has an incredibly convoluted payment regime: there are 24,000 ways its 78,000-strong workforce can be paid, based on 13 award structures.
This meant major defects were downgraded to minor to ensure the system could go live despite the known defects.
-
@b-redeker said:
@Zemm said:
problems with a payment system: Queensland Health's payroll system.
I'm not sure how that story is in any way related but it's a fantastic WTF
Yeah not completely related: NAB's issues appear to be "corruption of files" and WTFy backup practices. QH's problems were "let's drastically change our payroll system and do the bare minimum testing with the simplest employees (from a payroll perspective) and ignore the people like doctors and nurses who have all kinds of crazy overtime and rostering requirements - oh and other WTFy practices so no way to go back to the old system". Related in that two large organisations made mistakes that caused massive inconvenience (to use a light word) to large numbers of people.
There were several opportunities to clarify the project scope but frequent tension between Health and shared-services outfit CorpTech got in the way.
I've dealt with Shared Services Agency/CorpTech before. We'd have entire meetings debating the smallest details like the position of an icon relative to another or colour of a menu bar, but we kept getting vague descriptions of actual business requirements, which did keep changing anyway. (My company once did some development for them when they were first formed from various government departments)
-
@Scarlet Manuka said:
As b-redeker pointed out, it wasn't a "wild-ass guess", it was a guess based on what the spokesperson said,
Ok, but it's still a guess, "wild-ass" or not, and no more valid than if I said:
"I guess the reason for this outage is someone spilled coffee on their AS/400. NAB is full of retards if they kept coffeemakers on top of a AS/400 midrange! OMG WHAT MORONS! They should have moved the coffeemaker to the break room!"
If I worked there, I'd be pissed at your assumption of how stupid they are. Why don't we wait until the facts are in, *then* chide them for... whatever it is they did wrong? Assuming they did anything wrong at all? (They could be being screwed over by a vendor, through no fault of their own.)
Or go ahead and knee-jerk all over town, I guess knee-jerking is what the tech community is best at. "Chrome removed 'http://' from the URL bar! THE WORLD IS ENDING!"
-
@Scarlet Manuka said:
Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
BTW, does anybody know of an Excel-compatible strongly-typed spreadsheet program? One where, if you tell it column A is dates, it'll throw an error if you copy anything other than a date in it? I could really use something like that at work.
-
@blakeyrat said:
Ok, but it's still a guess, "wild-ass" or not, and no more valid than if I said:
I think one guess can definitely be more valid then another.
@blakeyrat said:
Or go ahead and knee-jerk all over town, I guess knee-jerking is what the tech community is best at.
I don't think that discussing possible causes of an interesting problem equals a knee-jerk reaction. I think you need to look in the mirror here.
-
@blakeyrat said:
@Scarlet Manuka said:
SSRR! It's the only software you'll ever need!Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
BTW, does anybody know of an Excel-compatible strongly-typed spreadsheet program? One where, if you tell it column A is dates, it'll throw an error if you copy anything other than a date in it? I could really use something like that at work.
-
@b-redeker said:
@blakeyrat said:
Or go ahead and knee-jerk all over town, I guess knee-jerking is what the tech community is best at.
I don't think that discussing possible causes of an interesting problem equals a knee-jerk reaction. I think you need to look in the mirror here.
Discussing it, yes.
Chiding them for being able to handle the situation you *guessed* was the cause is not fine... that's just being a jerk to everybody who works at that bank. You didn't start discussing the possible causes until after you'd already chided them for it.
-
@Xyro said:
@blakeyrat said:
@Scarlet Manuka said:
SSRR! It's the only software you'll ever need!Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
BTW, does anybody know of an Excel-compatible strongly-typed spreadsheet program? One where, if you tell it column A is dates, it'll throw an error if you copy anything other than a date in it? I could really use something like that at work.Which one? All I'm seeing is rottweilers and surgery equipment repair.
Does anybody else think we'll soon be entirely unable to communicate to other people via the use of acronyms?
-
@blakeyrat said:
@Scarlet Manuka said:
Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
BTW, does anybody know of an Excel-compatible strongly-typed spreadsheet program? One where, if you tell it column A is dates, it'll throw an error if you copy anything other than a date in it? I could really use something like that at work.
That makes two of us.
I find myself having to import data into a certain web application that is incapable of parsing a string cell into a date — if it's going into a DATE column in the database, the cell in the spreadsheet must be of type Date, or the entire import fails and gives no indication of which cell caused the problem.
One thing I've found slightly helpful: if you have a column of string cells whose values are really date strings, you can select it and go to Data > Text to Columns, which presents the same kind of interface you get when you open a CSV file — it allows a quick conversion to actual date cells based on a given date format. Of course, all the dates in selected cells need to be in the same format...
-
@blakeyrat said:
Which one? All I'm seeing is rottweilers and surgery equipment repair.
SpectateSwamp RandomRandom
-
@dhromed said:
@blakeyrat said:
Which one? All I'm seeing is rottweilers and surgery equipment repair.
SpectateSwamp RandomRandom
I registered it in <FONT color=#0e774a>acronyms.thefreedictionary.com/SSRR</FONT><FONT color=#767676> </FONT>, we should make it officially reconized, btw the funiest definition so far is "Silent Service Rod Rodriguez", how it passed site guidelines?
-
@dhromed said:
@blakeyrat said:
Which one? All I'm seeing is rottweilers and surgery equipment repair.
SpectateSwamp RandomRandom
Yeah the whole "SpectateSwamp" thing goes woosh over my head. From what I gather he's just a crazy person you guys troll on other forums? (I've seen that "stupidiest person in the world" sig before.)
-
@blakeyrat said:
Yeah the whole "SpectateSwamp" thing goes woosh over my head. From what I gather he's just a crazy person you guys troll on other forums? (I've seen that "stupidiest person in the world" sig before.)
I'm not sure if it's more him trolling us or us trolling him. If you want to check it out, go to the "Funny Stuff" forum and look for any thread with over 20 pages.
-
@b-redeker said:
Regarding the Queensland Health payroll debacle...
Warnings by SAP, IBM and others to properly test the system were not heeded because it was considered too great a task.
This meant major defects were downgraded to minor to ensure the system could go live despite the known defects.
Test management is part of my role and I have to say this is one of the most unprofessional and reckless attitudes that I have heard. Even without the benefit of hindsight the people responsible for testing must have known that this was a stupid thing to do.
-
@Scarlet Manuka said:
@blakeyrat said:
Yeah the whole "SpectateSwamp" thing goes woosh over my head. From what I gather he's just a crazy person you guys troll on other forums? (I've seen that "stupidiest person in the world" sig before.)
I'm not sure if it's more him trolling us or us trolling him. If you want to check it out, go to the "Funny Stuff" forum and look for any thread with over 20 pages.Nah I really don't care. But, you know, just FYI you might wanna define acronyms that aren't in at least the first 2 pages of Google results.
-
-
Do financial systems not take lessons from safety-critical systems and have things like checksums and watchdogs and plausibility checks to prevent such things from occurring?
-
@too_many_usernames said:
Do financial systems not take lessons from safety-critical systems and have things like checksums and watchdogs and plausibility checks to prevent such things from occurring?
Saw [url=http://www.smh.com.au/business/nabs-woes-trigger-missing-transactions-20101202-18htx.html]an update[/url] on this story. I liked the second comment...@Robert said:
NAB had an underwhelming mag tape based IT system when I worked there 25 yeas ago; it seems that nothing has changed!
-
@blakeyrat said:
@Scarlet Manuka said:
..... I like this concept. I'm going to go leaf through the OOXML documentation now, and have a good, long think.Though that last one may just be because I'm bitter with Excel today for insisting that anything that vaguely resembles something that could possibly be interpreted as a date must be a date, even if the other 8,000 values in the same column are not.
BTW, does anybody know of an Excel-compatible strongly-typed spreadsheet program? One where, if you tell it column A is dates, it'll throw an error if you copy anything other than a date in it? I could really use something like that at work.
