Daily Discourse – Outsourcing Went Bad

The failure of DBS Bank’s electronic banking services reminds me of an incident that is rarely known by the public but told to me by a friend who is a UNIX administrator. It was a classic example of outsourcing went bad. The following is the account of the incident, but the names of the company and the internationally acclaimed vendor are changed (for obvious reasons).

This is the story:

It was during one fine night shift, when a data centre operator delete *.* accidentally. According to her own confession, she thought she was deleting files in a folder. Unfortunately for her, she was actually in the root directory when she did so. (For the uninitiated, the root directory is sort of like the main trunk of the tree. In short, while she thought she was cutting off a useless branch, she had actually chopped the whole tree down.)

The result of her action was disastrous. She effectively wiped out the OS and the mount-points on the SANs etc. Almost 4.3 TB (Terabytes – i.e. 4 x 1012 bytes) of files were deleted. Unlike the command prompt in Windows, where one can terminate a bad command with CTRL-C or CTRL-BREAK or close it down with Task Manager, a command executed in UNIX just keep going until the job is done.

So, even if she had realised her mistake and tried to stop it, she can’t. 23 Servers all across MNC X which were connected to the same SAN volumes (all affected by her erroneous command) went down immediately. Slowly the other servers were affected. The final ‘body count’: 168 servers of MNC X in that data centre were affected. The end result, nobody from any country, in any outlets / offices, could connect back to MNC X. It is simply an IT Black Day.

Best of it all, this happened somewhere around midnight and that gave the operator time to cover up what she did. She quickly modified whatever log files she has access to and deleted all her entries. So when the monitoring system (which is miraculously still functional) sent alerts to the ‘owners’ of each affected system, the tech guys who were awaken discovered to their own dismay they couldn’t login remotely (since theirs servers are all down). They were left with no choice but to drag their tired bodies back on site.

It took all of them almost 3 hours to be back on site, and they found no trace of what happened and could only scratch their heads since the log files were manipulated. Even more puzzling was that the redundancy failed – the backup systems had not kick in.

To cut the long story short, MNC X was left with no choice but to restore from backups so they can bring all their systems back online. It was over 15 hours later before they finally restored some semblance of order to the entire IT infrastructure. Meantime, Vendor Y launched an investigation and ultimately a guru in UNIX administration discovered the log files manipulations and even found out exactly who did it.

The saddest part of it all was that MNC X never discover the truth, even though MNC X probably also lost millions that day since it also has offices in other parts of the world which is still running. According to Vendor Y’s findings, the reason why the redundant systems didn’t kick in was a result of the backup systems being too old and they no longer matched the same configuration as the primary systems. (Doesn’t that sound almost as ridiculous as the reason DBS gave in their official statement – an upgrade – as to the cause of the break down of ALL their electronic banking services?)

That was of obviously the many droppings of a bull’s behind. After all, any IT technical person worth his salt would have asked what the heck vendor Y has been doing if it has not implemented hardware and / or software upgrades to the redundant systems to keep them up to date! They would have taken the vendor to court and sue them for a substantial amount of damages.

The story gets even better from here. The culprit was totally untouched because she has been Vendor Y’s perm staff for 20 over years. Instead, a contract staff was made the scapegoat and fired to appease MNC X.

What is most ultimate to this sad story of outsourcing went bad was that the contract staff who got fired was one of the three UNIX gurus who discovered who altered the damned logs. I personally suspect that the story doesn’t end here. Vendor Y probably convinced MNC X and sold it yet another several million dollar worth of hardware to ‘make sure this will never happen again’. (Note: The part about the vendor profiting from this fiasco is just my own speculation, not that it actually happened.)

When I looked at the magnitude of the staggering damage caused by Vendor Y in this major fxxk-up, the lack of dedication of the staff hired by the vendor which my current employer outsourced some of its IT services to paled by comparison. After all, the minor delays caused by these morons who simply didn’t put themselves in the shoes of our business users is nothing compared to what MNC X suffered in that one single morning.

Daily Discourse – Discrimination

“Seems you are ok with discrimination as long as you are not the target.”

This is the allegation leveled against me in a comment to my previous post. And I am really sick of such comments.

For starters, let’s talk about discrimination. These are several types of discrimination which I know of:

  1. racial – e.g. against the African Americans back in the U.S. in the 60s, the Jews in Nazi Germany and the Chinese in the early 20th century;
  2. gender – e.g. females denied the right to vote, glass ceiling for female employees in organisations, jobs and promotion given in priority to males, or expecting sexual favors from a female employee;
  3. caste – e.g. the untouchables in India;
  4. age – e.g. rejecting a job applicant solely because of his age;
  5. disability – e.g. refusal to render to a person a medical aid because of his disability;
  6. religion – e.g. refusal to hire a person because of his religious beliefs;
  7. language – e.g. to disgrace the people speaking a certain dialect, such as those from Ryukyu Islands in early 20th Century Japan; and
  8. reverse – in my opinion, a fine example would be the Woman’s Charter of Singapore, which originally was used to protect the disadvantaged (i.e. lower educated, unemployed) women – usually home-makers – in the early days of post-colonial Singapore from abusive husbands. Sadly, it has become more of a bane to the modern man today.

Gender, age, disability, language and religion are also usually seen as a part of employment discrimination. That is why these days consultants advices interviewers never to ask questions related to them to avoid getting the company sued by the interviewee.


However, there is also another kind of discrimination – perceived discrimination. And it reminds me of an ex-colleague, who is in the opinion that our head of department looks down on him. It doesn’t matter that the real issue was the attitude and the lack of interest he displays towards his work. In the end he deliberately not do the things which our boss would consider him a good worker and a team player, and that goes onward to reinforce the boss continual dislike of him and his working attitude.

And so when I looked at the allegation leveled against me, I asked myself if there is any real discrimination in place against homosexual people, or is it all perceived as such. Are homosexuals in Singapore denied the right to vote? Are they denied the use of certain public services like health care or the buses? Are they getting thrown out of restaurants? Or are they denied employment even when they are more than qualified for the job?

Fortunately, I haven’t heard of such blatant acts of discrimination in Singapore. But the first thing that come to mind is Section 377A of the Penal Code, which I believe was necessary to protect the populace from certain sexual predators. However, before anyone screams about me drawing comparisons between male homosexuals and sexual predators, don’t put words in my mouth and consider this analogy: A prostitute may dress scantily, but is everyone who dress scantily a prostitute?

Then, there is also the ‘302’ classification and the straight downgrade of homosexuals to PES C in the army. I was reminded of this by fellow blogger Michael Cheng [aka Botak Cheng]. Personally, I have heard about incidents where those who are allegedly classified as such were bullied in camp by fellow soldiers. I am not sure if those classified as such are upset, but I do not see nor really understand the reason to the segregation. If homosexuals do not oppose those duties, then put them on duty like the rest of the men, until they do something against military regulations.

Anyway, parallel is continually being drawn in the U.S. between the homosexual community with that of the movement against discrimination of African American in the 1960s. But a quick check with a dozen of my friends who happens to be online in Windows Live Messenger showed that most cannot name any discrimination against homosexual here in Singapore. In fact, only 2 out of the dozen people I asked pointed me to this link to get myself duly informed.

Now, while I may oppose them, I have no issues with the homosexual community’s efforts to deal with the items listed on the Wiki page. But what has that got anything to do with DBS tie-in with FOTF? It was pretty obvious to me that it is nothing more than the fight against FOTF U.S. spilling over to FOTF Singapore. Has this ‘victory’ done anything to improve the situation of the homosexual community here in Singapore?

I won’t be surprised that in the near future, there is this opinion that all of these action groups are nothing more than small pompous groups presuming to speak for a larger community. In fact, I am in the opinion that the homosexual community by and large did not even give these ‘loud mouths’ the authority to speak on their behalf, nor are they very interested in the agenda.

It maybe possible these groups will raise the profile of those who claims to speak for the community, but it is doubtful it would have any real effects for the plight of the community or bring about any positive action. In fact, I am not even surprised if it actually galvanises the opposition.

To put it in the words of my friend, “Some people do nothing until they are called the enemy.”


Comics:


Recommended Reads:
Cobalt Paladin: Diary of an Entrepreneur – Money Saving Tip #2
Cobalt Paladin: Diary of an Entrepreneur -Signs of our times #4
Endoh’s Dungeon: Do you represent me… really?
Isaiahc: Seven Things to Love About WordPress 2.7

Daily Discourse – DBS & FOTF

I read with a little annoyance the following news [see below]:

Online campaign leads to rethink at DBS
Siow Li Sen
Fri, Dec 05, 2008
The Business Times

DBS Bank has removed all references to Focus On The Family (FOTF) in its advertising, after its credit card promotion supporting the evangelical Christian organisation provoked some angry reactions.

Since the bank’s Nov 13 credit card promotion where DBS said that it would donate money to FOTF, ‘a charity dedicated to helping children and families thrive’, members of the gay and lesbian community have called for a boycott of the bank.

FOTF in Singapore is an affiliate of a US-based organisation of the same name founded in 1977 by evangelical Christian James Dobson who campaigns against gay rights.

Last month, FOTF in the US retrenched some 200 staff after it spent US$600,000 to defeat marriage equality in California. Mr Dobson has also railed against US President-elect Barack Obama, who has countered that Mr Dobson ‘makes things up’.

DBS spokeswoman Karen Ngui said that it was never the intention of the bank to alienate any particular group. ‘DBS supports children and learning in Asia . . . it’s the cause that we are supporting and not FOTF, and or what it stands for,’ she said.

She added that DBS believes in diversity and inclusion.

‘We have since removed all references to FOTF in our advertising . . . however, we still support the cause . . . and thus will be contributing a small amount to their New Learning Centre for children with learning disabilities, due to be opened in March 2009.’

The bank’s earlier move had sparked an online campaign. Jean Chong, a gay activist with People Like Us, said that to date, 1,063 people, including non-gays, had signed up with a Facebook group attacking the bank’s support for an FOTF cause.

While some gay activists concede that DBS may not have known about the anti-gay agenda of FOTF, they felt that it still should not give to the charity because the donation could indirectly benefit its cause.

‘It is my view that unfortunately, DBS did not realise that FOTF has a lot of baggage. It is also unclear how by funding one part of an organisation’s activities, how much you also indirectly help another of its activities because money is fungible,’ said Alex Au, People Like Us gay activist.

Ms Ngui has said that its credit card team checked on FOTF and proceeded as they are endorsed by the Ministry of Community Development, Youth and Sports and National Council of Social Service (NCSS). ‘Going forward, the bank will conduct more extensive background checks,’ she said.

Why does it annoy me? First of all, it apparently doesn’t matter what the objective of the campaign was. It doesn’t matter FOTF is supporting a children and learning project. All that mattered to some group of people is this: FOTF was anti-homosexual.

By the same way they attempt public opinion against DBS’ association with FOTP, I wondered whether they are even aware that it also can used to ‘show’ homosexuals are anti-community? I am quite amused to see how they shot themselves in the foot, by going against a pro-community action.

Now that this has worked to cow DBS, I won’t be surprise that very soon, churches and the Christian community will have a hard time to work with secular organizations for community projects and activities, because such projects and activities are going to get boycott by ‘many in the [homosexual] community who have expressed outrage over’ the association.

In fact, I won’t be surprised that one day, when there’s a Christian bleeding to death – with a crucifix hanging obviously around his neck – homosexuals will just walk by and let him die, and would justify it because the dying man is anti-homosexual. After all, why aid a guy who is well, with the anti-homosexual groups? It doesn’t matter that it is my belief that most Christians would render aid to a homosexual in need.

Where, is such senseless reverse discrimination, going to lead us?

As a Christian, I can perfectly leave homosexuals alone and let them do whatever they want behind close doors, and even ignore what I see in public. As to how I personally felt about it, the leaders of the Church have already made their statement clear and I do not need to repeat it here. My stand has always been this: God is the ultimate moral judge and I do not pretend to speak His Judgement.

Beyond that, I would even champion for universal suffrage, health care, job opportunities for them if I see any discrimination in any of those areas in spite of my faith in Jesus Christ. But I will not stand for any form of reverse discrimination against my community as a matter of faith and differing views on morality.

And according to what I have read, Focus on the Family (FOTF) is also a voluntary welfare organisation (VWO) endorsed by the Ministry of Community Development, Youth and Sports. So, shouldn’t the ministry, and by extension the government gahmen of the Republic of Singapore, be boycotted by the homosexual community as well?

So, all homosexuals should quit their jobs in the civil service now because our gahmen is an evil organisation promoting an anti-gay agenda! If not, take what you already have and keep it, and stop agitating for more.

Enough, is enough already.


Comics:


Recommended Reads:
Cobalt Paladin: Diary of an Entrepreneur – Money Saving Tip #1
Endoh: Boycott DBS for being anti-family?


Addendum: Comment section for this post has been closed. Further comments in other parts of this blog on this issue will be indiscriminately deleted.

Daily Discourse – DBS High Notes 5

DBS has said it would take responsibility if customers are able to give evidence of mis-selling in relation to products affected by the collapse of Lehman Brothers.

Now, let’s look at the profile of some of these poor souls who lost their money. These are a few of them which I gathered from searching Google:

  1. 60-year-old retiree Tham Wai Wah (who has only an O-level education): S$ 125,000.
  2. 59-year-old retiree who only wanted to be known as Mrs Lim: S$25,000.

These are just the reported cases. Ask around and you will find friends telling you that their own elderly parents have been approached to invest in some of these products. In fact, ask your own elderly parents if they have been approached when they go down to the bank to either withdraw their matured Fixed Deposit (FD), or to start a new one. Mine has been, and I thanked God none of these products they are ‘advised’ to put their money in are Lehman related. Better still, some of these products are maturing in 2 months.

Now, can anyone honestly dare said these old folks understood what a structured product is and all the risks behind it? Why do we hear that some of these people lost their entire life savings? Why, is it even sold to old folks whose original idea was simply to earn some honest interest from fixed deposits?

I spoke with my dad on one of the products he has placed some money into, and all he could tell me was that the guy who sold it to him told him this: Principal guaranteed, and even if the markets is doing badly he could earn 10%.

And this appears to be the same thing I am reading about these old folks who lost their money. To me, it is obvious that this is mis-selling. But solid proof of this will be hard to come by. I am quite sure some of these old folks probably couldn’t even remember who sold the product to them, nor could they find that particular staff anymore and ‘interrogate’ them. Just how on earth are investors going to prove the mis-selling without a full recording of the entire series of events leading up to their decision to invest into that product?

No wonder my friend’s comment said this when he was told of the above news: “如果我呃你,我老咗会死!” [It losely translates as – If I have bluffed / cheated you, I will die when I get old.]

And I can’t help but agree with his comment.

By the way, I heard Hong Kong, Macau and Australia have all guaranteed 100% of deposits in banks. When will Singapore, with its immense reserves do the same? With all the money around for the GIC and Temasek to invest everywhere, can’t the government gahmen even do this little bit to assure its citizens?


Comics:


Recommendations:

Good reads:
The Online Citizen (Rachel Chung): How much does the Government care?
Communplugged: Metrics Are So Yesterday!


Other news:

Local blog aggregator ping.sg pR0n.sg has gotten a face lift and new features. Lycan Times gathered that users who contribute their posts to it may now choose to keep ‘ponging’ of their posts private. This means registered users who read the posts which are not well liked and yet fears repurcussion will now be hidden from the public.

Either way, it’s a useless feature when one look at the current state of the Top 10. It is clear that few are really concerned with being exposed for ponging the lar sup (dirty) aka ‘NSFW’ (Not Safe For Work) posts. The fear of ‘exposure’ was nothing more than a speed bump previously. Allowing them to pong anonymously now simply means they will now be ponging such post with abandon and bring the domination of NSFW posts in the Top 10 to new heights.

On top of this change, there is now ‘community based’ features so only blog posts of friends will be listed. It maybe a wonderful feature so you don’t miss your friends’ post, but you can do the same using an online RSS reader such as Google Reader. Either way, after looking at the screen shots on some of the blogs reporting this feature, the layout reminded me of either blogcatalog or mybloglog.

Personal Opinion: It’s to pR0n.sg just as Aero is to Windows Vista.