Tuesday, July 18, 2006

Work Woes The 2nd

The last couple weekends I've had to log into work for the same production issue. The problem it appeared stemmed from bad data received. But oddly enough, a different version of the same job works perfectly when it ran the following morning. It was truly a mystery.

Worse I couldn't test the job due to issues with the test file that another group maintains. It took them a week to fix it. When this was finally done yesterday, I was able to run both versions of our program, confirming that one version had the problem while the other didn't. I realized that the issue was inadvertently caused by another group that snuck in changes to the code without my knowledge. I hate when folks do that.

I've been there before. Two years back, a group needing data from one of our files decided to recreate our file into the format that they needed. Only thing, by doing that, they messed up how our normal process handled the data. That took several days to make right and the people responsible barely acknowledged what they did.

They didn't even say sorry. It's amazing how folks go around changing your code on you without even letting you know or bothering to test it out. Fortunately the group responsible for the latest fiasco has been keeping me in the loop regarding changes to other programs I own. What they're doing is part of a company wide change to upgrade our COBOL VS programs to COBOL 370 since IBM no longer supports VS.

The good thing too is that the version of the code that was messed up only runs on Sundays whereas the other version of the code (which is unharmed but is on the list for upgrading) runs Monday through Saturday. Thank goodness they didn't change that one. The miraculous thing is that the job didn't go down while I was in Charlotte. Thank goodness for small favors.

Unfortunately for me, even if we fix this issue, I'm still going to have to work this weekend, the third weekend in a row I have to work. This is due to the company's annual "disaster recovery" exercises that happen to be taking place this weekend. It simulates what would happen if our data center was lost due to a disaster and what would it take to recover those files. Basically in a nut shell, I'll need to manually run our production processes that morning. No rest for the weary...:-(

Nine more months to go!!!!!!



Blogger Bklyn Diva said...

awh shucks.. SOOOOOOOOOOO did you take a few courses to help you out? ARe you going to sit for your Series 7?

12:08 PM, July 18, 2006  
Blogger Soldier said...

Ok E, that was so technical, i have absolutely no idea what you were talkin' bout but it sounded good... lol

Workin on weekends is not that horrible, i'm tellin u, u could even get used to it :D so be careful

12:16 PM, July 18, 2006  
Blogger lj said...

sounds very technical

3:51 PM, July 18, 2006  
Blogger Cash S. said...

I thought COBOL was extinct.

8:03 PM, July 19, 2006  
Blogger Ladynay said...

All I keep thinking of was Cater to you by B and dem! LOL

It's almost ova...

1:16 PM, July 20, 2006  
Blogger The Captain said...

First, you need to have a lockdown on the source code in your production environments. That should already be mandated since you are in production. I guess your systems are not evaluated by audit on a daily basis or working on a legacy system.

Second, you need an exclusive CTE (Client Test Environment) environment to test production issues and fixes if it requires changes to the source code or making changes based on the results of the diagnostic or heartbeat maps. (Yes, I am a software development consultant)

Third, I would get the DBA's and development members involved to control the nature of the source codes in all environments and a change management tool should be used to track changes/modifications and approvals. It seems your environments are open, which can present audit issues!

Fourth, disaster recovery is a monster when doing dry runs. Why don't you guys go to 4A? Better yet, use Veritas that manual move from active to passive-active servers to process your manual jobs. You may even find that this would eliminate work being lost, hung or corrupted during the change overs in the system.

Lastly, developers, coders, DBA's will never say they are sorry. They are the computer God's! However, there are other ways to mess with their creations....know what I mean?

3:01 PM, July 24, 2006  

Post a Comment

<< Home