What is your worst IT Day?

[PNGBill]

I have just experienced, still are, one of my worst IT days.

Our system consists (did) a Linux Mail Server and 3 windows xp pc's using dual screen.

We wanted to add a fourth pc to the network and had an older pc that now only handles linux (don't know why).

Successfully got this to run as the Linux Mail server and was about to turn the old server, newer pc, into our 4th pc when &%#* happened

End result...

Linux File Server crashed with all sorts of noises coming from it.

The old server (newer pc) doesn't accept the Video Card into it's AGP slot. 3mm out!! Maybe the case is slightly off but no way it will fit unless some Panel Beating is done. The Slot needs to move 3mm away from the back of the box. The video card has worked with no problems on much older pc's.

We now have, no Mail Server and a pc that won't take dual monitors. Can't imagine life without two screens as we seldom use printed paper and rely on viewing one document while working on another.

Am now attempting to get a windows xp mailserver working on the old server/new pc so we can use it as dual mailserver and pc.

All email history is lost (IMAP) - still on hdd but a lot of messing around to get it off.

Luckily we have an adsl internet connection as well as our mailserver otherwise we would be off air entirely.

 

[pbaldy]

Ouch! Sounds like a mess. Mine was several years ago when the Nimda virus was wild. It somehow found a crack and got into the network. It infected two or three servers and several PC's. It took days to get everything back to normal, with nights/weekends thrown in for fun. It was a nightmare to clean up.

 

[The_Doc_Man]

Well, it wasn't my PERSONAL worst day, but it was a corporate worst day some years ago. An "unwise child" employee was using some removable hard platters back in the day when a removable disk had 5 Mby on it (and that was a LOT!) and it was two or three inches tall and maybe just over 30 inches across. It was an RK05 disk, for those who want to bother to look it up.

So let's call him Sam. Sam came to me and said, "Why does my disk make such a terrible grinding sound when I put it into that machine over there?" Well, that popped my eyes open instantly. He continued, "I thought maybe there was something wrong with the upper drive so I moved the disk to the lower drive and it did the same thing." I was beginning to look for a machete when he chilled me to the bone. "So I thought OK, that whole MACHINE must be bad, so I went to another system and tried that. You know, it didn't work THERE either."

To shorten this long story, the disk had wobbled ever so slightly. His disk had been scored down to the aluminum backing on a swatch at least three inches wide from the outermost tracks inward. NAKED METAL, no oxide coating, no nothing. He had destroyed three sets of disk heads. Oxide dust was everywhere. The filters had to be changed. We were down two systems while our tech did the repair work over the course of a couple of days. Our in-house service tech Jerry had to be physically restrained. I think he might actually have punched out Sam if we hadn't been there.

Oh, and of course Sam had not recently backed up that well so he lost two weeks worth of work. We sent Sam to do some desk work in a corner cubicle so that no one would kill him. We eventually fired him for cause because of the way he treated one of our paying customers, but that day was one that made me think the world was coming to an end. And Sam was supposed to be one of the "bright" guys. I still shudder to think of the things he did.

 

[PNGBill]

The_Doc_Man. I thought I had a Bad Day!

I remember when we first used IT at work we used Inventory Control on dumb terminals via a modem to a Prime Main Frame. Two Main Frames linked by a thick cable and using a 256kb hard disc like a LP record.

No Graphics and tape drives also used.

The two main frames supplied a Nationwide (PNG) motor vehicle spare parts inventory, mu hardware store and a large companies GL.

Our UPS was the size of a washing machine and we wondered why te screen were going washy one day only to find a salesman had plugged an electric concrete mixer into the ups power supply to the cash register
The ups supplied power from batteries rather then the modern ones that cut in instantly.

We had good performance, but no pictures.

Could word process, of a kind and use chat through the Intranet and even link onto the other mainframe but you noticed the speed reduction.

One floor of a building was the IT dept.

My home pc was a commodore 64.

 

[WayneRyan]

Doc,

Thanks for the laughs today!

Sam must have left and changed his name to Larry. I worked with him.

He paraded around shoving the same "platter" into numerous drives. That also wasn't a
good day.

After he left us, he had learned that you should label your disks. But, he put those gummy
little things on the disk surfaces themselves. Gotcha again.

He also turned a nice VAX into a single-user system, by defining it with working-sets so
large (like 32 MB) that it was unusable in a "normal" environment. He did a lot of work
at night.

Mark also worked with us. He thought nothing of pulling a plug out of a power strip
to free up a slot. That can have some severe repurcussions too.

Nice to relive those thoughts. If it doesn't kill you it'll make you stronger.

See ya,
Wayne

 

[WayneRyan]

Doc,

PS ...

If you REALLY miss the EDT editor like I did, it is still alive.

There is a commercial version from (I think) Boston Computing.

There is a freeware version called "ED-NT".

I loved that editor and had posted here looking for it because it had the
unique functionality of defining macros easily.

That's very useful when you are repetitiously processing something like a
list of files, tables or data columns.

It's great to be using it again, as it really amazes some folks with how
easily a block of text can be manipulated.

Wayne

 

[Steve R.]

We had a Unix system serving about 40 employees. I was fishing through the wiring and pulled out the wrong plug. I was greeted to the sound of the hard drives winding down. Expletives deleted.

In another incident, we had prepared a rather large environmental impact statement. The HQ office sent us revisions which we dutifully incorporated. We then sent the revised version back only to hear that we had "forgotten" to make the corrections. We began to sweat big time. It turned out that the HQ office had re-used the old document, so our lives were spared. All subsequent revisions had a date/time stamp on every page in the footer.

Here is an interesting home LAN story. We had two computers, both with UPS back-up. It was a weekend and the computers were being used for crunch-time school work, if I remember correctly. My wife turned on the vacuum, the house lights dimmed and both computers immediately began making clicking sounds. The computers did not fully crash, but I was getting bad sectors all over the place. CompUSA was still open. Fortunately, I was able to recover the data from the damaged hard drives so that the school work (?) could be completed on time.

 

[PNGBill]

So concrete mixers and vacuum cleaners need bigger ups's

 

[Vassago]

Here is an interesting home LAN story. We had two computers, both with UPS back-up. It was a weekend and the computers were being used for crunch-time school work, if I remember correctly. My wife turned on the vacuum, the house lights dimmed and both computers immediately began making clicking sounds. The computers did not fully crash, but I was getting bad sectors all over the place. CompUSA was still open. Fortunately, I was able to recover the data from the damaged hard drives so that the school work (?) could be completed on time.

That's bizarre. With UPS Backup, that definitely shouldn't happen as not only should they be protected from power spikes or shifts from turning on a vacuum, but they should also keep constant power with the backup.

I used to work at AOL in the tech support area years and years ago. I could tell you some stories that would make you *facepalm*.

 

[rainman89]

AOL= Facepalm!

 

[Vassago]

AOL= Facepalm!

Too true! The benefits of working there in the late 90s were definitely worth it though. There were people in their early 20s who retired millionaires from stock options alone. The stock split multiple times in 3 short years. Lucky people...

 

[ChipperT]

I worked as a mainframe operator in the underground at SAC Headquarters, many many years ago. The computer system I was assigned to was one of the main ones that ran the big boards in the war room there. It was an old IBM 7090 system. The emergency stop was a plunger-type lever on the upper front that had to be pulled out towards you and it would completely and immediately shut off all power to the system. We were in the middle of an exercise (war game) and the IBM field engineer was in the room standing by as they always did when an exercise was being run. In those days, the IBM'ers had a uniform that consisted of a gray suit, white shirt, tie. The FE had hung his suit jacket on that lever. Of course, he needed to leave the room to visit the gentleman's facililty and reached up and pulled his jacket off the lever, immediately shutting down the system! The exercise came to a complete halt, even though the primitive fail-safes took over and a less capable computer was brought on line within just a few minutes. There were general-ranked officers running everywhere! We never saw that FE again.

 

[rainman89]

Too true! The benefits of working there in the late 90s were definitely worth it though. There were people in their early 20s who retired millionaires from stock options alone. The stock split multiple times in 3 short years. Lucky people...

Yeah definitely woulda been in the money back then..

 

[Fifty2One]

Worst day EVER was yesterday when there was an earth tremour and it caused some power fluctuations, had to reset all sorts of machines which was no biggy... BUT... during the shaking my tea mug did not survive a tumble from my desk... I do not know how I made it through the day...

 

[rainman89]

Worst day EVER was yesterday when there was an earth tremour and it caused some power fluctuations, had to reset all sorts of machines which was no biggy... BUT... during the shaking my tea mug did not survive a tumble from my desk... I do not know how I made it through the day...

You felt it too eh? I didn't personally but some in my building said they did.

 

[jamesmor]

I used to work at AOL in the tech support area years and years ago. I could tell you some stories that would make you *facepalm*.

I worked at Earthlink tech support.

funniest call ever was when the northeast had that blackout earlier in the decade.

him: "i can't connect to the internet"
me: "what's the error message you're getting"
him: gives error (can't remember the error number)
me: "where do you live"
him: "new york"
me: "can you even turn on your lights"
him: "no the power is out"
me: "so what makes you think that our dialup servers in that area are going to have power?"

once he realized his mistake, he appologized for calling

 

[The_Doc_Man]

Wayne Ryan... regarding EDT and VAXen? I've graduated somewhat. Now, instead of running a heterogeneous VAXcluster and VMS 4.7, I'm up to running four Alpha ES40s and OpenVMS 8.3, and you had better believe I still use EDT as much as I can.

We might actually graduate one more time - to Integrity Servers in the rx3600 class, don't know the exact model number. Our main personnel app was written around VMS. I've been the sys admin for the 2-node VAX cluster, then a 4-node cluster, then we upgraded to 3 Alpha 2100 + 2 Alpha 1000 as file-servers in a token-ring setup with dual counter-rotating 100 Mbit rings on 2 different interfaces - so we really COULD reach peak data rates of 200 Mbit. Now we are on ES40s with 4 CPUs each, running some old fiber at about 1 Gbit but hoping to get to the Integrity servers with 4 Gbit fiber.

The reason we stayed with OpenVMS so long is because as expensive as it can be to upgrade the h/w, migrating the million-lines-of-code s/w to some other o/s is daunting even to the big-bucks navy guys. If I can put up with the insanity for five more years, I'll be retiring. After that, it ain't my problem.

Of course, where I work, it's not like the U.S. Post Office. Our guys don't go postal. They work for the navy. They go naval. Which brings to mind the old sea chantey, "What do you do with a drunken sailor...."

 

[Galaxiom]

I didn't have bad days, just bad years. I worked in a place where the people in control were so mindnumbingly, dumbfoundingly stupid that it took years for them to accept that their invoicing system had to be updated.

It was 2002. The system ran with PC emulated serial terminals connected to a nine year old server via a 10 port ISA board. Most of the serial ports buffers had been blown by electrical faults leaving three terminals in operation. The bookkeeper did not even have access to the system. She had others print reports which she then added up on a calculator (honestly) and typed into a spreadsheet for reporting.

The terminals were booted with similarly aged 720KB floppies. (Floppies actually worked surprising well back then) Most of these protected uncopyable specially formatted disks had failed. One terminal had a licence on a hard drive. They took one floppy disk around to boot two other terminals each morning.

The whole system had not been updated since 1992 and the vendors had totally deprecated it in 1997 in favour of a modern system.

On arrival to work in the business (for non computer duties) I immediately pointed out that they could be out of operation overnight. The bookkeeper said she would look into a new system. Meanwhile I did emergency repairs to the serial buffers, got lucky and tracked down unprotected versions of the terminal client software and got all the terminals running again.

By then some desks had a Windows computer and a terminal emulator. I even got hold of an Ethernet client for the terminals. However it would only run on Windows 9x and besides some of the terminals were DOS 086s (I kid you not) with no Ethernet. It was considered too expensive to provide Windows computers for everyone.

I pointed out the server could fail at any moment and was told that "was not a priority". I scrounged an old 266Mhz box (no point going faster since the system crashed if it was run faster than 350MHz) but because it could not support the ISA card properly it would mean dumping the 086 terminal emulators to something with an Ethernet capability. Too expensive.

The server did fail not long after and they were forced to the Ethernet clients anyway. Unfortunately a certain combination of video card and ethernet card would cause the terminal client to freeze multiple times a day. And they were stuck with Windows 98 on the clients.

Meanwhile they had one computer in the building with internet access and email. Those with Windows shared the same email account despite the fact that it expressly stated that it must only run a single session at a time.

I set up individual mail accounts, put in a free mail server and free proxy server onto the net connected computer to demonstate the benefits of getting everyone connected and applied to get a dedicated box to handle the extra load. Oh no, too much expense. The Windows 98 machine could run it but could I please stop it crashing so often because it interrupted the receptionists typing.

Still the bookkeeper did not come up with a recommended replacement for the invoicing system. It took me another year to convince the management that the bookkeeper was utterly clueless and incapable. I took on the research and I made a recomendation myself. The bookkeeper resigned in protest.

The decision to follow my recomendation was finally taken two weeks before the end of the financial year. They then asked me to have the new system up and running for the new year, leaving me a whole week to convert the data and configure the new system once the software arrived.

I actually had it going six weeks into the year but with many problems (unsurprisingly). I cut my teeth on databases working through the back end of that system to fix it which took another year or so until I had the sytem configured to my satisfaction. By then I was full time computer engineer and the job I came there to do languished.

I was then informed that all the problems with the system had been caused by my arrogance at which point I offered to show the facts I had carefully explained in a pile of reports I had written during the course of the negotiation. He told me that he "was not interested in facts" because he had made up his mind.

That was more than enough for me. I was crazy to have stayed so long but my parents had brought me up not to be a quitter. Still what I learned in that process of become an accidental dba got me the great job I have now.

 

[Extra_Cover]

This is an old story but still makes me smile. I got this from a Computer magazine ages ago (pre-internet) - think it was Computer Weekly? Readers were askes to send in help desk stories.

Help Desk : Can I help?

Caller: Yes, I'm having a problem with one of my floppy disks.

Help Desk: What sort of problem?

Caller: Well it won't let me do anything

Help Desk: What do you mean?

Caller: It's my data cannot update it.

Help Desk: How

Caller : (loosing patience) My disk won't work

Help Desk: (also loosing patience) Look why don't you send me a copy of the disk and I'll have a look at it. Can you do that?

Caller: .... yes ... yes I can.

A couple of days later the post (told you it was an old story) arrives. The help desk opperator opens it and find a photocopy of a 3.1/2 " floppy disk inside. Luckily they able to solve the problem as the write protection tab had been set.

 

[ChipperT]

This is an old story but still makes me smile. I got this from a Computer magazine ages ago (pre-internet) - think it was Computer Weekly? Readers were askes to send in help desk stories.


A couple of days later the post (told you it was an old story) arrives. The help desk opperator opens it and find a photocopy of a 3.1/2 " floppy disk inside. Luckily they able to solve the problem as the write protection tab had been set.

Reminds me of an incident many years ago when PCs were the new "thing" and all software companies were very small. There was only one database at that time, called DBase and it was version 1. I called and asked for a demo disk because the company I was working for was considering purchasing it. (There was no Internet, no "freeware" etc at that time). The admin at the company I talked to was very obliging and immediately sent me a demo floppy disk (3.5" when those really were floppy), with the installation instructions stapled to it....