The Morris Worm

While editing a college textbook on cybersecurity (still a secret project at this time), I’ve run across a description of the Morris Worm. This was the first true Internet worm, released in 1988, and it ravaged the early Internet. At the time, I was a software engineer, writing low-level communications programs in a networked device management system on PDP-11’s running RSTS and on Unix systems, primarily IBM RT and Sage Micro.

Our Internet connection at that time was dial-up, brokered through a well-known system called “unrvax,” a Digital VAX microcomputer running BSD Unix. Unrvax was infected by the Morris Worm, and it propagated to our Sage Micro system, but did not execute there as Sage was not one of the target systems. I recall hearing that the folks at UNR had to circle the wagons to deal with Morris, but our organization was not directly affected, aside from delays in email and news feeds.

I’m not sure whether the Morris Worm is covered in modern CS or cybersecurity courses.

Hand Coded Sorting

One of my earliest jobs was that of a programmer at Nevada National Bank in Reno, Nevada. I wrote programs in BASIC on a new IBM PC running DOS 1.0 with two floppy drives. The file system in DOS 1.0 had no subdirectories, only files.

This IBM PC also had a Corvus brand 5Mb hard drive that was about the size of the box that a pair of cowboy boots would come in. IBM PCs running DOS 1.0 did not support hard drives, but it worked because the hard drive had its own controller that DOS thought was a huge floppy drive. Still, no subdirectories.

I wrote and updated numerous programs performing various functions, including branch banking work measurement. Little did I know that this experience would soon prove valuable.

In a financial recession, the Bank conducted a reduction-in-force, and I found myself out of a job in December in a weak job market.

I was also an active member of Mensa and joined a local group on Monday evenings at a hotel bar. One of the regulars there found me a contract job at LifeTouch Portrait Studios’ Western U.S. processing center. This would prove to be key to a great career.

LifeTouch Portrait Studios hired me as a contractor to write a piecework payroll system for their photo printing factory. This was a tall order, but having taken a class the year before in Warnier-Orr structured program design analysis, I was well equipped to write the entire system. I worked in an office with the company’s IBM PC and got to work on the project.

The system I designed and created consisted of a data entry program, a data correction program, and a report program. I designed a database that consisted of a series of flat files as there were no relational database management systems available for IBM PCs. This was not an obstacle for me, as I already had a pretty good background in data design, and this piecework payroll system was not all that complicated.

I ran into a significant problem when I was writing the report program. Data entry records could be entered in any sequence, but the report output had to be in a particular sequence. There were thousands of records, and the DOS built-in sort program was no help: I had to write a sorting routine.

There was no Internet, nor even UUNet, at this time, so I hoofed over to the Nevada-Reno library and checked out a book on computer algorithms that contained pseudocode on various sort algorithms. After some reading, I elected to use the Shell-Mentzer algorithm, as it was purported to be pretty fast. I coded it up, tested it, and it seemed to function correctly.

One problem: the sort took 45 minutes to run. What to do!

I went down to Egghead Computers and picked up a copy of Microsoft BASIC Compliler, which cost me $50 if I recall. I compiled the report program and ran it again – this time, the sort took 1-2 minutes.

The reason that the compiled program sort ran faster is the elimination of the BASIC runtime system interpreting each instruction in the program, compounded by the overhead of reading and writing to a floppy disc.

This experience, including the design, programming, and working to ensure the customer understood how to use the system, would prove invaluable later the same year, when I was hired by Bally Systems as a software engineer. There, I initially wrote and updated BASIC-Plus programs on DEC PDP-11 computers for our customers, who were the largest casinos in the world. Later in this job, I learned Unix and C, and led a team of software engineers to rewrite one of our company’s applications in C on Unix with relational databases. This proved to be my breakout job that led to even bigger and better positions in the years to come.

Bursters and Decollators

In mainframe shops with line printers, printouts are produced on continuous “greenbar” paper.

Often, the users of these reports would want these greenbar reports “bursted”, meaning that each page was separated from the next, and the printout stacked. Users usually put these reports into large binders made for this purpose.

Further, some greenbar paper was “multi-part”, meaning there were multiple layers of paper with carbon paper in between. There was two-part, three-part, and five-part paper. The more “parts” there were, the thinner the paper, so that it would all feed through the line printer.

We computer operators would have to operate the machinery to disassemble these printouts so that they were human readable.

The first one is known as a decollator. This is a large (6-10 feet in length) machine that is used to separate all of the layers of paper, as well as remove the carbon paper in between. A decollator took a few minutes just to set up a run, as you had to carefully thread the printout through several series of sprokets, and wind the first few feet of carbon paper around a spool. Then you turned a large wheel to advance the paper very slowly to make sure it was all set. Then you throw the switch, and the whole thing comes alive. There is a large speed control dial, and it’s smart to start it out very slowly.

You didn’t dare turn your back on a decollator, for all kinds of things could go wrong with it. First, the carbon paper winding up on the spool could go hayware and shoot out all over the floor.  Next, the individual copies of printout might not stack neatly (often the case with 5-part).

I believe the decollator we had could decollate three-part, so if you had a five part report you had to run part of it through again, as one of the stacks would still be two-part.

Decollator

Then there is the burster. This is a machine that takes continuous greenbar printout and separates (bursts) the pages and stacks them.  Easier said than done.  Bursters had an uncanny ability to rip individual pages and make a real mess of things. Next to every burster is a large spool of cellophane tape for repairing disasters.

Large computer printouts could consume several BOXES of greenbar, and they could be multi-part. It could take an hour or more to take a multi-box printout, and decollate it all, then burst it. And I tell you that bursting five part paper is no fun – it’s so fragile that the machine would rip a report to shreds if given half a chance.

Burster

I learned some of my best swearing while operating these machines. But, I also learned patience, for it was possible to run an entire large five-part report through these machines without incident. Then again, it’s possible to bowl a perfect game.

Core Memory

Several of the first computers I used had core memory for RAM. Core was reliable, and one plus is that it retained its contents when power was removed. Some computers took advantage of this and, upon bootup, could save the contents to disk.

I have a couple of core boards, one from a PDP-10 and one from a PDP-11. I haven’t seen since I last moved a few years ago; I’m sure they are in a box someplace.

Here are some images of core memory.

4K Memory Card

Here is a close-up view of a core board – you can see the individual bits:

Close-up view of core memory

And here is what one bit looks like – not all that small.

Early core memory boards were wired by hand. Makes my eyes hurt thinking about that.

Program Library

While employed at the university computing center, I had an opportunity to visit the university’s program library.

In the days of punch cards and mag tapes, a program library was not a THING, but instead it was a PLACE. In this case it was a locked room with several large cabinets with long sliding drawers. It reminded me of a morgue. On each sliding drawer was a tray with punch cards. On the front of each tray was the name of the program, generally encoded. A payroll program, for instance, might be called PR0201.

The programs in the program library were the source code, typically written in COBOL. When a change was needed to a program, a programmer would check out the source code; a clerk would go to the program library, remove the card deck, and take it to the programmer’s office. The programmer would go to one of the rooms with a key punch machine, punch new lines of the program, and insert them into the correct places in the deck.

The programmer would then bring the deck to the operations counter and submit it for compiling. One of us operators would take the deck into the computer room and run it all through the card reader.  Our card reader was very high speed, something like 300-500 cards per minute.

Once all the cards were read in, the batch job would begin. The compiler would compile the program and prepare a report. If there was a compiler error, the report would indicate the approximate locations of the error.

Once successfuly compiled and tested, the binary would be transferred to one of the disk packs where compiled business programs resided. And the source code would be checked in to the program library.

Punch Cards

Early in my career I did a lot with punch cards.  My first encounter with punch cards was in a Freshman level introduction to FORTRAN programming class. At the University, programming was done on punch cards; the computer: a CDC-6400, a high-end number cruncher back in the day.

The university had perhaps a dozen IBM 029 card punch machines that could be used by students. It resembled a typewriter that was built into a desk, with card reading and punching apparatus up on top. It took quite a bit of practice to really understand the 029 and be productive on it.

Punch cards have 80 columns. Each column is a character. The punch card in the illustration here is a single line from a FORTRAN program. If you had a big FORTRAN program, then you had a big deck of cards. If you had a program with hundreds or thousands of lines of code, then chances are you had one of those metal trays used to securely hold a card deck.

If you dropped a large card deck, you were hosed, UNLESS you had sequence numbers punched into the cards (or had hand-written the sequence). If your deck was especially large AND sequenced, you could go to the computing center and ask someone to run your deck through the card sorter, a large machine that would, after several passes and correct handling, get your deck back into order.

In case of fire, first run backups

Not long into my first full time employment in a data center (the DEC PDP-10 mainframe), we had an operations manager who was a little ornery and liked to prank his employees.

We had a young computer operator named Rose; she was probably in her late 20s and a pleasant colleague.

One day, Ralph sauntered into the computer room to see what was going on. Rose asked him about fire in the computer room and what should be done. This was a room protected with inert gas fire suppression (Halon in those days), and from my training at the university computer center, I knew we were supposed to get the hell out of the room if the fire alarm sounded.

But Ralph had an entirely different answer for Rose. He told her, “If the fire alarm sounds, you need to pull the backup tape list, grab all of those tapes and take them with you as you leave the room. But if backups have not been done yet, you need to run them and then take all of the tapes with you.”

“Oh, okay,” she replied.

Uh oh, she took the bait.

Honestly I don’t remember in great detail how Ralph backed out of that. Of course, if there really was a fire, he wanted her to get out as quickly as possible, leaving everything behind.

Ralph, despite his orneriness and dry humor, was pretty respectful of his staff and others in the organization, so I know he wasn’t aiming to make a fool of her. But he did have a moment of fun.

The MG Set

I was employed at an agency that had a DEC PDP-10 mainframe. A year after my arrival, we migrated our applications to a Unisys 1100/60 mainframe. The Unisys occupied a brand new raised floor computer room with all the latest technologies.

For power management, we had a new power distribution unit, essentially a large cabinet with power controls for routing power to all of the components in the data center.

We had no generator or UPS, but instead we had a motor-generator, or an MG Set as it was called. The MG was in a separate locked room that was seldome entered. Only the operations manager and the MIS director had access to this room.

The room measured about 15 feet wide and 25 feet long. In the middle of the floor was the MG set, and man what a racket it made. It was so loud in there that it was painful. We instinctfully plugged our ears and had to shout the few words we had in there.

The MG works like this: electricity from the public utility powers a large motor. A shaft connects the motor to a large flywheel, and on the other end of the shaft, a large alternator. The purpose of the MG is to clean up the power coming in, since it was converted into mechanical energy, and then fresh clean power generated by the alternator. The MG could withstand a brief power outage, just a few seconds worth, before the spinning flywheel slowed down enough that the alternator could no longer generate enough power.

This is the only MG unit in operation that I have ever seen. Newer data centers employ UPS (uninterruptible power supplies) and gasoline or diesel powered generators.

Emergency Off

JR’s recent post about a failed UPS test reminded me of a similar event.

In the late 1980s I was employed as a software engineer / systems architect for a company that built casino management system software. We had a modestly sized raised floor computer room but no UPS or generator. We were not supporting customers from our computer room: it was merely our development and testing platform, and also used to support our customers all over the world.  We had a PDP-11/70 and a PDP-11/44 in there, together with a wide range of disk pack technologies. We also had a DELNI, but I’ll save that for another post.

One afternoon there were three or four of us in the computer room, chatting about something or other. We were all pretty relaxed, and Mike S was standing with his back to the wall.

Abruptly and without warning, the room suddently went dark and silent, and after a few moments, an emergency light came on.  We were all puzzled as to the reason for the power failure. We assumed that city power was cut off.

But then Mike turned around and looked at the wall. He had backed up against the emergency power off switch and apparently pressed it hard enough to activate it. He was not aware that it was even there.

Just then, the division chief came in and asked what had happened. We all turned to Mike, who fessed up. None of us knew how to reset the emergency off button, but Ted, the division chief, was familiar with it and showed us how. He reset the switch, and the data center roared to life. We booted the PDP-11’s and all was right with the world.

How’d the test go?

Most of us who started off operating computers, and later networks, had to also become the first facilities engineers for the rooms that housed our toys.  All over the United States in the 1990’s various broom closets, unused offices, and basement storage rooms became data centers for the emerging client-server (and subsequent dot com) explosion.

We all had to learn the intricacies of power distribution, cooling, rack spacing and placement, physical security, fire suppression, alarming and alerting, safety, and even how to organize our data centers for showing off to our new clients and prospects.

My first foray into managing such facilities was as a sysadmin (and later operations director) for a dot-com company that produced news and stock market information.  We had an office in a warehouse district building in Minneapolis that was over one hundred years old and had, at one time, been used as a parking garage for a Ford plant to park new trucks in.

Needless to say, this building was not engineered to be a data center by any stretch of the imagination.  Our office on the third floor became a data center by virtue of necessity, and many changes to the building had to be made to accommodate our thirst for processing power.  The original building electricity was fed by an ancient, and massive, 115 volt utility feed (versus the more common 230 today).  We had to have a 480 volt power system brought into the building, all whilst trying to preserve the historical significance of our former warehouse.  The first day crews came in and started boring 16″ holes in the concrete floor for a new power distribution system, I was thinking, “What have I gotten myself into?”.

After much discussion, and days of analysis on our needed power capabilities, we decided on an APC Silcon UPS system.  At the time (and to this day), this system was designed for ultra-high availability, and based upon our calculations it would run the data center for around 90 minutes without power.  Now this seems like not much time, and it isn’t, but this 90 minutes would allow us time to perform an orderly shutdown of the data center and transfer processing to our backup data center–which is all we needed.

After having the UPS system installed, we determined that we needed to test the system regularly while under actual load, which for us meant during stock market hours.  We decided that the first Friday of the month, in the early afternoon, we would test the UPS by shutting off the incoming power feed to simulate a power failure.  The procedure would then be to monitor how much runtime the UPS had, and monitor for situations such as a bad battery in the UPS or other conditions.

As the leader of the team, it was my responsibility to flip the switch.  There were three switches, one for the incoming power feed, one for the power feed leaving the UPS to feed the data center, and a switch in the middle to bypass the UPS and feed the data center directly from utility power.  For this test, the left-hand knob was turned with a satisfying “thump”.  The UPS would begin alarming that it was on battery power, we would take our measurements, and I would then restore utility power by closing the switch. 

Now, this test was not without consequence, and we all knew it.  An abrupt shutdown of our data center would bring our business to a halt, and we had service level agreements with clients that required us to pay them if we had excessive unavailability of our services. Because of the seriousness of this test, I would send an email out to the IT department prior to the test.  The email went something like this:

“Team, we are about ready to execute the monthly UPS system test.  If the test goes as planned, you will not notice anything.  If the test does not go as planned, you will likely get an email with my name in the subject line, indicating that I don’t work here anymore.”

The email always brought about a laugh from the team, and one August day, turned out to be nearly prophetic.  As I walked into the data center to begin the test, I failed to look at the small display on the front of the UPS system.  As it turns out, the UPS had experienced a failure of a couple of $20 fuses that connected the battery cabinet to the UPS controller, and this small two-line LCD display would have told me of this condition had I looked at it-but I didn’t. 

I walked up to the switch, dutifully turned it, and the deafening sound of a silent data center was instant.  Data center down.  I stood by the panel for what seemed like an eternity just as the rest of the team was walking in to investigate.  We were all horrified.

We began the slow procedure of bringing systems back online, and this had to be conducted in a particular order:  Network switches first, then routers, load balancers, domain controllers, databases, application servers, and finally web servers.  The process took several hours.

Afterwards, I walked into the office of my boss (his name was Jamie).  With a lump in my throat, I explained to him what I did and that I realized that I didn’t observe the control panel on the front of the UPS before flipping the switch. I asked if he was going to fire me. With a smile on his face, he replied, “No, why would I do that?  I just invested hundreds of thousands of dollars into your education.”  It was a very compassionate moment from a man who had every reason to be upset with me, and I was (and am) very grateful for his kindness during my mistake.

I made my way back to my computer after one of the most terrible and exhausting days.  Upon opening my email box, there was a “reply all” to my original test announcement from one of my coworkers.  It was one line:

“How’d the test go?”

My response:

“The UPS test is complete.  No further testing is required at this time.”

There is no sound louder than a quiet data center.