Being a sysadmin is not sexy (part 2)

In the previous post we learned a bit about servers and sysadmins, and how the sysadmin’s job is usually fairly unexciting.

But then occasionally something will go wrong. A server application which was working perfectly well five minutes ago is now giving the users nothing but error messages. This is when the job becomes terrifying, because some users feel compelled to explain to the sysadmin how (s)he has personally failed them. The phone starts ringing off the hook, the email inbox fills up in a hurry, and people stop by the sysadmin’s desk to point out the painfully obvious fact that the system is down. Boredom is far preferable to this.

When things go off the rails, there are a number of things the sysadmin can do to diagnose the problem:

  • She might look at a list of active processes running on the server to see if something important is missing. Sometimes a service will stop for some reason, and simply restarting the service will get things moving again. After everyone settles down a bit, the sysadmin can try to figure why the damn thing stopped in the first place.
  • A program called a packet-sniffer can help analyze the server’s network connections. It could be that something about the network (completely external to the server itself) has changed, and that this is causing connectivity problems. This is the sysadmin’s favorite explanation, because everything immediately becomes someone else’s problem, and it gives the sysadmin an excuse to go yell at the network pukes.
  • Log files may be the most common diagnostic tool. If the server application experiences some kind of problem, it will hopefully write a useful message to a log file. Often (not always, but often) looking at the log file will reveal the problem, and hopefully there will be a straightforward solution that the sysadmin can apply promptly. Getting things back to normal in a hurry is certainly a priority in these situations, but it’s not always that easy. Sometimes it takes a while to diagnose a problem, and the solution may require unscheduled downtime.

The adage about an ounce of prevention governs the work of an experienced sysadmin, who will expend no small amount of effort putting a lot of canaries in the coal mines. A big part of this job is avoiding common or recurring problems. Examples of this might include some of the following:

  • Setting up a process that emails the sysadmin when a hard drive starts to run out of space.
  • Reviewing log files every day. (This is deadly dull, but sometimes it identifies problems before they break things.)
  • Keeping a detailed list of upgrades and configuration changes so you can put stuff back the way it was days or weeks later.

Anyone who has been doing this kind of work for a while has horror stories, and I have a few of my own. I’ll write those up as short posts from time to time.

Advertisements

2015-12-28 Adobe updates

Adobe has released updates to Adobe Flash Player. This update addresses critical security problems. If you you have Flash Player installed on your computer (which is likely), please update it.

Looks like if you let Firefox and Google Chrome update themselves, that may be enough to update Flash Player in those browsers. Otherwise, the Adobe Flash Player page can tell you if you need an update (you may want to visit this page in each web browser you use).

For more technical information, see the Adobe security bulletin.

This bulletin also includes an update for Adobe AIR. If you think AIR is installed on your computer, here’s the Adobe AIR page.

Being a sysadmin is not sexy (part 1)

A sysadmin is someone whose job is the care and feeding of one or more servers (sysadmins frequently look after several servers). This is a job which is mostly boring (day-to-day stuff) but which is occasionally terrifying (when something unexpectedly stops working). There’s often not a lot in between those two things. If you are writing a story with a character who does this kind of work, don’t portray them as someone whose job is constant excitement, because no one will believe that.

A sysadmin’s celebrity among the non-technology people in the organization is often inversely proportional to how well she does her job. If this sysadmin is effective at her work, people tend not to know who she is. If that sysadmin is not good at her job, many people tend to know her name. This is a job where anonymity can be desirable.

Servers tend to run server applications (also known as services), which can be all kinds of things:

  1. A server might run an application commonly called a web server (Apache and Microsoft’s IIS are popular web servers). You typically interact with this service through your web browser.
  2. A server might be a file server. If you’ve ever used the “map a network drive” feature, then you’ve downloaded files from (or uploaded files to) a file server.
  3. If your organization runs a centralized accounting system, then that accounting system likely runs as a service on one or more servers. You might interact with this service through a desktop application or even a sophisticated web-based interface.
  4. email is another good example of a high-profile server application. Although users interact with email through a single desktop application (like Outlook), email usually happens by way of a pair of services: one for sending and receiving email (to and from other email servers), and one for retrieving email messages from the server to the desktop application.

A single server might run several different services.

Ideally a server and its server applications run smoothly and don’t need a lot of help beyond ordinary maintenance. This is the relatively tedious part of the job:

  • The server requires backups, which are frequently automated processes.
  • If the sysadmin manages a complicated application running on a server, that application may have many users. As people come and go, there’s a lot of giving this new person access to that, taking away that person’s access because they left, etc.
  • The server and its applications often have upgrades become available. This is a never-ending chore. It’s not uncommon for the organization to schedule a maintenance period during which the services will not be available to users. This is called “downtime,” and it is unpopular with users. The sysadmins use this time to apply the updates. This is typically routine, but can be stressful, because it can be difficult to recover from a failed upgrade (this may require restoring data and/or the operating system from a backup). These maintenance periods are commonly at night, over a weekend, or even during a holiday, which is not a fun time for the sysadmin to have to work.

In the next post we’ll look at what happens when things don’t go smoothly.

The worst explanation of networking, ever

(This post is going to introduce a lot of jargon, and I’ll probably refer to it from future posts.)

Making a network connection to a computer involves an IP address (the computer’s address on the internet) and a port number. This is a flimsy analogy, but you could think of it like finding your way into a house: the IP address (the host) is like the street address, and the port number is like which entrance to use (the front door, or the window on east side).

For example, most web sites are on port 80 or 443: 80 is the standard port for web servers, and 443 is the standard port for a secure web site (HTTPS). When you want your browser to display the CNN homepage, your browser figures out the IP address for http://www.cnn.com (which at the moment appears to be 23.235.44.73), connects to port 80 on that host, and begins an HTTP transaction to download the home page.

A firewall is network software that allows or rejects network traffic according to a set of rules. An organization which hosts its own web site might have a firewall which allows the Internet to connect to the web server on allowed ports like 80 and 443, but the firewall would reject other inbound traffic to that host.

A port scanner is a program used to interrogate a host’s ports, looking for services listening on those ports. If you point a port scanner at a web server, it’s likely to tell you that a web service is listening on port 80 (and maybe also on port 443). Port scanners are powerful programs which can be used in many ways. A company might hire a security analyst who would use a port scanner to identify weak points on the company’s network (a good analyst would also give the company some suggestions about how to address those shortcomings). A malicious person might use a port scanner to the same effect but for a different reason: the port scanner can tell the attacker the ports on which services are listening on the company’s hosts, identifying targets he or she might try to compromise.

Sophisticated port scanners can identify specific software products listening on a port, and even the version of that software. If a cybercriminal used a port scanner to determine that a host was running version 2.2.15 of the Apache web server, he could use a search engine to look for vulnerabilities in that version of that product. He might find that that version has a race condition error resulting in a remote execution vulnerability and that someone has published an exploit which takes advantage of that software defect. He could download the exploit and run it against the web server for any number of malicious purposes (like stealing otherwise inaccessible information, sending spam and phishing attacks, etc.).

nmap has been a popular port scanner for a long time (it was first released in 1997). In The Matrix Reloaded, Trinity uses nmap to port scan a host she wants to compromise. nmap identifies a vulnerable version of a network service called ssh, and she then uses an exploit to elevate her privileges on that host. This is a good (and rare) example of credible hacking in popular media: the writer(s) included enough realistic detail to make the scene plausible.

nmap has starred in lots of films.

Statement of purpose

For years I’ve had a ridiculous fantasy of being a fiction writer. It seems that the best-selling novel I want to have written isn’t going to write itself. I’m having trouble getting motivated, so maybe what I need is another distraction: a blog.

I thought that technology in writing might be an interesting theme. Nothing ruins a story for me faster than a character hacking the FBI network after tapping on a keyboard for ten seconds. It probably works for many readers/viewers, but some of us see it as lazy writing.

In my day job I write lots of web applications for a public university. Many of my assignments are to convert paper processes into online forms. My job also involves a fair bit of Linux server administration. Most of this goes on the open Internet and is subject to daily cyber-attacks from all over the world (my server logs once revealed malicious traffic from Antarctica).

So the purpose of this blog has a couple of goals. One is to get me in the habit of writing. But I thought it might be useful to share some of what I’ve learned in a format that may be helpful to other prospective writers. I may also write about how technology can affect a writer. Here are some topics I have in mind:

  • credible hacking
    • port scanning
    • realistic exploitable security vulnerabilities
    • case studies of actual security breaches (like Target)
  • a writer’s technology
    • safe(r) Internet use (account security, security-related Firefox extensions, password managers)
    • affordable and effective backups
    • writing tools like scrivener and wordpress (I know a fair bit about the latter and would like to learn more about the former)
  • the day-to-day life of a web programmer
    • server administration is not sexy
    • the importance (and challenge) of making web sites accessible
    • the horrors of working with vendors and ticketing systems

This blog may at times earn a PG-13 rating. I’ll mostly keep it clean, but there may be the occasional bit of salty language.

I’ll try to post every seven to fourteen days (historically I’ve really struggled with self-imposed routines like that), and I’ll try to keep individual posts fairly short (preferring to break up longer topics into multiple posts).