Friday, March 24, 2006

Article on Hollistic Network Security

Ran across this article on Security Focus..It's a decent set of pointers to understanding a more hollistic view of network security. Rather than wholly focusing on just looking at the Layers 1-3, it emplores learning programming concepts and patterns. By understanding how these concepts and patterns are used, it is much easier to realize deficiencies in these patterns in Layers 4-7. I suggest any Systems or Network Administrator take a look.

Monday, February 06, 2006

Awesome Personal Colo Deal

My friend Michael, who writes the "Life and Times of an Infrastructure Architect" blog at has a post about an awesome deal on personal colo. Read about it at -- His company, BitPusher, is a great resource for outsourcing your IT and production infrastructures. You should check it out.

Wednesday, December 28, 2005

My next non-work related projects :)

I've decided to start designing and eventually (read: when I'm not broke) implement a mobile Internet access solution in my Pimp Ass Ride(tm). Not so much for the hell of it, but in order to put an MP3 jukebox in there as well. I guess that DOES qualifies as "for the hell of it", but oh well. The plans are to set up the MP3 server to serve random playlists from Ampache through some sort of FM Modulator. There's a couple tricky parts, mainly, how do I maintain power, or enough power to the mp3 server so that it doesn't just die when I shut the car off and doesn't drain too much power from the battery..also, how do I get the wifi card to prefer my hope access point when it's close enough to it (in the driveway). The main reason I want to be able to do that is to be able to use Rsync to grab whatever is on my home MP3 server when it can. Oh stuff.

The other fun project that I'm working on is restoring an original Street Fighter 2 cabinet as a combination MAME box and MP3 Jukebox. My plans are to put a mini-atx case in the coin deposit assembly, and mount one of my old stereo amplifiers and a couple speakers in a logistically nice fashion. This one's gonna take a bit more planning, but I think it'll be a lot of fun.

Oh well..enough from me, yo voy a dormir.

Tuesday, December 27, 2005

Where do we go from here?

I've been wondering for a while now what I.T. will look like over the next few years. The emergence of reasonably priced shared service providers in the I.T. space is starting to make dinosaurs of the traditional hosting paradigm. This, plus a movement towards normalization of skills across a wide base of employees in a fairly large industry, makes me wonder how much legs the traditional systems or network administration role truly has.

One of the interesting things about the "Web 2.0" craze, is that it underlines how seriously far behind the curve most of I.T. is. Ask most systems administrators how to secure a site that uses AJAX from cross site scripting, or why PHP 5 could be safer from a code standpoint than PHP 4 for front end development, and they'll stare at you menacingly and/or blankly. This isn't necessarily the administrators fault. There are much newer paradigms in play today then when most qualified I.T. players got in the game. Systems administrators have a whole new mess dropped on their lap with these new technologies, and unfortunately, until they catch up, the development community is supporting themselves for the most part.

As an aside, one thing that I think needs to be reinforced in our community is that, in the current industry landscape, systems and network administrators can not survive without developers to support. There simply are precious few I.T.-centric shops out there anymore. If developers and the development community are the only sources of support for a new technology, the I.T. world will become somewhat obsolete (This doesn't include desktop or end-user support of course, which will probably still be around for a bit longer.)

You may ask yourself, depending on what side of the technology boat you're on, whether or not developer self-support is a good or bad thing. I have a feeling that most developers prefer not to have an administrator for the most part (until something breaks). Of course, no I.T. employee in their right mind would say that getting rid of their job would be a good thing, right?

I think this is where we as a community need to take a hard look at what our skills actually mean to a company's ongoing success to answer that question. Does mucking with development environments make us important? Do checking permissions make us relevant?

My answer is no, our skills are needed in other areas, and the quicker we make our way there, the more important and relevant each of us can be. Here are my ideas for moving to the next I.T. paradigm.


1. Outsource all or most of your companies tasks and projects that would not be helped by keeping them in-house.

Obviously, if your company does something like I.T. Asset Management as boxed software, it would be silly to outsource tracking your inventory. However, does it really make sense for that same company to be managing their own hardware repair, or production systems management? With the overall cost of shared services coming to earth, outsourcing your non-essentials is more and more viable.

2. Become procedure-oriented and policy focused, not issue-oriented and operationally focused.

By documenting and following your procedures and policies, you become integral to the implementation, management and growth of your outsourced services. By moving away from operational issues, you become relevant to the strategic growth of your department and company.

3. Learn relevant skills for the current wave of needs.

Ten years ago, while Technical Writing was important, it wasn't necessary for all companies. If your company ever has the desire to go public, get funded, or get purchased, writing technical documents is almost a necessity. Legislators have finally figured I.T. out (well, to a certain extent) and are requiring more and more well prepared documentation. If you are a grammatical wizard, and understand technology, your skills are going to be extremely important and relevant.

Security focused administration is a huge field as well. This is not simply knowing how to build a good packet filter ruleset. It is important to look at the entirety of your company's liabilities. How much sense does it make to successfully track which files have changed on a system and which users are attempting to access your network, when a malformed POST string could inject SQL and allow malicious users to manipulate your database?

My big hope is that a new field, which for now I'll call Software Scaling and Support, takes off. People in this field are well versed in the technologies that the development team is using, and how best to both scale and support this technology. This field would require a fair amount of knowledge of enterprise software patterns and concepts, enterprise administration, and infrastructure architecture. Eventually, this type of function could probably end up being outsourced as well, but due to the tight integration necessary with an internal development team, is more likely to be in-house.

4. Get out in the community.

One thing that is interestingly lacking, considering the proliferation of user groups, conferences, and seminars, is the feeling of community that seems to pervade other parts of the technology industry. How many people can name the guy who co-founded Blogger/Pyra back in the day? A lot more people than can name the person who designed and built the first Apple G5-based supercomputer cluster back in 2003 (the answers are Evan Williams and Srinidhi Varadarajan, respectively.)

We should be looking to help new entrants into our profession, as it only allows us to move up and on to better things. Having a healthy and vibrant I.T. community helps everyone learn, grow, and network with many people they would otherwise never meet. It's an important tactic that our counterparts in software development have used to create some of the coolest new technology. We should be doing the same.

Tips and Tricks For the Career I.T. Person

People that are starting their journey into making systems and network administration a career are in an interesting bind.

If you know a little about a lot of different things, you increase your employment potential, but limit your overall income potential. On the other hand, you could pigeonhole yourself with extremely specific knowledge and possibly get paid quite nicely if you can ever land a long term gig. But what company in their right mind will hire, long term, someone who knows EVERYTHING about a specific MTA but has little to no working knowledge of LDAP, Kerberos, SMTP, IMAP, DNS or drop-in mail appliances that best both commercial and open source solutions in terms of maintainability, ease of support, and ease of configuration? Eventually, whatever a specialist specializes in becomes either an extremely small niche market or an obsolete market.

I saw, through my father, the pain that mainframe operators had to go through in order to stay relevant in the late 80's-mid 90's. Through hard work and sheer luck, he was able to keep himself relevant, but the same cannot be said for most of his peers. The ones still in I.T. are still punching cards, unloading reel to reel tapes, and hoping that their current hardware doesn't crash, because there's only so many spare parts you can have on hand and it's kind of hard to get support on a machine that's 30 years out of production anymore.

Now, most current I.T. employees fall somewhere in the middle. Most lean towards a broader knowledge of technologies and add to that a specific focus that makes them unique.

Well, guess what? Most companies don't want uniqueness. Companies are looking for reproducibility, redundancy, and ways to ensure that some random geek doesn't forget to oil their well oiled machine.

So, what can a someone who wants to be in this field long term do? I'm glad you asked, because I'm about to post my list of survival tips for beginning to intermediate I.T. workers. These tips are not in any significant order.

1. Clean up your act

I understand that most people that are reading this document are not in the 'underarm-stain troll in the corner' category of administrators. If you are, however, make your best attempt to clean up your look and dress in the same fashion as other employees in your company. This isn't about individualism, it's about your CEO not being embarrassed to introduce you to investors.

You may think that your 15 year old dreadlocks, I Love Midget Porn t-shirt, ratty shorts and sockless feet make you look like a 7337 h4x0r. Nope. For most companies, it's inappropriate work attire at best, and at worst looks amateurish to people that may be paying your wages through a large capital investment.

2. Learn How to Write and Document

No one is asking you to become a prolific or perfect writer here. I know I have problems with my writing style, that I'm a decent but not perfect speller, and when writing casually, I have a problem with keeping with whatever tense I am in.

Here's a few tips, however, to make your documentation easier to write, understand, and maintain.

a. Spell out any acronym the first time you use it. This gives the reader an idea of what the text means before you start spouting random TLA (Three Letter Acronyms) at them.

b. Write as if you are completing a book report for your 6th grade class. It's OK to be verbose, and it's OK to explain everything. Just because people in your department may understand your more complex documentation does not mean that it is accessible to your executive team. Due to legislation like Sarbannes-Oxley, SAS 70, and CISP, your documentation may be used for legal purposes. Make sure it's clear and easily readable, even by someone not as versed in the subject as you.

c. SPELLCHECK! No one's spelling is perfect, but documentation looks much more professional when common words are not misspelled.

3. Automate Everything

It is not an effective use of your time to repeatedly do the same work. Look for patterns in your tasks, and determine how they can be grouped and repeated in an automated fashion. This may mean breaking up very large tasks into a series of smaller interdependent tasks. It may mean grouping many small tasks into one larger script. With any automated task, the goal is to minimize exceptions to your pattern, however, exceptions WILL exist. Make sure that the amount of exceptions isn't higher than the amount of matches to your pattern, and also deal with these exceptions cleanly.

Remember also that while this automated process may still be running three years from now, you may not be in a role where you will be supporting it at that time. Make sure that you document how your automation works, your reasons for doing things certain ways, and that however you automate, that it's done in a way that it can be easily maintained, replaced, and supported without any detrimental effect to your business.

4. Learn How You Learn

I personally believe that the way that people learn stems from their physical senses and which of those senses is most developed. Some people are tactile (hands-on), others aural, or visual. Smell and taste aren't typically needed, however, it can help to know what fried electronics smell like.

Additionally, another thing to note is how you associate things in your memory. Some very hands-on learners can learn most quickly through tactile means, but can only remember through visually associating what they've learned with something they already know. Some people who are extremely visual learners only remember something if they also talk to someone about it.

Try to teach yourself a new skill through the three main methods (hands on, aural, visual) and see which method, or combination of methods, work best for you. Also, try different methods for remembering key points about your new skill. Then make sure that when teaching yourself that you try to use the same combination of methods. This works much better than forcing yourself to learn through means that simply don't match up with your particular needs.

5. Constantly Research

There are nearly an unlimited amount of research options available to I.T. workers. Once you know how you learn, determine what avenues are best for your style of learning and go for it. If you are a visual learner, read articles, web pages, or books. If you're aural, there are podcasts and webinars. If you're tactile, look for user groups, seminars, work groups, etc. Not knowing about new technologies doesn't get you off the hook from having to support them.

6. Learn Programming Concepts

Learning programming concepts will help you support your developer base, ease your automation tasks, and help with your ability to identify patterns in your workload. Learning more about how your developer base works makes supporting their work easier, and allows you to contribute to a supportable and maintainable development environment.

Once you do this, those seemingly random error messages might start making sense. A more concrete example is learning how the programming language C works. A basic knowledge of system calls and a tool like strace or truss pulls the curtain from in front of the wizard, allowing you to dig very deep into what your system is doing at a very low level.

7. Know What You Can and Can't Do

It is extremely important to know your limitations, because no one else will know unless you tell them. It is not a sign of weakness to not know something, it is simply human.

Let's say your CEO calls you up and demands that you fix a bug in the code that supports your production website. First question that you should ask yourself is whether or not you are the right person to make this change. If not, you should know who your next call is to, and how to get a hold of them. Don't let your ego get in the way here, or you will find yourself in a load of trouble. The worst thing that you can do is to start digging in without knowing what you are doing first. Instead of people rewarding you for your heroic efforts in learning PHP 5 in an hour, they will be much more upset that you didn't call someone who could have fixed the actual problem in 5 minutes.

8. Manage Your Time and Communications Effectively

Time and communications management make the difference in being an I.T. worker and being a professional. Being able to commit to timelines and meet them make you far more valuable than someone that simply is more well versed in technology. Along with this comes the need to communicate your status on your commitments to interested parties at times when it is most convenient for THEM. This may make your project or task take longer, so make sure to factor the time it takes to communicate into any project or task plan. Make sure that you communicate in a fashion that makes sense to your interested parties. An update to your CFO that "LDAP was wonky and caused ppl from to be unable to get to QB" is probably not appropriate. Try using some of the documentation conventions from Tip #2 to make this something that is more appropriate for a larger audience than yourself, like "Directory Services were unstable and caused users in the office to be unable to start Quickbooks."

9. Constant Heroics Ruin Companies

You may be an extremely hard worker, always available, and constantly being the "hero", fixing huge problems through Herculean effort. This sounds great, you get spot bonuses and free dinners from the CEO. However, they are simply feeding a cycle that can literally ruin your department's reputation, and possibly your company's viability.

It is important to note what failures are causing your heroics to be needed and start planning immediately how to resolve it long term so that additional heroics aren't needed the next time something major breaks. Your job is to make this process easier.

10. Broaden Your Horizons

Take a step back from your day to day work and look at what you actually thrive and enjoy doing. Are you better at managing schedules and communications than automating your tasks? You may be a very good fit in a managerial or project manager role. Do you find that your documentation is top notch? You could possibly turn this administration role into a technical writing role, which is becoming increasingly important with the current state of legislation concerning I.T. practices. Do you feel that your technical acumen is up to snuff, and that you really enjoy making architectural decisions? You may find that being an Infrastructure Architect is to your liking.


The most important thing here is to always continue to learn and better yourself. It's the only real way to keep yourself relevant in this industry long term.