Liveblogging: Senior Skills: Sysadmin Patterns

The Beacon Pattern:
– This is a “Get out of the business” pattern
– Identify an oft-occurring and annoying task
– Automate and document it to the point of being able to hand it off to someone far less technical

Example:
– System admins were being put in charge of scheduling rooms in the building
– They wrote a PHP web application to help them automate the task
– They refined the app, documented how to use it, and handed it off to a secretary
– They have to maintain the app, but it’s far less work.

The Community Pattern:

– Prior to launch of a new service, create user documentation for it.
– Point a few early adopters at the documentation and see if they can use the service with minimal support
– Use feedback to improve documentation, and the service
– Upon launch, create a mailing list, forum, IRC channel, or Jabber chat room and ask early adopters to help test it out.
– Upon launch, your early adopters are the community, and they’ll tell new users to use the tools you’ve provided instead of calling you.

Example:
– A beowulf cluster for an academic department
– Documented like crazy, early adopters were given early access to the cluster (demand was high)
– Crated a mailing list, early adopters were added to it with their consent, functionality was tested with them.
– Email announcing launch mentioned the early adopters in a ‘thank you’ secion, and linked them to their mailing list.

The DRY pattern
DRY = Don’t repeat yourself
Identify duplicate code in your automation scripts
Put subroutines that exist in an include file, and include them in your scripts.

Example:
– “sysadmin library”
– /var/lib/adm/.*pl
– Elapsed time and # of lines to script a task for which the library was useful plunged dramatically
– new tasks were thought up that were not considered before but were obvious now (ie, users that want to change their username)
– migrating to new services became much easier

The Chameleon Pattern
– Identify commonalities among your services
– Leverage those to create “Chameleon” servers that can be re-purposed on the fly
– Abstract as much of this away from the physical hardware
– Doesn’t need to involve virtualization, though it’s awfully handy if you can do it that way.
[this one is a bit harder to do with MySQL config files]

Example:
[puppet/cfengine were mentioned…]
ldapconfig.py – more than a script: a methodology

– But isn’t installing packages you don’t need bad? Depends on the package….ie, gcc is bad for enterprise

“Junior annoynances”

Terminal issues

Junior:
open terminal, login to machine1
think issue is with machine2, talks to machine1.
log out of machine1
log into machine2

Senior:
opens 2 terminals each of machine1 and machine2 to start

Junior:
networking issue ticket arrives
logs into server
runs tcpdump

Senior:
networking issue ticket arrives
logs into server
looks at logs

“Fix” vs. “Solution” ie “taking orders”
Junior will try fix a problem, senior will try to figure out what the problem is. ie, “I need a samba directory mounted under an NFS mount” a junior admin will try to do exactly that, a senior admin will ask “what are you trying to do with that?” because maybe all they need is a symlink.

Fanboyism
Signs you might be a fanboy:
– Disparaging users of latest stable release of $THING for not using the nightly (unstable) build which fixes more issues
– Creating false/invalid comparisons based on popular opinion instead of experience/facts
– Going against internal standards, breaking environmental consistency, to use $THING instead of $STANDARD (but this is also how disruptive technology works)
– Being in complete denial that most technology at some point or another stinks.
– Evaluating solutions based on “I like” instead of “we need” and “this does”.

The Beacon Pattern:
– This is a “Get out of the business” pattern
– Identify an oft-occurring and annoying task
– Automate and document it to the point of being able to hand it off to someone far less technical

Example:
– System admins were being put in charge of scheduling rooms in the building
– They wrote a PHP web application to help them automate the task
– They refined the app, documented how to use it, and handed it off to a secretary
– They have to maintain the app, but it’s far less work.

The Community Pattern:

– Prior to launch of a new service, create user documentation for it.
– Point a few early adopters at the documentation and see if they can use the service with minimal support
– Use feedback to improve documentation, and the service
– Upon launch, create a mailing list, forum, IRC channel, or Jabber chat room and ask early adopters to help test it out.
– Upon launch, your early adopters are the community, and they’ll tell new users to use the tools you’ve provided instead of calling you.

Example:
– A beowulf cluster for an academic department
– Documented like crazy, early adopters were given early access to the cluster (demand was high)
– Crated a mailing list, early adopters were added to it with their consent, functionality was tested with them.
– Email announcing launch mentioned the early adopters in a ‘thank you’ secion, and linked them to their mailing list.

The DRY pattern
DRY = Don’t repeat yourself
Identify duplicate code in your automation scripts
Put subroutines that exist in an include file, and include them in your scripts.

Example:
– “sysadmin library”
– /var/lib/adm/.*pl
– Elapsed time and # of lines to script a task for which the library was useful plunged dramatically
– new tasks were thought up that were not considered before but were obvious now (ie, users that want to change their username)
– migrating to new services became much easier

The Chameleon Pattern
– Identify commonalities among your services
– Leverage those to create “Chameleon” servers that can be re-purposed on the fly
– Abstract as much of this away from the physical hardware
– Doesn’t need to involve virtualization, though it’s awfully handy if you can do it that way.
[this one is a bit harder to do with MySQL config files]

Example:
[puppet/cfengine were mentioned…]
ldapconfig.py – more than a script: a methodology

– But isn’t installing packages you don’t need bad? Depends on the package….ie, gcc is bad for enterprise

“Junior annoynances”

Terminal issues

Junior:
open terminal, login to machine1
think issue is with machine2, talks to machine1.
log out of machine1
log into machine2

Senior:
opens 2 terminals each of machine1 and machine2 to start

Junior:
networking issue ticket arrives
logs into server
runs tcpdump

Senior:
networking issue ticket arrives
logs into server
looks at logs

“Fix” vs. “Solution” ie “taking orders”
Junior will try fix a problem, senior will try to figure out what the problem is. ie, “I need a samba directory mounted under an NFS mount” a junior admin will try to do exactly that, a senior admin will ask “what are you trying to do with that?” because maybe all they need is a symlink.

Fanboyism
Signs you might be a fanboy:
– Disparaging users of latest stable release of $THING for not using the nightly (unstable) build which fixes more issues
– Creating false/invalid comparisons based on popular opinion instead of experience/facts
– Going against internal standards, breaking environmental consistency, to use $THING instead of $STANDARD (but this is also how disruptive technology works)
– Being in complete denial that most technology at some point or another stinks.
– Evaluating solutions based on “I like” instead of “we need” and “this does”.