Douglas K. O'Leary
Objective: Senior System Administrator; increase company's productivity through efficient management & professional network/system/database administration
Home   Phone   Cell
dkoleary@olearycomputers.com   630-904-6098   630-248-2749


Hardware/OS Experience (20 yrs):

Vendor Hardware OS # of years
HP HP 9000/800, ia64s & ia64 blades HPUX 9.04/5, 10.X, 11.X
17
Linux Various RHEL4/5/6, Fedora, Ubuntu, Mandrake, etc
15
SUN Enterprise 10,000/4500/5500 SUNOS 4.1.3, Solaris 2.5/6/7/8/9
14
SGI Challenge XL, Origin 2000 Irix 5.3/6.X
3
IBM RS6000 980, 990, R24, Power PCs AIX 3.25, 4.1, 4.3, 5.X
5



Software/Utilities

Perl
ksh
expect
HTML
CGI
sqlplus
OSSEC
MC ServiceGuard
Redhat KVM
SNMP
SMTP/sendmail
Veritas Volume Manager
Openssh
Openssl
Openvpn
Openldap
Veritas Netbackup
F-Secure ssh
stunnel
HP Npar,Vpar, and HPVM
 
Oracle 10g
Oracle RMAN
AWS



Certifications
  1. Certified Information System Security Professional (CISSP)
  2. Certified Information Systems Auditor (CISA)
  3. Certified Ethical Hacker (C|EH)
  4. Oracle Certified Professional (10g)
  5. HP Star Certified Technical Professional (HP-UX 11.0)
  6. SUN Certified System Adminstrator (Solaris 7 & 8)
  7. SUN Certified Network Administrator (Solaris 8)

Work Experience:


Company O'Leary Computer Enterprises
Title Senior Unix Admin
Time Frame 08/01 - present





  1. Multiplan: Assisted w/unboundID/LDAP support. Wrote scripts to automate the mundane tasks of ldap maintenance. Reviewed corporate security policy. Provided numerous security proposals including 2-factor authentication to root/administrative accounts, distribution of sensitive security information via gpg encrypted git repos, and standards for Linux hardening. Audited use of secure shell keys across an 800 server environment. Generated a demonstration of enterprise class configuration management using puppet and several Amazon Web Services (AWS) systems. Designed, documented, and implemented an operational readiness test (ORT) to ensure systems are mission ready and configured to standards. Designed, developed, documented, and implemented process for web configuration file updates that approproate teams can use to support their applications w/o root/unix team interaction.

  2. TransUnion: Assisted in various projects to achieve PCI compliance. Performed work as lead engineer on Arcsight SIEM expansion and analytics projects. Also provided advice on UNIX/Linux PCI security implementations, researched software vs LVM mirrioring, and developed/documented a plethora of Linux based procedures.

  3. Jewish United Fund: Performed security assessment for HP-UX environment, including file/directory permissions, account controls, script review, and network and OS hardening. The final report also included recommendations on increasing security posture to be more in-line with PCI DSS standards and, by request, recommendations and possible plans to migrate from HPUX to Linux. Developed a high level plan to remediate the findings. Re-engineered backup environment, eliminating manual process and faulty backup assumption.

  4. XSell: Infrastructure support for cloud based concierge startup. Consulted on and implemented security for rhel6 IaaS systems. Designed process to quickly and easily generate new systems with all requisite security hardening and software already done and/or installed

  5. Bradford Group: Installed, configured, and documented installation of Host Based Intrusion Detection system (HIDS), OSSEC. Trained client personnel on procedures for maintenance. Consulted on interpretation of PCI DSS v2 standards and how they apply to UNIX environments. Drafted procedures for alert remediation which accounts for segregation of duties and reduces the chances for collusion.

  6. Multiplan: Assisted in a short term datacenter migration to include Linux system builds and configuration, data copies, and troubleshooting. Researched, documented, and migrated unsupported, mission critical application which sends and receives files from external clients. Documented inputs, outputs to the application as well as the hundreds of scripts that this application calls and/or interfaces.

  7. Alcatel-Lucent Provided 24x7 support over a seven year contract to SAP environment running under Oracle on HPs and a variety of other database servers running on both HPs and SUN systems. Support included datacenter and data migrations, security and Sarbanes-Oxley (SOX) remediation, OS and MC Serviceguard cluster upgrades, software, hardware, and OS maintenance, develop, document, and implement automation and procedures including OS upgrades and installs and disaster recovery.
    1. Oracle
      1. Troubleshot listener issues that arose upon package failover.
      2. Assisted in production Oracle performance troubleshooting. Issue was finally determined to be corrupted statistics.
      3. Recovered Oracle 10g production database during DR exercise.
    1. Assisted in three datacenter migrations.
      1. Planned, scripted, and implemented host data migrations to a staging EMC symmetrix
      2. Updated MC Serviceguard (MCSG) cluster configurations for systems that had volume group renames or consolidations
      3. Installed/configured new systems in target location.
      4. Imported volume groups onto new systems after SRDF transfer.
    2. MC Serviceguard
      1. Documented/troubleshot poorly built and poorly maintained clusters prior to their migration
      2. Generated new clusters to replace old ones:
        1. 6 2-node clusters, 1 3-node cluster, and 1 5-node cluster
        2. Designed/implemented standard for package fstab files
        3. Designed/implemented stardard for package exports
        4. Designed/implemented functions for automounter updates
        5. Upgraded OS and MCSG version on 6-node SAP production cluster
        6. Developed/published web based package/cluster documentation
    3. Security
      1. Participated in minimum of two security audits per year. Wrote scripts to collect evidence for audit responses. Most of the scripts were used by security/tripwire team to 'preanswer' questions for future audits.
      2. openssh:
        1. Generated white paper on benefits of 2-factor ssh/public key authentication (ssh/pka) over sudo/password access to root and other sensitive accounts. This documentation became the basis for ssh/pka being the preferred method of root access
        2. Configured ssh on all systems to log ssh/pka key fingerprints used to access the systems and to prevent direct root password based access via ssh.
        3. Designed, documented, and implemented forced commands to log what commands were executed as root via ssh/pka
        4. Eliminated uucp in favor of ssh/pka
      3. Configured centralized syslog server using syslog-ng
        1. Wrote/implemented scripts to monitor access to root and other sensitive accounts.
        2. Wrote scripts to monitor HP EMS hardware alerts.
      4. Implemented stunnel for encrypted sqlnet traffic between Oracle servers supporting financial transactions. Implementation included generating a company certificate authority to sign certificate requests, developing, documenting procedures for generating certificate requests, and developing, documenting procedures for configuring and testing stunnel implementations.
      5. Patching
        1. Designed and implemented process for routine semi-annual patching exercises
        2. Wrote scripts to automate patch analyses and patch staging using HP's new software assistant. The time required to run patch analyses dropped from three weeks to two hours for the entire HP environment.
    4. Disaster recovery
      1. Designed, tested, and documented
        1. procedures for creating a DR ignite server.
        2. procedures for cloning production systems
        3. procedures for creating alternate boot disks from copies of the DR ignite server's OS volume group
      2. Procedures tested/validated on three separate disaster recovery exercises.
      3. Wrote and implemented scripts to routinely collect DR information from critical production systems
    5. Scripts
      1. ksh script to automatically update cluster based vg maps on adoptive nodes and backup servers. Script also automatically exports/imports vgs on adoptive nodes as appropriate.
      2. ksh script to automatically run make_net_recovery commands as needed.
      3. perl script to automatically create/update/query company directory service as needed.
      4. perl script to automate user and root password changes enterprise wide.
      5. perl script to identify when account passwords are about to expire, identify the email address of the account owner via the company directory service, and send them a warning email.
      6. ksh scripts to automate the LVM data collection and analysis for Oracle database refreshes.
      7. ksh script to identify which LVM disks are used in vgs, which aren't, and which aren't but are supposed to be. Helps eliminate data corruption when a inappropriate pvlinked disk is used in a new volume group
      8. perl script to parse the output of emc's inq command and combine it with LVM/VXVM information to show EMC disks on the system and how they're being used

  8. Bank One: Provided 24x7 support to Peoplesoft/SAP environment running under Oracle on HPs of various classes. Support included security scanning and remediation, scripting, software and hardware maintenance, operating system upgrades, backup and system troubleshooting, and disaster recovery
    1. Security administrator for SAP/Peoplesoft Environments:
      1. Responsible for monthly ISS scan remediation for 50+ HP systems.
      2. Eliminated rhost implementation.
    2. Implemented secure shell
      1. Compiled/distributed installation
      2. Configured public/private key authentication
      3. Redesigned rsh enabled scripts to use ssh
      4. Designed method for using non-interactive ssh-enabled scripts
      5. Documented ssh environment and procedures for using it.
    3. Documented tape vaulting procedure for six cell Omniback backup environment.
    4. Documented complex Autosys environment.
    5. Scripts:
      1. Developed and implemented a set of perl scripts to monitor Omniback backups for progress and failures. Script uses a centralized database of backup schedules to determine when to initiate a backup monitor on each of six cell servers. Resulted in identifying failed backups up to 18 hours sooner than before script implementation.
      2. Rewrote/standardized operating system backup ksh scripts resulting in a more easily maintained OS backup/restore process.
      3. Developed and implemented a perl script that transfers up to 15 Oracle archive logs from production system to DR site simultaneously via secure copy (scp)
      4. Developed and implemented a web based account creation function using perl as the common gateway interface (cgi). Script uses ssh to create remote accounts, set initial password, and displays results via another web page. Reduced by at least 75% the time it took to create user accounts on multiple systems.
      5. Developed and implemented web based backup tracking system:
        1. Tracks system (fbackup) results for all HP systems.
        2. Tracks Omniback backups for all cell servers.
        3. Tracks tape pool related information:
          1. Total number of tapes in the pool
          2. Number of scratch tapes in the pool
          3. Number of tapes by status (good, fair, poor)
    6. Participated in Bank One Disaster Recovery test in 11/2002.
      1. Failed over & back HP superdome SAP Central Instance and V2250 application server.
      2. Failed over & back system password and trusted password databases.
      3. Failed over & back Omniback implementation
      4. Troubleshot issues with Oracle archive log corruption
      5. Aided DBAs by scripting required file changes (system names/Oracle instance identifiers, etc)

  9. Acxiom: Created and documented HP MC/Serviceguard cluster protecting Oracle SAP implementation. Provided additional backup and scripting support as well
    1. Created and documented an MC/Serviceguard cluster protecting mission critical Oracle SAP implementation.
    2. Created, tested, and implemented an Oracle hot backup script with similar functionality to EMC Timefinder software. Script splits HP LVM mirrors, mounts the split volumes, backs them up, then resilvers the mirrors. Backup times reduced from 12 hours of Oracle outage to no customer impact.
    3. Created, tested, and implemented import/export ksh scripts to maintain LVM consistency between disparate cell nodes (N4000/K460). Script identifies which HDS XP256 LUN is in use and identifies the specific disk device driver on each system. It then updates the HP LVM data structures using HP LVM commands.

  10. Rockwell Automation/Volt Services: Developed/implemented perl scripts on over 200 SUN/HP/AIX systems to automate password changes for privileged accounts. Developed/implemented perl scripts which allowed helpdesk personnel to securely change nonprivileged account passwords. Standard system admin support
    1. Developed/implemented perl scripts which enabled system users to change their passwords on over 200 UNIX systems. Reduced the time required for this necessary security protocol from several hours to half an hour.
    2. Developed/implemented perl scripts which enabled help desk personnel to assume responsibility for account resets on nonprivileged user accounts. Script verified account status and last login time in accordance with corporate security policy. Script maintained system integrity through use of sudo to limit privileged exposure and comparison of encrypted security strings.
    3. Shared in responsibility for management of trouble ticket queue which saw everything from account creations through performance evaluations to system crash analyses.

  11. Sevenspace/Nuclio: Developed/implemented perl script to scan 1800 routers/switches via snmp and send HP Openview IT Operations (ITO) messages if user definable parameters/thresholds are broken
    1. Troubleshot modifications to previously written SNMP/perl script.
    2. Developed/implemented perl-based script which scans up to 1800 routes/switches in five minutes and sends HP Openview ITO messages when configurable parameters break user definable thresholds. Script uses an external table to identify the parameter to check, the formula to use to identify the threshold and the action to take when the threshold is broken.

  12. Allstate: Provided hardware installation, configuration, and scripting support
    1. Installed/configured HP9000/800/L2000 as production system failover and test box.
    2. Developed and implemented a system hot backup script using HP LVM Mirroring and Omniback 4.X backup software. Reduced system outage required for backups from 12 hours to 15 minutes.

  13. CSC/Hyatt: Upgraded and configured sendmail on variety of IBM AIX 4.3 servers
    1. Upgrade/configured sendmail.
      1. Identified and documented email requirements including relaying, masquerading, direct mail hosts, exposed users, and address translations.
      2. Compiled, installed sendmail and configured using m4 macros.
      3. Result is a well documented, easily maintained and consistent email environment across all corporate systems.

  14. General Board of Pensions and Health Benefits for the United Methodist Church: Project management for a HP MC/Serviceguard cluster supporting Lawson running on Oracle
    1. Project management for installation, configuration, and documentation of a 3-node HP MC/Serviceguard cluster.
    2. Verified minimal hardware/software requirements, drafted project plan, coordinated schedule, and verified required preparation work was completed.
    3. Re-IP.ed, renamed three systems and used originals as multiplexed, relocatable IP addresses to circumvent hours of client reconfigurations.
    4. Led newly trained cluster admins through cluster and package initialization and configuration.
    5. Developed and disseminated documentation covering project plan, timeline, and cluster configuration.

  15. Govenors State University: Provided on-call UNIX support and project management as-needed.
    1. Troubleshot DNS issue preventing transmission/reception of email
    2. Upgraded Veritas Volume Manager
    3. Troubleshot complex NFS cross mounted environment
    4. Reconfigured sendmail and anonymous ftp services
    5. Upgraded E450 operating system from Solaris 2.6 to 2.8.

Company Nuclio
Title Senior Technical Consultant
Time Frame 10/00 - 08/01





  1. Designed and implemented EMC environment for Network Attached Storage (NAS) project. Environment consisted of dual EMC 8300s with EMC SRDF for DR and EMC Timefinder for backups.
  2. Developed, coordinated, and executed project plan to migrate two 120+ gigabyte Oracle databases from older disk systems to newer ones with a simultaneous cold HPUX OS install/upgrade.
  3. Ran performance analysis on several systems for one of our clients. Identified memory and CPU shortfalls on two systems and that other systems weren't running into bottlenecks.
  4. Planned, installed, and configured Veritas Netbackup on a ten node network for a client.
    1. Ensured backup classes and schedules met requirements.
    2. Wrote scripts to verify that backups ran the previous night and were successful.
    3. Verified backups periodically.
    4. Migrated master server from older Ultra 5 to a newer Enterprise 250.
    5. Migrated tapes from older L9 library to a four tape autoloader.
  5. Identified and worked issues involving the Oracle database that run Nuclio's ITO and Remedy systems. Wrote, documented, and implemented a script to scan the Oracle alert logs and report on problems as well as alerts DBAs and sysadmins when critical tablespaces are getting full.
  6. Wrote replacement script to scan client's 140 routers. Previous script used ICMP to check on routers and critical interfaces; caused performance problems on both the client network and Nuclio's ITO database. Replacement script uses SNMP to get interface statistics directly from the router; reduces network traffic by at least 66%; decreases runtime 80% on average.


Company Sysix Technologies
Title Senior Technical Consultant
Time Frame 08/99 - 10/00





  1. Certified HPUX technical consultant, 98% first time testing.
  2. Certified Solaris System Administrator.
  3. Billed out a minimum of 85% of the time.
  4. Developed, documented, and disseminated procedures for standard system peformance evaluations.
  5. Developed, documented, and disseminated procedures for standard UNIX system security evaluations.
  6. Developed, documented, and disseminated procedures for standard Veritas Netbackup installation/configurations
  7. Developed, documented, and disseminated procedures for standard EMC Timefinder implementations
  8. Client Accomplishments:
    1. Numerous performance evaluations. Examples:
      1. V2200: ID'ed I/O bottleneck caused by large sequential writes. Developed, coordinated, and implemented plan to circumvent. Archive log switch performance improved 329%.
      2. K370: ID'ed CPU bottleneck that was causing unacceptable performance degradations for a warehouse control system.
      3. K580: Verified that the system had no systemic bottlenecks and helped identify a buggy system monitoring package as the cause for periodic system "hangs".
    2. Planned breakup of NIS domain which was allowing all company personnel to log onto the primary and backup HR/payroll systems.
    3. Planned and implemented IP change on a geographically distant system.
    4. Repaired, documented, and/or updated system configuration:
      1. Removed badly misconfigured MC/ServiceGuard configuration, allowing Oracle data volume groups to be activated on boot.
      2. Repaired NFS mount configuration, allowing ERP applications to start automatically on system boot.
      3. ID'ed, ordered, and configured requisite hardware to increase system's fault tolerance.
      4. ID'ed and fixed numerous security weaknesses such as user modifiable scripts running via root's cron, system directory ownership/permissions, etc
    5. Scripts:
      1. Documented complex, poorly designed ksh scripts; cleaned up and debugged where necessary.
      2. Designed and developed web based dynamic system performance graphs.
        1. Graphed data included cpu, filesystem, memory, and Openmail capacity/performance information.
        2. Graphs were generated dynamically and automatically removed when appropriate.
        3. Graphs used for capacity planning and performance troubleshooting.
      3. System documentation script which formats and prints information on
        1. CPU
        2. Memory
        3. Disks
        4. Network configuration
        5. Printers
        6. Kernel
    6. Designed/developed policies/procedures for:
      1. Change control
      2. Security policies/procedures
      3. Backup procedures
      4. User access additions/deletions/modifications



Company: Acxiom - May & Speh
Title: Associate:  System Administrator
Time frame: 1/96 through 8/99





  1. Overall management responsibility of UNIX support team which provides support for Inter/Intranet services, system performance monitoring, tuning and maintenance for two separate data warehousing functions supporting company's direct mail clients. Database products are Fastcount andOracle. Network services include DNS, sendmail, firewall, and www
  2. Developed, coordinated, and implemented site security policy, detailing policies for change management, user installation/modification/deletion, system configuration and maintenance, and quarterly system audits. Oversaw the development of the procedures required to support and implement the security policy.
  3. Team lead on installation, configuration, and support for SUN Enterprise 10,000, currently running with 28 CPUs, 24 gigs of RAM and approximately 2.5 terabytes of disk supporting 6 domains.  Support consists of routine system maintenance, performance monitoring and tuning, system troubleshooting and, when necessary, crash analysis.
  4. Migrated 4.3 terabytes of data from older SUN/EMC disk to newer EMC 3930 disk array.
    1. ID'ed storage requirements and coordinated disk system purchase.
    2. Designed and tested various migration strategies depending on system, resources, clients, and allowable outage window.
    3. Coordinated, scheduled, and oversaw/implemented data migration.
  5. Researched upgrade from SGI Challenge XL to Origin 2000; drafted upgrade proposal; coordinated installation, certification, testing, and migration of proprietary database clients. System response time increased by minimum of 30% based on database benchmarks.
  6. Coordinated the reinvestigation of disk storage technology; formed and led the team that generated the project requirements; coordinated appropriate vendor briefings, and helped to develop the benchmarks for data warehousing evaluations which were used to determine the company's standard disk vendor.
  7. Developed web based change control logging system which allows system and database administrators to log proposed changes.
  8. Developed and implemented web based system documentation which employs network programming via perl to display real-time system configuration information and performance metrics.
  9. Researched, coordinated, and installed system to directly transfer data from company mainframes to various UNIX platforms. Reduced Oracle database load times from two weeks to three days. Reduced Fastcount load times from 7 days to 1.5 days.
  10. Planned, coordinated and upgraded SGI Challenge XL to IRIX 6.2 with simultaneous filesystem reorganization. Improved client proprietary database performance by 23%, measured by system accounting statistics.
  11. Migrated multimillion dollar client company between IBM RS/6000 servers. Coordinated; organized tests for system certification; reorganized existing filesystems and users for greater efficiency; transferred existing RAID controllers with no loss of data and minimum downtime.
  12. Identified and acted on a problem with SGI system performance; using IRIX 5.3 sar and OS performance monitors, identified scope of I/O bottlenecks and designed tests to determine best fit of filesystems to controllers and I/O. Striped filesystems and enabled command tag queuing; resulted in over 1000% drop in average time spent waiting I/O.


Management Experience

  1. Three years with May & Speh and Acxiom as UNIX team lead. Supervised a team of between two and eight administrators of various skill levels. Assigned projects and training based on experience and capability; tracked project status and provided feedback as needed
  2. Seven years experience managing 13 man Communications Computers branch for the USAF; developed work schedules, branch budgets, training plans and programs; trained subordinates from novice to advanced and senior system administrators
  3. Project management:
    1. Project manager for Litton DSD network connectivity during move. Provided tasking, schedules, and support for 7 man team tasked with planning and accomplishing the network move.
    2. Project manager for client migration. Planned, coordinated, and accomplished the requisite steps to move multimillion dollar client with no data loss and only six hours down time
    3. Team lead for SUN E10K installation, configuration, and client migration from older HP systems.
    4. Project manager for SUN->EMC Disk migration.
      1. ID storage requirements; coordinated purchase with vendor and corporate resoources.
      2. Coordinated installation/configuraiton of new EMC equipment.
      3. Designed specific migration strategies and procedures for each client and system.
      4. Coordinated removal of old disk equipment.


Training
Org Courses
SUN:
  1. Solaris Admin I/II
  2. SUN Fault Analysis
  3. SUN Server Administration
  4. Veritas Volume Management
  5. SUN Enterprise 10,000 administration
HP:
  1. HP Openview
  2. HP Advanced administration
  3. HP performance monitoring and tuning.
SGI:
  1. SGI Advanced administration.
USAF:
  1. Completed 13 semester hours in Leadership and Managerial Communications through professional military education in residence courses
EMC:
  1. SRDF
  2. Timefinder
  3. Powerpath
  4. Symmgr
  5. Control Center
  6. Fibre Zone
Misc:
  1. Accredited course from Loyola on management and leadership communications
  2. UNIX Security Administration
  3. Network Security Administration
  4. sqlplus
  5. Advanced sqlplus
  6. Oracle Database Administration.