SRE/Devops Manager • October, 2019 — October, 2020
Led team responsible for SRE and devops for a large non-profit CRM SaaS platform and managed a small Data Services/Import team.
- Supervised devops tooling for development process/improvements, including:
- Rollout of kubernetes cluster of dev/QA environments to replace old single-instance docker QA env
- Worked with lead dev to move from CircleCI to GH Actions as prerequisite to speed up testsuite and prep for improving test coverage
- Oversaw SRE maintenance and improvements to Ruby on Rails SaaS platform in GCP:
- Introduced terraform for any new infrastructure deployment (previous infrastructure was hand-rolled)
- Wrote some GCP SDK tooling for inventorying/cost auditing
- Introduced ansible for both config management and one-off bulk config changes (previous infrastructure was all hand-configured with ruby scripts)
- Implemented opsgenie and proper on-call rotation
- Implemented statuspage for outages/downtime and coordination with support team.
- Upgraded Postgres to 9.11
- Worked with CTO and VP of Engineering to guide development process improvements:
- Initiated and facilitated transition of workforce to remote working
- Planned for large rails migration to recent version
- Recommended extensive test coverage improvement as prerequisite to this upgrade
- Recruiting/talent acquisition
- Managed small Data Services team:
- Managed small team of data services for importing new customer data.
- Improved tooling and process improvements both to spreadsheet/csv intake and small ruby app to do data sanitization and import
- Managerial duties for both teams, including 1:1 meetings, reviews, and feedback.
Director of IT/Security • September, 2015 — September, 2019
Led team responsible for both devops/systems automation and security improvements for a large suite of e-commerce products.
- Revamped and maintained traditional baremetal (leased) production web app platform
- Juniper networking (redundant core routers, switches, firewalls)
- BigIP F5 for L4-L7 security/filtering, load-balancing
- MDS SAN switch cluster
- EMC VNX storage arrays
- VMWare Vsphere for virtualization
- Clusters of nodes for:
- Frontend web nodes (NGINX)
- App nodes (PHP/Symfony)
- MySQL and Mongo DB clusters
- Queuing/caching (redis, rabbitmq, et al)
- Misc/utility - cron, jenkins, CI tooling
- Data warehouse -- massive reporting/analytics storehouse for combined MySQL/mongo data
- Built and managed non-production development/operational architecture
- Built/maintained Nashville data center for development environments (largely smaller replica of above architecture with xencenter for virtualization)
- Extensive tunnelling (IPSEC) between prod, dev, office networks
- Maintained and supported legacy (Juniper SSL-based) VPN access
- Refactored config management from antiquated/disorganized puppet3 to puppet4+hiera following modern best practices as prerequisite for migrating to Alibabacloud
- Replaced antiquated proprietary java/SSL vpn w/ redundant openvpn and greatly pared down and restricted unnecessary IPSEC tunnelling
- Migrated production architecture (largely intact) to alibabacloud using new puppet4 tooling and terraform
- Built out more securely segmented VPCs for development and intranet applications
- Migrated baremetal development architecture to alibabacloud, decommissioning nashville datacenter for cost-savings
- Rebuilt development VM based on vagrant and puppet4 tooling to replace antiquated fixed image they were using
CentreSource, Inc.
COO • May, 2004 — September, 2015
Led an IT consulting/web development agency (30-50 employees, ~$3-5million revenue)
- Built and ran successful managed anti-spam/virus email filtering service
- MX record redirect service for e-mail filtering
- Postfix MTA cluster
- Blacklist, greylist, content-based (spamassassin, amavis/clamav, et al) filtering and quarantine
- MySQL cluster for backend quarantine storage
- PHP frontend for user quarantine false positive/negative management&refinement
- Built team/tech for web app development
- LAMP architecture
- Symfony/Ruby on Rails framework
- SVN and later git/github workflow
- Pivoted to management as COO
- Basic accounting -- P&L, balance sheet reporting, budgeting, accounts payable
- CRM and sales pipeline tracking/analysis
- Revenue forecasting (based on sales pipeline and project load)
- Managed 3 person team for company operations
- Developed project estimation, management and scheduling workflow with ERP software (Workamajig)
- IT/devops
- Legal -- Master Services Agreement (MSA) negotiation, individual project agreements
- Client dispute negotiation
Systems Engineer • May, 2001 — May, 2004
Provided systems integration and network engineering for a state-wide network supporting over 250,000 computers and one million users.
- Planned and deployed of caching server/content filters to 5 egress locations for caching and filtering of web traffic in the TN K-12 network
- Engineered the replacement of a legacy DEC Alpha OpenVMS cluster, home to 60,000+ e-mail accounts with modern openldap/postfix/courier setup
- Implemented primitive config management using CVS, and rancid for nameservers, routers, caching servers, and others
- Designed and implemented a state-wide DNS hierarchy, home to nearly 500 domains, in a fully redundant, any-cast setup including two (primary and secondary) authoritative, non-recursive servers.
- Designed / Implemented OpenBSD/pf firewall system to replace costly and broken Watchguard Firebox