I have 10+ years of experience working in software/hardware development environments in a broad range of roles including: operating a cloud service at massive scale, architecting new product features, building and managing physical datacenters, designing hardware appliances, supporting customer escalations, profiling software performance, and automating everything in my way. A get it done kind of guy who likes to work with innovative people and solve interesting problems.
Systems Development Engineer, 2016 - Present. Hitachi Vantara - Lumada (Remote)
Systems Development Engineer, 2014 - 2016. Amazon Web Services - Lambda (Seattle, WA)
- Live migrated production environment from a single AWS account into multiple regional accounts with zero downtime. This helped reduce blast radius for production issues from a global to regional level. Technologies used: #python #dynamodb #dns #aws #zookeeper
- Launched Lambda in 5 new regions and in parallel reduced region build time from one month down to a single week. This involved documenting the process from scratch, creation of new tools to automate this process, building a suite of new region validation tests, and updating build tools and pipelines to support these new regions. Technologies used: #aws #golang #python
- Automated capacity metrics, tracking, and purchasing of thousands of instances a week - a time consuming job that was previously done sporadically in a spreadsheet. These metrics helped defined an algorithm for ongoing reserved instance purchases that was both economical and at the same time maintained enough capacity for bursts in service usage. This helped move the majority of Lambda capacity from on demand instances to reserved instances and saved millions of dollars in the process. This additional reserved capacity significantly reduced capacity related outages during sharp spikes in customer usage. Technologies used: #ruby #rails
- Resolved legacy configuration issue that resulted in IP address exhaustion in high growth regions. Quickly proposed, tested, and implemented solution that allowed instances to slowly migrate into new subnets and old subnets to be cleaned up. This migration process was done with zero downtime or customer impact.
- Reduced attack surface for production AWS accounts. Highlights include automated key rotation, migrating manual account setup to a reproducible config management system, removal of all production access from dev desktops, and mandatory Kerberos authenticated bastion hosts for access to production environments.
- The only designated change management bar raiser on the team. This responsibility included reviewing and approving all deployments going to production. Helped meet team goal of less than 3% deployments issues due to operator error.
- Created team TV dashboard for viewing key service health metrics. Improved awareness of system health among team and helped discover anomalies in our data. Technologies used: #angularjs
Senior Technical Architect, 2011 - 2014. Hitachi Data Systems - UCP (Bellevue, WA).
- Built development infrastructure from scratch for greenfield project. This included automating installation and backup of all infrastructure services as well as creating a streamlined continuous integration environment with end to end testing and code coverage. Technologies used: #linux #ansible #packer #vagrant #docker #gitlab #jenkins #sonarqube #gradle
- Wrote automation framework for deployment of management software stack; a side project that turned into a key component of the product. Decreased onsite software deployment time by 95%. Technologies used: #powershell #powercli #xml
- Created a suite of tools to detect hardware misconfigurations/failures and ready systems for shipping. These tools immediately reduced the number of support issues seen at time of onsite deployment. Tools directly interfaced with Ethernet and FC switches, blade servers, and SAN devices.
- Reverse engineered a third party software product and produced a POC API to prevent a core release feature from being dropped. Feature ended up releasing on time and native API was later released. Technologies used: #flask #fiddler #burpsuite #wireshark #xmlrpc
- Developed web frontend to allow shared scheduling of hardware environments. Created an automatic reclaim feature that found more than 30% of gear unused. Helped automation team integrate with webapp and schedule automatic runs on idle hardware. Technologies used: #php #bootstrap #jquery #mysql #racktables
- Designed and implemented a product feature to support unattended baremetal installation of Windows, Linux, and ESXi on a shared network. Technologies used: #ipxe #wds #autodeploy #syslog #csharp
Solutions Architect, 2008 - 2011. Hitachi Data Systems - HCP (Waltham, MA)
- Resolved high priority customer data outage by writing code to run in parallel across the customer’s cluster to find missing data. Initial estimate to recover data relying on product feature was reduced from two weeks down to a single day. Technologies used: #bash #postgres.
- Developed tools for end-to-end automated deployment of product through PXE infrastructure. Tools were added to automation framework and allowed for continuous testing against latest software. Technologies used: #python #django #postgres #pxe
- Discovered and reproduced hard to find Java memory leak in customer environment that was causing slowdowns and periodic outages for 2+ months. Hotfix was created and later integrated into product. Technologies used: #exchange #visualvm #standardlinuxutils
Certifications / Education
- AWS Certified Solutions Architect – Associate Level.
- B.S. Network Security and System Administration, from Rochester Institute of Technology with 3.91 GPA.
- Coffee, Side Projects, Learning, Privacy, Security, Cryptocurrency, Immutable Infrastructure, Gym, Reddit, Hacker News, Hiking, Soccer, Podcasts.