Manager, Grid Services Operations & Architect. Starting
from zero, built a team of nine systems administrators and architects
distributed between Santa Clara and Bangalore, India. Grid Ops
maintains large grids using several open source utility computing
packages, primarily Hadoop. The team is responsible for supporting
and expanding grid computing environments for both Yahoo! production
processes and Research & Development. In 2008, we handled average
growth of over 1,500 computers per month, while simultaneously
supporting internal customers, new features and new software projects.
My role was to manage service capacity, negotiate scope with new
customers, justify additional headcount, recruit and retain new
talent, and integrate them into the existing team. I devised and
built concensus for the following policies: software roll-out,
escalation procedures, on-call rotation, ticket management SLA, and
post-mortems. I organized and championed feedback to the development
and product management teams and tracked those changes through the
product lifecycle.
In 2009, my role changed to Manager, Grid Operations
Architecture. To support a large number of systems administrators,
I directed the efforts of two operations architects to build and
support a standards-based infrastructure capable of supporting tens
of thousands of computers in multiple locations. I negotiated and
managed projects related to designing and deploying this architecture.
We integrated Kickstart, Kerberos, and LDAP with Yahoo!'s internal
components as well as the open source technologies (Hadoop) offered
by Grid Services.
July 2006 - October 2006
Member, Grid Services Team. Designed and developed a system
to reliably transfer the updated contents of a data warehouse to the
Yahoo! grid computing cluster and distributed file system. Implemented
in Python, using XML-RPC and Berkeley DB, the system transfers
hundreds of gigabytes per day, and re-populates any gaps in the data
when and if it becomes available.
YAHOO!
SEARCH
TECHNOLOGY
November 2004 - June 2006
Manager, Search Cluster Automation and Monitoring (CAM).
Built a team of three developers to address automation concerns
within the search core product. Designed and co-authored System
Knowledge Tracker, a configuration and state tracking system capable
of recording the entire life-history of tens of thousands of nodes.
Also delivered Database Traffic Control, a project of almost 30,000
lines of Java code. DTC modified and automated an existing manual
process, trimming days off of a week-long cycle and scheduling
operations more optimally than the previous system.
WALMART.COM
October 2001 - October 2004
Sr. Systems Engineer. Design and develop high-end systems
solutions, using elements such as multi-terabyte storage architectures,
HA and clustered deployments. Provide second-level escalation support
for production systems. Mentor junior staff and provide technical
leadership on large-scale projects.
The Systems Engineering team is responsible for the design and
deployment of the Walmart.COM production sites. Our team works with
the rest of the company (engineering and business sides) to ensure
that our site has sufficient capacity and maintains an adequate
level of fault tolerance. Metrics available on request.
We specify all facets of new project implementations that affect
the production sites. This includes overall architecture, capacity
planning, product selection, third party integration, storage and
computing layers. I was the single systems engineer responsible for
search engine replacement and babycenter.com integration. Initial
implementer of ProactiveNet 4 and 5 monitoring system, BladeLogic
automation, and IronPort A60 MTAs. Engaged Keynote and Mercury
Interactive services for production load testing.
VIRALON
CORPORATION
June 2000 - April 2001
Manager, Technical Operations. Built a team of five who
were responsible for production systems, database administration,
IT and technical support. Ensured that schedules and goals were
met on a consistent basis. The team successfully deployed both
staging and production systems comprised of network infrastructure,
firewalls, load balancers, as well as HTTP, BEA Tuxedo, and Oracle
servers. Negotiated and managed relationships with all hardware and
service suppliers. Worked effectively across teams to manage project
planning. Assisted the Business Development team on technical sales
calls.
Using Apache, HTML::Mason, Perl, BEA Tuxedo, Linux and Oracle
on Sun Solaris, the Viralon team created a best-of-breed ASP service.
The service provides dynamically created web content for the clients
and their customers to deploy advanced marketing campaigns. Viralon's
product included highly customized customer user interfaces as well
as a functional client interface for initiating, controlling and
monitoring campaigns.
Also as Manager of Technical Operations,
I was responsible for all IT facilities and maintenance. This
included Cisco-powered WAN and LAN, NT file servers, IIS and Microsoft
Exchange. Our team was also responsible for all Windows desktop
support and printers.
INKTOMI
CORPORATION
August 1999 - December 1999
Architect, Custom Services. Technical leadership for the
Custom Services team of administrators. Designer of both the Dedicated
Search Cluster product and other advanced search services. Responsible
for longer term technical vision, development and operation of Custom
Services infrastructure. Custom systems include all Mini-Clusters,
Dedicated Search Clusters, and Data Blends. Deployed services, on
schedule, for clients such as Yahoo!, The Wall Street Journal, AOL,
and BellSouth.
As Architect, I was the primary author of an automation framework
comprising more than 10,000 lines of Perl code. This product was
actively used by the entire Search operations division to manage and
implement practically all Inktomi search services. It was also the
primary delivery vehicle for all Dedicated Search services.
April 1998 - August 1999
Portal Services Staff. Production Systems Operator for
Search services. Original member of Custom Services team, responsible
for Mini-Cluster product development and operation. Third member
of a team that implemented and managed the large-scale (1 billion
transactions per month, over 1000 nodes) general web search engines
and crawlers.
The Portal Services team refined techniques for effectively
administering thousands of geographically diverse Sun Solaris
machines. This included all aspects of administration: hardware
maintenance, OS upgrades, security, networking and application
performance. We also evolved the monitoring standard to allow for a
high degree of accountability for even the most demanding
clients. This work included systems programming in C and Perl.
UC
BERKELEY's
DEPARTMENT of
MOLECULAR and
CELLULAR
BIOLOGY
Spring 1994 - March 1998
Unix System Administrator. Sole support staff for
Stanley Hall. Maintained approximately 15 SGI workstations
and various others throughout the entire building. I ultimately
reported to Professor Koshland (retired editor of Science) but
worked for all of the professors in the building. Here is a list
of my duties.
GLOBAL
NETWORK
NAVIGATOR
Summer 1996
Technical Staff. Originally a service of O'Reilly and
Associates, GNN was a subsidiary of AOL, producing one of the Web's
first successful on-line magazines. Worked as aide for an extremely
overloaded Scott Silvey. Behind-the-scenes work on the GNN WWW
site. Wrote CGI programs and perl scripts for site-analysis. In
addition, I set up many, many WWW servers and references for them,
along with installation of miscellaneous software. Participated in
the planning and implementation of a major revision of the production
system.
BANK of
AMERICA,
CORPORATE
TREASURY
Summer of 1992 and 1993
Systems Administrator. General troubleshooter and problem
solver for two Vice Presidents of the bank. This involved work with
users and modification/creation of several utilities and applications.
MAXIS
SOFTWARE
Summer 1991
Intern. Aided in completion of SimCity graphics sets to
meet software publisher's deadline. Reported directly to the head
artist, Richard Payne.
