Software Was a Means to an End
I was a firefighter and paramedic working in Aircraft Rescue and Firefighting when I started building software. Much of the operational work around us was still managed through paper, spreadsheets, disconnected records, and knowledge held by the people doing the job.
I started building software because we needed better systems. I built applications for firefighting inventory and equipment management, inspection tracking, FAA Part 139 airfield reporting, and internal NOTAM tracking with historical records. The first versions were built in Access and Visual Basic because those were sufficient for the problem in front of me. As the systems grew beyond what that environment could reasonably support, I rebuilt and expanded parts of them using Python, Flask, and MySQL.
This was not software I was building from requirements handed to me by an IT organization. I was one of the people doing the operational work. I knew what needed to be inspected, what the FAA required us to document, what information mattered during airport operations, and where the existing processes were inadequate. I learned the technology necessary to build the systems because the systems needed to exist.
The software worked, but there is an important part of that story I would understand differently later in my career. When I left the department, they stopped using it.
At the time, I had solved the operational problem. I had not solved for the organization eventually operating the system without me. Years later, after watching engineering organizations grow, contract, lose institutional knowledge, and inherit systems whose original developers were gone, I came to understand that as part of the engineering problem too.
Modernization Is More Than Replacing Software
When I moved into healthcare technology at EBMS, the scale of the systems changed considerably. EBMS had been administering health benefits and adjudicating claims for years, and its technology reflected an earlier generation of enterprise computing. Claims processing, member information, eligibility, historical records, business rules, customer service processes, and years of operational knowledge were tied to systems that had evolved with the company. The industry was beginning to move toward modern cloud-based claims platforms, and EBMS was undertaking that transition.
Calling it a software conversion made the problem sound much simpler than it was. Much of the legacy information lived in flat files, while the new platform expected a normalized data model. The structures did not correspond neatly, and the old data represented far more than records that needed to be copied. It represented years of decisions about how the business operated.
The internal technology organization had spent most of its history maintaining the environment it already had and did not have experience with this kind of transformation. Even the vendors selling the emerging cloud solutions did not have a convincing answer for how to get us from the system we had to the system they were selling.
I began building that bridge in Python. I wrote ETL processes to extract the legacy information, interpret and transform it, and load it into the normalized structures required by the new platform. Problems that initially appeared to make the conversion impractical became solvable once the work stopped being treated simply as moving data. We had to understand what the old structures represented, preserve the business meaning inside them, and then express that meaning correctly in a very different system.
Replacing the claims engine also exposed capabilities that had grown around the legacy environment and were missing from the new platform. Case management was one of them. Customer service was another. When a member called about a claim, the person answering the phone needed immediate context about that member and the claims behind the conversation. We built integrations between the telephone environment and the new systems so an incoming call could identify the customer and surface the appropriate information for the representative handling it.
Healthcare technology was only beginning to make that transition, and many of the capabilities required to operate the business simply did not exist in the products being sold to replace the old systems.
Some of my most consequential work at EBMS did not begin as a technology project. The company introduced Kaizen process improvement, and I led its first initiative. I started by examining the production of Explanation of Benefits statements.
EBMS produced EOBs entirely in-house. The real cost of sending one was therefore much greater than postage. There were industrial printers, ink, paper, envelopes, sorting equipment, envelope-stuffing machines, maintenance, mail-room labor, postage, and all of the physical handling required to turn claim information into a document delivered to a member. It was an expensive process, but it had existed for so long that most of the expense was simply accepted as part of operating the business.
I traced the entire process and its costs and built a system that allowed members to elect electronic EOB delivery. This was still early in the industry's transition toward electronic member communications, and there was not an established product we could simply install to solve the problem. I built the capability internally.
In the first year, only about 20 percent of the client base adopted electronic delivery. That was enough to increase EBMS's net profitability by 4.5 percent. The company's annual goal for profitability improvement across all departments that year was 4 percent. A single process change had exceeded it.
That result changed my role at EBMS. I was promoted to Director, and process improvement became a continuing part of my responsibility. We conducted another Kaizen approximately every six months, moving into different areas of the business to understand how work was actually being performed, identify unnecessary cost or friction, and develop solutions. Some of those solutions involved technology and some did not.
The process continued after I left the company.
When the System Became the Interaction Between Systems
CrowdCompass introduced me to a different kind of software. The company was building mobile event applications during the period when conferences and large events were moving rapidly from printed schedules and materials onto smartphones.
Events are inherently dynamic. Schedules change, sessions move, speakers change, attendees interact, notifications need to reach people while the event is happening, and devices cannot be assumed to remain continuously connected.
I worked on the platform using event-driven architecture, where behavior increasingly existed in the interactions between components rather than inside a single application and database. Events represented things that had happened, different parts of the platform reacted independently, and state moved through the system asynchronously.
The problems were no longer confined to what happened inside one application. Correct behavior depended increasingly on what happened between systems, what each component knew, when it knew it, and how the platform behaved when those assumptions changed.
Complexity Has an Economic Cost
I joined Renew Financial in 2015 as its eighth developer. Renew was still very much a startup, and it was building technology around a financial product the company had helped create.
Property Assessed Clean Energy financing crossed boundaries that conventional lending software had not been designed around. The financing involved a consumer, but eligibility was also connected to the property, its value, taxes and assessments, geographic jurisdiction, government programs, existing obligations, project characteristics, contractor information, financial history, and regulatory requirements that varied across programs and locations.
I worked across the call-center loan origination system and the systems behind it, including automated decisioning engines, rules engines, workflow engines, stateless backend services, geolocation services, county tax services, loan services, property tax systems, government regulatory servicing systems, and the infrastructure that connected them.
As Renew grew, that environment eventually became an ecosystem of 27 applications. The Portland engineering organization grew from the small group I joined to more than one hundred developers.
The external environment was nearly as complicated as the internal one. More than thirty third-party services participated in core business operations. CoreLogic and DataTree provided property and real-estate information. Census data and geographic shapefiles helped establish location and jurisdiction. DocuSign participated in document execution. Salesforce supported customer and operational workflows. Box was part of document management. Credit, property, tax, financial, and other providers supplied information required at different points in origination and servicing.
Each source represented a different part of the decision the business was trying to make, and the platform had to turn all of that information into a coherent loan process.
Those integrations also had an economic consequence. Many of the providers charged us each time we requested information. A straightforward implementation could gather everything potentially needed for an application and make the eligibility decision afterward, but doing so meant purchasing information that had no value if something we already knew made the application ineligible. At enough volume, the architecture of the decisioning process became part of the cost of originating a loan.
PACE decisioning could not be reduced to pulling a conventional credit score. We had to evaluate payment histories, tradelines, property characteristics, tax information, program requirements, geographic eligibility, and other factors to determine whether an application could proceed. Some information was inexpensive to obtain, some was expensive, and some decisions could eliminate an application without purchasing anything further.
We designed the decisioning process to take advantage of that. If the evidence already available established that the application would fail, processing stopped before the next paid integration was called. More expensive information was acquired only as the application survived the decisions ahead of it.
That work reduced the average external-data cost involved in acquiring and decisioning a loan from roughly $25 to about $11. The sequence in which the software asked questions had changed the economics of the product.
Reliability Changed the Definition of Working
Renew also forced me much deeper into the operational side of software. Some of the infrastructure I encountered was quite literally running on servers in a closet. As the company and platform grew, I became heavily involved in Site Reliability Engineering and in moving the platform toward highly available infrastructure in AWS.
That work included the infrastructure itself, but it also meant deployment, automation, monitoring, failure recovery, production operations, and the way the engineering organization responded when something went wrong.
Incident response brought me unexpectedly back to firefighting. Emergency services have spent generations learning how people should operate when something important is already failing. Command is established. Responsibilities are understood. Communication has structure. Someone maintains the larger operational picture while other people work individual problems. Information moves deliberately because confusion during an emergency creates additional problems.
Software organizations frequently approached incidents very differently. An outage occurred, a large group of highly capable engineers entered a room or chat channel, and everyone began trying to help simultaneously. The technical talent could be exceptional while the response itself remained poorly coordinated.
I began adapting principles from emergency incident management to software operations, creating a structured approach to command, communication, responsibility, escalation, monitoring, and learning after an incident.
That work eventually became something I was hired to bring into other engineering organizations. I conducted paid engagements with companies including Puppet Labs, JanRain, Urban Airship, New Relic, and CrowdCompass by Cvent. These were not conference talks or brief presentations. A typical engagement involved an initial two-hour education session followed by hands-on work over subsequent weeks, roughly twenty hours in total, establishing company-specific incident documentation and processes, improving alerting and monitoring practices, and putting disciplined postmortem processes in place.
My approach to reliability was never based on preventing every failure. Failures are not conditions outside the architecture. They are normal operating conditions the architecture has to account for. How a system behaves when something fails is part of whether the system works.
The Organization Can Change Faster Than the System
My role at Renew changed along with the company. I moved through Senior Engineer and Site Reliability Engineering into Senior Staff and eventually Principal Engineer. I had joined as developer eight and watched the Portland engineering organization grow beyond one hundred developers.
Then I watched it move in the other direction.
Major layoffs reduced the organization by roughly half, and another large reduction followed. Attrition continued afterward. The number of people capable of carrying the system declined much faster than the complexity of the system itself.
After the first two major layoffs, Renew had a legitimate problem. We were carrying a large custom software ecosystem while the engineering organization capable of maintaining it was shrinking rapidly. The executive decision was to reduce that dependency by sunsetting much of the homegrown platform and replacing it with commercial software.
The objective was reasonable. The first replatform was not.
PACE was not conventional loan origination with a few additional rules. Much of the complexity in our platform existed because the product crossed consumer finance, property taxation, geography, government programs, contractor workflows, regulatory requirements, and servicing rules that conventional lending platforms were never designed to represent. The replacement product could handle pieces of that world, but the fit was never close enough to replace the system we actually had.
I disagreed with the architectural direction, but once the decision was made I worked to make it succeed. As the gaps became apparent and delivery commitments remained, I increasingly found myself building adjunct software, shims, and integration points around the commercial platform to provide capabilities it did not have. We were trying to eliminate custom software while progressively writing custom software around the product intended to replace it.
The first replatform ultimately failed, but it had already changed the architecture. Most importantly, decisioning had been moved out of our platform. The shims and integrations I built allowed the systems to continue operating, but they also meant that simply abandoning the replacement and returning to the previous architecture was no longer a practical option.
The second replatform took a different approach. Rather than buying a product and attempting to make PACE fit inside it, Renew partnered with a loan-servicing software company that wanted to add PACE to its existing portfolio. They would extend their platform to support the domain, and I would work with their engineering organization to integrate that capability into the ecosystem we already had.
The premise was better. The execution exposed another version of the same problem.
The complexity of the domain was undersold during the business and architectural discussions, while the vendor oversold its ability to accommodate it. When implementation began, their team started building what they believed PACE required before working deeply enough with me to understand the system and the domain they were replacing. Some of those early assumptions were substantially wrong.
That mattered because the engagement had been structured as a fixed-cost project. Once the scope of those mistakes became clear, correcting them produced significant cost overruns for the vendor. Contractually, they were still responsible for delivering the system. Operationally, their enthusiasm for doing so deteriorated as the economics of the project moved against them.
At the same time, Renew could not simply walk away. The first replatform had already moved critical decisioning outside the homegrown platform, and the work underway for the second had taken the transition further. We had reached a point where the organizations were technically coupled even as the business relationship between them was deteriorating. Neither side particularly wanted to continue, and neither side had a clean way to stop.
The dispute eventually became a legal and contractual matter, and the vendor was required to complete the work. That resolved the contractual question. It did not make the engineering problem disappear.
By then, the Portland engineering organization I had joined as its eighth developer and watched grow beyond one hundred had contracted until I was the last internal developer. Offshore contractors remained, and I was responsible for the technical continuity necessary to get the platform through the transition. As gaps appeared between what had been promised and what could actually be delivered, I picked up the work necessary to keep the replatform moving and the business operating.
Eventually the replacement was declared complete.
I shut down the homegrown platform I had spent years building, operating, scaling, migrating, and keeping alive. With the transition complete, my role was eliminated as well.
About a year later, the replacement platform collapsed. Renew contacted me about returning to take technical ownership again and rebuilding a custom solution around the business. I declined.
There is no clean success story in either replatform, and that is part of why they were valuable experiences.
The first attempt assumed that moving to commercial software could remove complexity that actually belonged to the business. The second recognized that PACE required specialization but still began with an incomplete understanding of how much specialization was required. In one case the product did not fit the domain. In the other, the organization agreeing to extend its product began implementing its understanding of the domain before sufficiently understanding the system it had committed to replace.
Both projects also demonstrated something less visible in architecture discussions: technical decisions change the choices available later. Moving decisioning out of the original platform solved an immediate transition problem, but it also changed our ability to retreat when the replacement strategy deteriorated. By the time the relationship around the second replatform became adversarial, the question was no longer simply which architecture we preferred. Earlier decisions had already removed some of the alternatives.
That is one reason I think about architecture as commitment rather than arrangement. A diagram shows where the components are today. It rarely shows which doors were closed in order to put them there.
The Data Is Not the Meaning
When I joined Jack Henry, I entered another part of financial technology. Before moving into money movement and ACH, I worked on the systems supporting Personal Financial Management.
Financial institutions generate enormous amounts of transaction data through different upstream systems and aggregation providers, but the raw transaction that arrives is rarely the transaction a person expects to see in a useful financial application. Merchant names vary. Transaction descriptions are inconsistent. Categories may be missing or incorrect. Different upstream sources can represent similar financial activity in completely different ways.
I worked on a high-performance financial-data aggregation platform responsible for turning those different representations into a consistent domain model and enriching the resulting transactions. My work centered on the architecture behind those problems: data modeling, indexing and access patterns, asynchronous processing, transaction and merchant normalization, categorization, enrichment, and historical reprocessing as our understanding of the underlying data evolved.
That last problem is more important than it initially appears. Interpretation changes. Merchant mappings improve. Categorization rules evolve. New information becomes available. Users change preferences. A system that becomes better at understanding a transaction tomorrow should be capable of reconsidering the transactions it interpreted yesterday without compromising its ability to process what is happening today.
That creates a different class of problem from simply importing financial records. Current ingestion and historical reinterpretation compete for many of the same resources. The data model has to support both. Indexes and access patterns have to reflect how the information will actually be consumed. Enrichment needs explicit ownership, particularly when several stages may contribute different pieces of meaning to the same transaction.
The objective behind all of that complexity is considerably simpler than the implementation. Financial information presented to a customer has to be understandable and trustworthy. The customer should not have to understand the peculiarities of the systems that produced the raw data in order to understand where their money went.
When Ambiguity Means Money
My work later moved from interpreting financial activity into money movement and ACH. That changed the consequences of the decisions considerably. In a data product, an incorrect interpretation can produce a poor customer experience. In money movement, ambiguity about whether something happened, happened twice, or can safely happen again becomes a correctness problem.
The domain itself is not something I need to be vague about. ACH processing means dealing with files, batches, entries and addenda as they move through RDFI and ODFI responsibilities, returns, posting, warehousing, file generation, and interaction with the payment network. What I cannot describe publicly is how Jack Henry's systems implement those responsibilities or how the internal services are arranged around them.
The engineering problems at those boundaries are familiar across distributed financial systems. A timeout may mean an operation failed, or it may mean the operation succeeded and the response never returned. A message may arrive more than once. Processing may stop after one durable change but before the next. Recovery therefore cannot simply mean trying everything again, particularly when the thing being processed represents money.
My work has increasingly centered on the architectural decisions behind those conditions: establishing identity that remains meaningful across processing stages, making lifecycle and state transitions explicit, determining where transactional guarantees are actually required, designing operations that can be retried safely, and preserving enough information to recover correctly when processing stops somewhere between beginning and completion.
That work crosses service boundaries. My role is not limited to implementing an isolated component. I challenge where responsibilities are drawn, trace assumptions across services, identify cases where locally reasonable behavior produces unsafe system behavior, and build working implementations when code is the fastest way to test an architectural argument. Much of the difficult work is not deciding how one service should behave. It is establishing what must remain true when several independently operating parts of the system participate in the same financial lifecycle.
That is about as far as I am comfortable describing the current architecture publicly. The implementation belongs to Jack Henry. The engineering problems, and the judgment required to reason through them, are the work I can discuss.
The Boundary of the Problem Kept Moving
Looking backward across more than twenty-five years, the technologies are almost the least interesting way to describe the progression. I have worked through several generations of languages, frameworks, databases, infrastructure, deployment models, and architectural styles. Physical servers became virtual infrastructure and then cloud platforms. Applications became services. Synchronous workflows increasingly became asynchronous ones. Systems that once lived inside one company now depend on networks of external platforms and providers.
The progression was never from programmer to someone who stopped programming. I still write code, and I still think implementation is one of the most useful tests of an architectural idea. A working prototype can answer questions that another week of discussion cannot. Reading an existing codebase often reveals a system very different from the one described in its diagrams. Sometimes the fastest way to understand whether a boundary is correct is to build far enough through it to discover what the abstraction is hiding.
What changed over the course of my career was the boundary of the problem I was trying to solve.
Early on, that boundary might have been an inspection process or an inventory system. Later it became a healthcare claims platform, an event-driven product, a lending ecosystem spanning dozens of applications and integrations, the production infrastructure underneath those applications, a financial-data platform, and eventually systems responsible for moving money.
As that boundary grew, purely technical answers became less useful. The difficult engineering problems increasingly existed between the code and the business, between one system and another, between architecture and operations, between technical possibility and economic reality, and between the organization that creates a system and the organization that eventually has to live with it.
That is the part of engineering I have spent my career learning.