This is the first of two essays about programming with AI. Part two can be found here.

I recently used Cursor to rewrite in Golang a substantial program that I first wrote in Java more than a decade ago. It took two weeks to write it using AI, and since then I’ve messed with it casually, adding some features and making minor changes.

The effort went very well. Cursor informs me that the new version comprises 15,460 lines of Go, 992 lines of Python, 1584 lines of shell scripts, and 3842 lines of HTML and JavaScript. That’s 20,557 lines written in a long couple of weeks, plus documentation and a manual.

That’s a lot of code in very short time, and the throughput of the resulting program is an order of magnitude greater than the original.

Given that level of productivity, it’s easy to see why undergraduates are voting with their feet, running from Computer Science programs like they are plague wards. Unsurprisingly, the old Unix geezers snort at the idea that AI could ever program anything meaningful, goddammit. But so what–those guys snort at everything.

I think both opinions are wrong. The truth is more complex and more hopeful.

The Project

The purpose of the program is to extract the latest subjects that people are Tweeting about from the torrent of text that is the X/Twitter firehose. 

The program has very high performance, with a lot of concurrency, i.e., multi-threading, interactions with messaging systems and files, some intense algorithmic processing, and a simple front end to visualize results. There are also some odds and ends of utilities and programs to do statistical analysis. As there has already been a V.1, rewriting was all about the programming rather than working through conceptual problems. 

The fact that it looks for new subjects is critical. It’s important not simply because “new” implies low latency, but because extracting all of the subjects that are active in X at any one time would generate too much data to be useful. If you limit it to new subjects, not only is the data flow manageable, you get the most relevant subjects first, and over time you approach having gotten them all. 

The trick to doing it fast is identifying the Tweets on new subjects without being forced to do any non-linear time operations on the full stream of Tweets. The algorithm looks at each Tweet exactly twice in order to identify the tiny subset that are new subjects. Any non-linear operations involving comparisons apply only to that small subset, and take very little time.

The brass ring is being able to find the new subjects with a latency that is competitive with how long it takes for a person to read and respond to a Tweet. That level of immediacy would allow automated processes to interact with the public as events happen. The implied throughput means it is fast enough to analyze bulk historical data.

The Environment

I downloaded “Cursor” which is a clone of the venerable Visual Studio Code with an added AI coding assistant. You can get a free couple of weeks of trial period with most of the capabilities of the full professional product, and my project was almost finished by the time the trial was over. In fact, I didn’t notice any practical effect when Cursor reverted to the free version.

The original program was in Java, and it processed a couple of “decahoses” on a 2011 MacBook that was old even at the time. A decahose is 1/10 of the 5000 Tweet/second firehose, or about 500 Tweets/second. (5000 per second is typical–at times the flow surges up to 20,000 per second.)

The new version is written in Go, which is faster, simpler, and more pleasant to work with than Java. I did not use AI to transliterate the original, but simply reviewed my notes on how the original worked and jumped into writing it from scratch with AI.

Another reason for choosing Go was my level of experience. I used Go only briefly several years ago, which makes me a junior level Go programmer at best, albeit one with a relatively senior level of general programming and architectural knowledge.

Not To Bury The Lede

The original program took months–I don’t remember how many, but certainly more than two. With AI doing the coding, it only took a couple of weeks to reproduce the original program. The functionality of the two versions is very similar, but the Go version is more than an order of magnitude faster, and highly reliable.

The two efforts weren’t exactly comparable because the first version served as a prototype for the second, but the AI coding felt incredibly fast. Each step of the rote part, things like laying out the project, setting up Rabbit MQ, getting Tweets parsed and moving through a pipeline, reading and writing files, and other mundane tasks, took barely longer than the time it took to type out a description of what to build. It’s very fast.

The lessons learned are complicated. The implications are not as simple as “programming is over, man.” 

Section two of this blog piece will be a list of many things I learned about dealing with an AI assistant at a nuts-and-bolts level. This section will present what I think are the deeper insights first.

Broad Strokes

To be clear, the AI absolutely smoked hand-coding for the rote parts, which surely added up to at least 85% of the total code. There was no contest.

The AI knew the keywords relating to concurrency, and had a reasonable idea of how to apply them, but it struggled with the subtleties and made numerous major mistakes. It could not seem to understand even the simpler principles used in concurrent programming, such as using queues to provide elasticity among the processing pipeline components.

It didn’t grasp many of the basic design principles that high-performance depends on, but was nevertheless very strong at coding modules if they were described reasonably unambiguously. If the intended function is straightforward, it doesn’t take a lot of detail; the AI is good at filling in the blanks in your description because it’s good at language.

That’s the main thing; even with powerful AI assistance, everything about the structure of the program at the whiteboard level is up to you, and that’s the level that makes or breaks a substantial program. AI can’t lay out the broad strokes for you unless it’s essentially a stock single threaded design. You have to know what you are doing.

On the other hand, it was surprisingly capable regarding some not at all obvious aspects of the program, so long as they had limited scope across the code. For instance, in the Java version I hand-implemented K-means for clustering related Tweets. The clustering is a critical piece, but it is basically one operation confined to a single place in the pipeline. Oddly, there wasn’t a good library with K-means available in Go, so I asked Cursor to implement it, which it did, in seconds.

The AI’s K-means worked, but then it suggested switching to a canned graph-clustering algorithm. I said, great, let’s try it, but please make the choice of which to use configurable. Cursor set it all up without much input from me. It also schooled me on the theoretical reasons why the graph algorithm it suggested was more suitable than K-means (and I believe its analysis was sound–K-means was not a great choice.)

The critical point is, while AI was incredibly effective on a large subset of the code, it wasn’t effective on all of it.  Which leads us to an often neglected design principle.

Amdahl’s Law

Amdahl’s Law, which is so often a source of disappointment to programmers, applies here. This engineering principle gives a formula for how much the overall performance of a system can be speeded up by optimizations.   Wikipedia summarizes it as follows:

“The overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”.

It is a simple principle. Suppose that a program spends 70% of its time executing a few intense lines of code, and you optimize the hell out of that few lines, making them run 100x faster. That would be an incredible success, right? Yet, applying the formula above, we see that the entire program speedup is a factor of 0.307, i.e., overall, the program will be only 3.2573 times as fast, not 100x. 

It feels like it should be more, but our speedup applied to only 70% of the program.  The other 30% is as slow as ever.

In our case, the “system” we are talking about optimizing is the creation of the program rather than the execution of the program, but the same principle applies. 

We can’t narrowly consider just the actual coding to be “the system” that AI is optimizing. The system is everything that happens between the boss handing you the assignment and you hitting the return key to release version 1.0 into the wild. 

So how much of this total system is the coding? I don’t know, but my experienced guess is that head-down coding rarely occupies as much as half of a programmer’s day and programmers aren’t the only technical staff involved. When you add in the system administrators, Kubernetes experts, QA staff, scrum doctors, managers, people writing documentation, etc.–and I’m making the numbers up–coding adds up to probably what, a quarter of the work hours that go into the first release?

Let’s say AI programming is 4x as fast as coding the old fashioned way. Plugging the numbers into Amdahl’s formula, AI would only speed up development by a factor 1.2X in total person hours. It makes sense; we only speeded up half of what the programmer does, and the programmer is only one part of the technical team.

That’s disappointing, so let’s be more optimistic and say that coding goes 20x faster with AI, and half of all person hours on the project are coding. Even with that unrealistically rosy scenario, development is only 1.904x faster. A 20x speedup in coding doesn’t even double the overall development speed!

The principle is obvious once it’s pointed out: if the coding hours are 1/x of the total work, speeding up coding will not increase overall development speed more than 1-1/x. 

Yet The World Isn’t That Simple

While Amdahl’s Law, as usual, is disappointing, it’s not the last word on the subject because writing computer programs isn’t just mechanics. It is a creative process that is subject to ordinary human emotions and social influences.

My brief experience with AI so far is that the speed increment it provides is exhilarating. It radically lowers the barrier to creativity. In the pre-AI world, you’d get that bolt of inspiration from the blue, and then the other 95% was just typing. If AI can reduce that 95% to 9.5%, the universe of things that are worth the effort expands greatly and there is an order of magnitude less intellectual mud to slog through between conceptual leaps.

AI programming won’t be simply change how fast we do things; it will expand and diversify the set of things we judge to be worth doing and change the level of excitement around doing them. Programming is a cultural activity–excitement matters.

We’ve seen this before. In the late 1980’s C was the mode. Punch cards were still around. Machines connected on local area networks were the exception and “client-server” was the buzzword of the day. The Internet was known only to insiders.  

Then suddenly, windowing systems, the WWW, broadband, high-level languages, modern IDE’s, and high-quality interpreted languages arrived and there was an explosion in the range of things that seemed to be worth computerizing.

Programmers suddenly got a lot more done in a work day, but as is typical of labor saving innovations, it didn’t actually save any labor. What happened is, it became possible to get much more out of the same amount of labor. The barriers to engineering creativity went from something like a wall topped in barbed wire to more like a picket fence you could swing your sneakers over. By the mid-90’s, the programmer’s world had changed radically and programmers were cultural heroes.

AI is reducing the barriers to something like a curb.

Fred Brooks

The software engineering classic, “The Mythical Man Month”, written by Fred Brooks in 1973, was already a relic from an earlier generation when I first saw a computer. The massive IBM mainframes that Brooks’ team wrote the OS/360 operating system for had at most a few megabytes of memory, sometimes less than one, and their clock rates topped out at about 3Mhz. Green-screen CRT display consoles were just replacing teletypes.

Yet Brooks’ essential insight is as true today as it was fifty years ago. He said that there are two kinds of complexity in computer programs. One kind is the accidental complexity of tools, languages, machines, and processes. This is complexity we can reduce with clever programming and tools.

But there is also essential complexity–the inherent complexity of the problem itself. Essential complexity is far more recalcitrant. Tools aren’t much help because essential complexity is a direct reflection of the complexity of the world itself, and how we want to interact with it.

AI is the latest in a series of epochal improvements in dealing with accidental complexity, but like the brilliant tools that came before it, it mostly leaves that essential complexity untouched.

I can certainly believe that it’s a bad time to be starting out as a developer. AI can already kick a junior or middle level programmer’s ass. However, it might actually bring on a golden age for people with more advanced computer science and architectural level skills, the skills that deal with essential complexity.

It seems unlikely that LLM based AI will replace these top people anytime soon. To do that is going to require artificial general intelligence, which, like fusion power, seems to be perennially just around the corner.

We shall see.

AI Programming Part Two: The Particulars