Tag Archives: technology

algorithms, data science, not-hadoop, twitter, Uncategorized

AI Needs a Better Acronym*

A team at METR Research recently did a study of the effectiveness of AI in software development and got some very surprising results.  At the risk of oversimplifying, they report that the developers they studied were significantly slower when using AI, and moreover, tended not to recognize that fact.

METR’s summary of the paper runs to multiple pages–the full paper is longer–so a couple of lines here can’t really do it justice, but the developers on average had expected AI to speed them up by about 24%, and estimated after doing the tasks that using AI had sped them up by about 20%. In fact, on average, it had taken them about 19% longer to solve problems using AI than it did using just their unadorned skills.

My statistics days are behind me, but the work looked to be of high quality, and the people at METR seem to know how do do a study.  Surprising though the result may be, the numbers are the numbers.

Yikes! How Can That Be?

Superficially, METR’s results seem wildly at odds with my experience using AI for coding. I used it for a month, and felt that it vastly increased my output, not by a double digit percentage, but by a large factor. Four times? Five times? More? It’s impossible to know for sure, but a lot.

Yet, after reading the study more carefully, not only did it begin to seem like a dog-bites-man story, I came to feel that their results strongly validate my own experience and understanding of AI. If you take the message to be, “AI doesn’t make developers faster“, you are probably missing the point.

(Don’t take my one paragraph summary too seriously. METR’s summary can can be seen here. The full study can also be reached from within that page.)

Continue reading →
Standard
Uncategorized

AI Programming Part 2: The Particulars

Part one of of this article is about AI programming at a high level and where it might take us. This second part is about the practical lessons I learned in using it. Read this piece, take all of my advice as gospel, and by the end of the article, you’ll be an expert too.1

If you didn’t catch part one, I used AI to rewrite in Go a program that I wrote by hand in Java more than ten years ago. The project comprises about 15,000 lines of Go, plus another 5,000 lines of Python, JavaScript, and shell scripts. It took two weeks and change to write using AI, a small fraction of the time it took to hand-write the original.

The program does a simple thing very fast: it reads the X (formerly Twitter) firehose and extracts the newly emerging subjects. It does this on up to about 8,000 Tweets/second when reading from RabbitMQ, and at about 50,000 Tweets/second reading directly from disk files.

This program makes a good test project for using AI because of its diversity. It’s a high-performance program with lots of concurrency, lots of disk reads and writes, text parsing, tokenizing, some intense algorithmic processing, some esoteric graphing libraries, and a simple Web front end.

Continue reading →
Standard
Uncategorized

AI Programming, Part 1

This is the first of two essays about programming with AI. Part two can be found here.

I recently used Cursor to rewrite in Golang a substantial program that I first wrote in Java more than a decade ago. It took two weeks to write it using AI, and since then I’ve messed with it casually, adding some features and making minor changes.

The effort went very well. Cursor informs me that the new version comprises 15,460 lines of Go, 992 lines of Python, 1584 lines of shell scripts, and 3842 lines of HTML and JavaScript. That’s 20,557 lines written in a long couple of weeks, plus documentation and a manual.

That’s a lot of code in very short time, and the throughput of the resulting program is an order of magnitude greater than the original.

Given that level of productivity, it’s easy to see why undergraduates are voting with their feet, running from Computer Science programs like they are plague wards. Unsurprisingly, the old Unix geezers snort at the idea that AI could ever program anything meaningful, goddammit. But so what–those guys snort at everything.

I think both opinions are wrong. The truth is more complex and more hopeful.

Continue reading →
Standard