In mid-2024, Tidelift fielded its third survey of open source maintainers. More than 400 maintainers responded and shared details about their work, including how they fund it, who pays for it, and what kinds of security, maintenance, and documentation practices they have in place today or would consider in the future. They also shared their thoughts about some “in the headlines” issues like the recent xz utils hack and the impact of AI-based coding tools. In this post, we share the tenth of twelve key findings. If you don’t want to wait for the rest of the results, you can download the full survey report right now.

Nothing says it is 2024 in the technology industry like a headline about AI, so we would have been remiss if we hadn’t asked maintainers a few questions about their perceptions of AI. One of the biggest AI-related headlines of 2024 has been the rapid growth and acceptance of AI-based coding tools. So we wanted to start by asking maintainers to share their assessment of the impact that AI-based coding tools will have on their maintenance work.

What do maintainers think will be the impact of AI-based coding tools on their work?

First, we got a baseline impression of whether the impact of AI-based coding tools on maintainer work was positive or negative. The overall maintainer perception of AI-based coding tools leaned negative, with almost half (45%) of maintainers predicting that these tools will have a somewhat negative (22%) or extremely negative (23%) impact on their work.

In fact, maintainers split pretty evenly across every choice but “extremely positive” (9%), with 22% of maintainers saying AI-based coding tools will have a somewhat positive impact and 24% saying they will have neither a positive nor a negative impact. 

Maintainers explain the impact of AI-based coding tools in their own words

We gave maintainers a chance to elaborate on their response to the previous question. After categorizing the comments, we found that the highest percentage of responses voiced concerns about the quality of code AI-based coding tools currently produce.

As one maintainer said:

“My experience is also that AI-based tools often produce incorrect code in more complex situations, and it can be hard to identify issues with their code unless you already know how to do it.”

Other maintainers agreed with that assessment, and added more context:

“AI-based coding tools exponentially increase the chance that someone without context of the codebase or the project will build a PR that looks correct but contains breakages that can bring the entire language ecosystem down until a patch fix is released. The codebase is simple but the impact is absolutely beyond enormous.”

“AI makes it easy to quickly generate lots of code which nobody understands, including the AI creator. As a technology, it is great at simple stand-alone tasks, or boilerplate which aligns well with existing code upon which it has been trained. As such, it is an occasionally useful tool for working programmers. But used for anything deeper, it frequently generates code with errors both subtle and glaring, and has to be carefully and fastidiously corralled into the desired behavior by an expert level programmer.”

The next most common concern voiced by maintainers was the increased maintenance burden they believe AI-based coding tools will create for them.

A few example comments:

“The increase of spam PRs, comments, and false positives from AI tools and users has been enormous and very frustrating.”

“LLMs and machine learning tools have demonstrated potential aid only for the mechanical, non-creative aspects of software development. These are at the expense of increased burden to vet their output for mistakes, and the tools are incapable of explaining their work, so this is worse than with a human collaborator.”

“I don't want to become the gate for reviewing tons of automatically generated pull requests. Sounds like it would further wear me and my co-maintainers down.”

Other maintainers’ comments expressed critiques of the current generation of AI-based coding tools.

“I don't find AI-based coding tools useful yet, there's a lot of nonsense in the suggestions and they don't feel well-integrated into coding tools yet. For example, they may override or be confused for type-based suggestions. I imagine those issues will be fixed, and these tools will save some time with boilerplate tasks, but also introduce overhead elsewhere. I don't expect a huge impact on my life either way.”

“I don't mind AI making suggestions, but all suggestions whether by an AI or a human require checking and thought. First and foremost programming towards a specific goal requires a clear understanding of the problem and clear thinking about the ways to best accomplish this. Sometimes deeper theorems about the problem and special algorithms are needed. For these kinds of things AI has not been very helpful. I am not even sure AI purports to do this kind of thing either.”

Still, despite the reservations, a good number of maintainers expressed optimism about the possibilities of AI-based coding tools.

“I use Copilot and I'm sure it will take adjusting to get used to these new tools but I think the payoff is definitely going to be worth it. They just need to be used within reason.”

“I am hopeful that it will help me with the boring tasks I keep putting off, such as documentation and code tests.”

“For me GitHub Copilot has been a spark that has given me interest in maintaining my projects again because it cuts a lot of the mundane parts out.”

How willing are maintainers to review and accept contributions created using AI

We also wanted to learn more about how willing maintainers would be to accept code contributions that they knew were produced using AI-based coding tools. We asked:

If you knew code contributors were using AI-based coding tools, how would this impact your willingness to review and accept their contributions?

On this question, maintainers’ responses were significantly more negative than their general perceptions of AI-based coding tools. Almost two-thirds of maintainers (64%) said they’d be much less willing (37%) or somewhat less willing (27%) to review and accept contributions they knew were produced using AI-based coding tools. 

Only 9% said they would be much more willing (3%) or somewhat more willing (6%) to review and accept contributions they knew were produced using AI-based coding tools. And just over one-fourth (27%) aren't sure or don’t yet know enough to make a decision.

How useful is the information from automated pull requests for vulnerability remediation?

Finally, because we’ve heard reports that maintainers were receiving many more AI-generated pull requests for vulnerability remediation than they have in the past, we wanted to get a sense for how useful these pull requests are. 

On this question, the bulk of maintainers responded toward the middle of the spectrum and not at either extreme. The highest percentage of maintainers (36%) find the information they receive from automated pull requests “not very useful” for vulnerability remediation. The next most popular response was “very useful” (25%), with only 10% at the negative extreme answering “not at all useful” and 8% at the positive extreme answering “extremely useful.”

As with some of our other AI-related questions, a good percentage of respondents had not yet made up their minds: 21% answered that they are not sure or don’t know whether the information they receive from automated pull requests are useful for vulnerability remediation.

It’s a pretty clear bet that AI-based coding tools are here to stay. And in our survey maintainers raised some valid concerns regarding how these tools will impact their maintenance work, although many also see a lot of positive potential in AI-based coding tools as well. 

To best serve the needs of open source maintainers, the ideal path the creators of AI-based coding tools will need to navigate as they continue to innovate is to ensure they remove more maintainer work than they create and, at least with maintainers, on that front there are many good suggestions in this survey for making this a reality.