Hold on, we aren't even good at estimating but now we know how much time we saved by vibe coding? I can't wait to read the source of this info when you share it.
No they aren’t. I’ve been doing this for a few decades working everywhere from small startups to the most successful tech companies, and none of them are good at estimating.
If you read the blog post, they were able to manually migrate 3% of the tests in 1 week. Extrapolating that gives you an estimate of less than half of the 1.5 year estimate.
I’d also say there’s a pretty good chance that the 3% of tests the automated process couldn’t handle were more complicated than average and the devs would have gotten faster at doing these migrations after the first week.
It’s very unclear what was actually being compared. A team did a POC in 2023 and then some work in 2024 and then spent 6 weeks tweaking the pipeline and a final week manually migrating the tests that the automated process couldn’t handle.
But they don’t specify how many people worked on this or how many people were going to work on the original project. It could have been 1 guy on the team.
As for looking at what the actual work was, it was migrating from one test framework to another one. I’d be surprised if a team couldn’t have written a compiler to do something similar in a similar amount of time.
Whatever the term “vibe coding” is taken to mean, it assuredly doesn’t apply to a large scale migration undertaken by a professional software organization.
it means having it generate code and then not looking at the code and going purely off of vibes. If you have it generate some code and then dig deep into the code and edit it to the point that you're able to explain every variable name and the reasoning for each if clause like your boss is gonna call you out for cheating and using an LLM and is gonna fire you if you can't explain "your" code, that's gonna slow you down and you're no longer purely going off of ~/vibes/~
"Airbnb recently completed our first large-scale, LLM-driven code migration, updating nearly 3.5K React component test files from Enzyme to use React Testing Library (RTL) instead. We’d originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks."
Color me unimpressed - it converted some test files. It didn't design any architecture, create any databases, handle any security concerns or any of the other things programmers have to do/worry about on a daily basis. It basically did source to source translation, which has been around for 30+ years.
If you told me five years ago that such a conversion had been done in six weeks, I would not have believed it. Even though some level of source-to-source existed. And I would definitely expect that such a conversion would have resulted in hideous, non-idiomatic code in the target language.
I’ve done something similar to this type of work with an llm. It produces code that is often too idiomatic, in that it introduces conventional approaches that are overkill for the task at hand. But this is almost an ideal scenario, because if the previous tests run clean, the newly converted tests can only fail by being wrong.
They can also silently reduce the tested scenarios, but that’s what code reviews are for.
I use LLM for coding every day. LLMs can totally make something super dumb: oh this test is broken, gatcha, lets replace it with something which is not broken (but very different).
> They can also silently reduce the tested scenarios, but that’s what code reviews are for.
code review of complicated scenarios is about as resource demanding as actually writing code, so shrinking time spent from 1.5y to 6 weeks totally can produce lower quality.
great comeback - "Dude - Trust me" :-P
got anything more compelling then your opinion ?
Have you ever done source to source translation work (I have ...)
Or is this just so exciting to you cuz you've never seen it before ?
exactly, you didn't provide any evidence or even decent arguements to support your position. I provided 3 distinct areas (of many) that it didn't touch that programmers have to deal with everyday. I also pointed out that the capabilities to do the sorts of transformations it did have existed for decades.
But given your comment history, backing up your arguments doesn't seem to be your strong suit...
"Vibe" coding is here to stay and it's only devs who don't know how to adapt that are wishfully hoping for otherwise.