OpenAI Withdraws 3 Math Papers

(github.com)

73 points | by theemathas 4 hours ago

9 comments

  • margorczynski 8 minutes ago
    If you do a dump like this all of it should be formalized, there's simply too much material to review by hand and additionally it is AI-written which makes it hard to read compared to human work.
  • qoez 48 minutes ago
    Without a thriving mathematical community to point out these things it would have stayed broken. With automated math that community as tao pointed out is at risk.
    • viraptor 42 minutes ago
      Have you got a link to someone pointing it out? It looks like they're still going through formal proofs so likely found the problems that way.
      • sanxiyn 34 minutes ago
        • viraptor 30 minutes ago
          So someone ran a different LLM to find an issue they'd find anyway during formalisation? That's not the same as relying on thriving community.
          • oliculipolicula 9 minutes ago
            The bigger question is why there was internal pressure to rush such a historic launch without having someone in the company, anyone, check the proofs first.

            This concerns the Hodge conjecture (millennium prize related) paper. Seems to me like PhD nerds weren't confident bosses pushed ahead anyway.

        • afavour 29 minutes ago
          That’s not proof though is it? If the original LLM output is fallible surely the LLM review of that output is also very much fallible?
  • TrackerFF 8 minutes ago
    And added 6 new ones. Might want to add that to the headline.
  • hmate9 40 minutes ago
    3 mistakes (so far) out of ~400 is still a pretty good hit rate
    • afavour 31 minutes ago
      I’m conflicted. I guess we’ll see what the final total is once an enormous level of unpaid human effort is expended verifying the AI outputs. A little sad if that’s the future of math.

      It kind of reminds me of when tech giants open source a project as a means of putting a positive spin on abandonware. “Here’s the source! Any problems are yours to fix now. You’re welcome”

      • ndriscoll 1 minute ago
        Where are you getting unpaid from? Almost all people who are qualified to analyze the results are paid researchers. And if it is unpaid, then it sounds like they're looking it over for their own reasons, and that's fine?
      • rubyfan 9 minutes ago
        We are all reverse centaurs now.
    • watinthedeutsch 18 minutes ago
      I would say not. For a mathematicians, having to retract more than 2 papers in a lifetime is already a big issue in their career.
      • jacobstokes 14 minutes ago
        But is it equivalent to withdrawing post-publication or is it more akin to not passing peer review with major revisions requested?
      • zzzeek 14 minutes ago
        Most mathematicians don't produce 400 papers in a 48 hour window either so I'm not sure comparisons are helpful
    • jansport123 4 minutes ago
      well, it might be that these proofs are correct or it might be that people aren't bothering to spend a lot of time checking whether they are correct. OpenAI already has a pretty bad reputation in the mathematics community for how they are approaching this process, they seem to be more interested in creating a story for their IPO than advancing math.
    • matsemann 38 minutes ago
      But can the others even be "disproven", given that they apparently are so messy and awful that no humans can follow them? Shouldn't the onus instead be on OpenAI to prove that they're right, instead of hundreds of mathematicians wading through slop?
      • true_religion 20 minutes ago
        The onus is on formal verification when it comes to computer generated results. So far only 22% of the papers have it.
    • catlifeonmars 14 minutes ago
      How many mathematicians need to retract ~1% of their papers?
    • malux85 36 minutes ago
      Exactly, I was glad to see these withdrawals, its a natural part of a healthy ecosystem of scientific review, hypothesis, claim, test, refute, extend, withdraw, its the heart of science.

      IMO If you take out all the stupid human aspects mostly related to fear, egos, etc, we should brace the imperfect and helpful tools, whatever they are, improve them so they are as easy as possible to review, and keep that core scientific discovery loop going

  • soltanov 1 hour ago
    Proof by authority works until human mathematicians actually run the code. Back to prompt engineering.
  • renyicircle 42 minutes ago
  • falconBrisk47 1 hour ago
    [flagged]
  • breezybottom 1 hour ago
    So much for the "it's lean verified" defense.
    • n2d4 55 minutes ago
      The withdrawn papers were not lean verified nor claimed to be.
      • hckrme 48 minutes ago
        I wonder why the heck were they provided / uploaded then. Perhaps just a fast and loose play-out on their part. What I don't understand is how come engineers / scientists working on these are okay with this kind of attitude.
        • amelius 46 minutes ago
          I suspect they are not okay with this way of working.
          • mcmcmc 22 minutes ago
            I’m sure the outsized pay packages help quite a bit
          • ModernMech 14 minutes ago
            They’re okay enough to do the work and collect a paycheck and stock options. I don’t think their arms are being twisted that hard.
        • boxed 9 minutes ago
          How come mathematicians are ok with human mathematicians are ok with that kind of sketchy submissions? That's pretty bad too.
        • xjdirkdn 7 minutes ago
          Lies travel around the world before the truth has time to put it’s sneaker’s on…

          The headlines keep the hype train arunnin

      • breezybottom 37 minutes ago
        But that's the argument that was used when people here were skeptical about the results.
        • yreg 33 minutes ago
          not these results
    • nkmnz 29 minutes ago
      Can you point to where that defense has been made?
  • chairhairair 14 minutes ago
    This company is just irresponsible. We (at least, Americans that vote and can therefore decide indirectly what’s legal) should not allow them to continue.
    • monideas 3 minutes ago
      This type of sentiment is almost always motivated by fear over the potential negative personal economic impact from AI (e.g. losing employment).

      Publishing a math paper and then unpublishing it is not "irresponsible". It's just a math paper.

    • trio8453 13 minutes ago
      How is withdrawing a paper irresponsible?
      • jansport123 0 minutes ago
        it's like if you vibe coded something and the onus is now on the reviewer and the reviewer tells you your work contains bugs and is messy - that is not acceptable from the reviewers pov - why should the reviewer spend all his human effort, a scarce resource, reviewing your code while you've spent barely a fraction of his effort generating this. OpenAI is a trillion dollar company, surely they can verify stuff before pushing it out? The problem is not that they are solving the problems, they don't care at all about the actual process of doing mathematics. Using compute to mine problems and throwing results in github and letting human reviewers spend effort to correct these is not going to win them any favor.
      • moregrist 10 minutes ago
        They only withdrew it after someone pointed out that it was a flawed paper.

        In one case by asking Astra to review it.

        Not exactly encouraging that they did their homework before publishing results.

      • spider-mario 5 minutes ago
        Withdrawing it, in itself, might not be, but it kind of skips over the part where they have a paper that warrants it in the first place.