Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test

(blog.jetbrains.com)

23 points | by Sandman 2 days ago

9 comments

  • jpease 1 minute ago
    Why not Mandarin Chinese? Logographic is pretty compact.

    Also, this really should be more precise that it’s talking about neo-caveman. Legit caveman no speak English.

  • DougN7 27 minutes ago
    I don’t know about the rest of you, but the bulk of my token usage is not what I type in - it’s that data the AI is processing for me. I’m shocked they even got an 8% reduction.
  • abofh 39 minutes ago
    If you just want a smaller vocabulary, use French? If your goal is to communicate to an LLM, maybe saving tokens isn't the all in win, unless you like reading assert gronkHitThing(true)
    • kardos 24 minutes ago
      French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
      • ViscountPenguin 23 minutes ago
        What would matter more is the token length, depends how well your tokenizer was trained on french I guess.
  • pineappletooth_ 52 minutes ago
    Well 8%-10% saving without measured quality degradation is not nothing, specially now that newer models seem to use more tokens than previous ones.

    Also it was tested on reasoning low, i'd have liked to have them tested on higher reasoning levels.

    • mjevans 37 minutes ago
      Ask, what is the function of grammar in language? I would argue two major functions. Forcing an encoder to marshal ideas into a coherent structure. Preserving the integrity of that structure when communicated and decoded.

      Grammar sounds off? Was the message correctly understood; is there enough left to apply error correction and regenerate the intended message?

  • igor_nast 2 days ago
    As the test shows the real gain is somewhat marginal to most people, but still some ppl claim a wide skills portfolio is required to move faster or save tokens. Mixed feelings, I personally use superpowers as my basic setup - and only them.

    What are your thoughts on it?

    • glitchc 38 minutes ago
      I use my local GPU. Yes, I realize it's not exactly a superpower, but it's technology that's pretty darn close to one.
  • mmastrac 36 minutes ago
    I suspect this means that there's just a ~10% inefficiency in token to information mapping.
  • ButlerianJihad 16 minutes ago
    https://groups.google.com/g/alt.nerd.obsessive/c/EGuKIN_4dME...

      alt.nerd.obsessive FAQ v1.4
    
      In the episode where they were filming the Radioactive Man movie [1995], the comic book store guy tells Bart that he can find out the star of the RM movie. He promptly posts a message to alt.nerd.obsessive which states "Need know star RM pic". This information is relayed through the nerd world until it reaches a nerd hiding under the table at a meeting of movie moguls casting the film. He relays back the answer immediately (Rainier Wolfcastle).
    
    https://en.wikipedia.org/wiki/Radioactive_Man_(The_Simpsons_...
  • sh34r 28 minutes ago
    gpt 4 moto razr wen?
  • altmanaltman 44 minutes ago
    Yes all the caveman spoke English like its freaking Flinstone. Like wtf is the caveman dilect, just english words with some randomly skipped? Why not try with actual other human languages and see if there is a token benefit in savings
    • gruntled-worker 6 minutes ago
      Me mechanic not speak English. But he know what me mean when me say "car no go", and we best friends. So me think: why waste time, say lot word when few word do trick?
    • strictnein 34 minutes ago
      Sounds like some good research. You should pursue it.
      • inasense 25 minutes ago
        lol - so easy a caveman can do it.