All posts tagged: LLM

Meet GPT-Red: an LLM super-hacker OpenAI built to make its models safer

Meet GPT-Red: an LLM super-hacker OpenAI built to make its models safer

Published by skeptic

As LLMs become more complex and get used in a wider variety of tasks—especially in the form of agents, which can interact with computer files, websites, and third-party code as well as other agents—it’s hard for teams of people by themselves to keep up with all the types of attacks that might take place. “The risk surface grows and the blast radius also grows,” says Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red. OpenAI built GPT-Red to future-proof its safety testing process. “As more capable models become available, we will have already designed the system that can discover new modes of attack,” says Dylan Hunn, a research scientist at the company and fellow co-creator of GPT-Red. The researchers say it has already come up with new types of attack that had not been seen before. OpenAI focused most of its efforts on a type of attack known as a prompt injection, where a hacker slips an LLM instructions to make it do things its developers or users do not want it to, such …

DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%

DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%

Published by skeptic

Even as the geopolitical conversation around AI continues to grow more fraught following the U.S. government’s actions to limit the new models from Anthropic and OpenAI, Chinese open source darling DeepSeek is back with yet another open release that could once again change AI development around the globe. Over the weekend, the firm released DSpark, a new, MIT-Licensed system designed to make large language models answer faster without changing what the underlying model is trying to say. The easiest way to think about it is this: most AI chatbots write like someone crossing a river one stepping stone at a time. They choose one small chunk of text, then the next, then the next. DSpark gives the system a scout that runs a few steps ahead, guesses the likely path, and lets the larger model quickly check which steps are safe. When the guesses are good, the model moves faster. When the guesses are weak, DSpark tries not to waste time checking them. DeepSeek published the work with a technical paper, model checkpoints and DeepSpec, …

Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit

Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit

Published by skeptic

Context windows are becoming a computational bottleneck. The longer an agent runs, the more tokens accumulate from retrieved documents, reasoning traces and conversation history, and the more memory and compute that growing context demands. Most existing solutions either degrade model accuracy, require the full context to load before compression begins, or produce memory savings that don’t translate into real speedups in standard serving infrastructure. A research team from NYU, Columbia, Princeton, University of Maryland, Harvard and Lawrence Livermore National Laboratory published a paper this week that proposes a novel fix. The researchers introduce the concept of  Latent Context Language Models, or LCLMs, a family of encoder-decoder compression models that compress input context before it reaches the decoder. The models are open-sourced on HuggingFace. Unlike KV cache compression methods — the dominant approach in the field, which still materialize the full KV cache before evicting entries — LCLMs compress the input token sequence before decoder prefill, so higher compression ratios directly reduce decoder-side compute and memory. The paper reports LCLMs at 16x compression produced output 8.8 …

I built a private LLM on my home PC using a USB drive — it only knows what I put on it

I built a private LLM on my home PC using a USB drive — it only knows what I put on it

Published by skeptic

When you feed sensitive code or research into a cloud-based AI, you lose control over where that information travels. It might seem like a small trade-off for the convenience of a smart assistant, but you are effectively handing your data over to servers you don’t own. It’s a risk that most people accept without a second thought. However, you don’t have to choose between advanced language models and the safety of your own private files; just switch to a local LLM. Related I’ll never pay for AI again AI doesn’t have to cost you a dime—local models are fast, private, and finally worth switching to. Privacy risks of cloud processing Your data is never as safe as you think it is when you use the cloud Jorge Aguilar / MakeUseOf When you send personal questions and code snippets to an external server, you are sending your private text, intellectual property, and sensitive information over the internet to computers owned by someone else. It’s easy to forget that because it feels like your own home computer. …

MIT’s MeMo lets teams swap in a better LLM without retraining — and performance jumps 26%

MIT’s MeMo lets teams swap in a better LLM without retraining — and performance jumps 26%

Published by skeptic

Enabling LLMs to acquire new knowledge after training remains a major hurdle for enterprise AI — current solutions are either too expensive, too slow, or constrained by context window limits. MeMo, a framework from researchers at multiple universities, encodes new knowledge into a dedicated smaller memory model that operates separately from the main LLM. The modular architecture works with both open- and closed-source models and sidesteps the complexity of RAG pipelines and full model retraining. Experiments show that MeMo handles complex queries reliably even when retrieval pipelines are noisy. It avoids the catastrophic forgetting associated with direct fine-tuning and provides a cost-effective pathway for continuous knowledge updates. The challenge of updating LLM memory Large language models are frozen after training and their internal knowledge remains static until they undergo subsequent, computationally massive updates. Comparison of different LLM memory frameworks (source: arXiv) Currently, developers rely on three main approaches to integrate external knowledge into an LLM, each with distinct drawbacks: Non-parametric methods, such as retrieval-augmented generation (RAG) and in-context learning, retrieve relevant documents from an external …

MIT’s MeMo lets teams swap in a better LLM without retraining — and performance jumps 26%

MeMo’s memory model lets teams upgrade their LLM without retraining it — and performance jumps 26%

Published by skeptic

Enabling LLMs to acquire new knowledge after training remains a major hurdle for enterprise AI — current solutions are either too expensive, too slow, or constrained by context window limits. MeMo, a framework from researchers at multiple universities, encodes new knowledge into a dedicated smaller memory model that operates separately from the main LLM. The modular architecture works with both open- and closed-source models and sidesteps the complexity of RAG pipelines and full model retraining. Experiments show that MeMo handles complex queries reliably even when retrieval pipelines are noisy. It avoids the catastrophic forgetting associated with direct fine-tuning and provides a cost-effective pathway for continuous knowledge updates. The challenge of updating LLM memory Large language models are frozen after training and their internal knowledge remains static until they undergo subsequent, computationally massive updates. Comparison of different LLM memory frameworks (source: arXiv) Currently, developers rely on three main approaches to integrate external knowledge into an LLM, each with distinct drawbacks: Non-parametric methods, such as retrieval-augmented generation (RAG) and in-context learning, retrieve relevant documents from an external …

Researchers automated LLM reasoning strategy design and cut token usage by 69.5%

Researchers automated LLM reasoning strategy design and cut token usage by 69.5%

Published by skeptic

Test-time scaling (TTS) has emerged as a proven method to improve the performance of large language models in real-world applications by giving them extra compute cycles at inference time. However, TTS strategies have historically been handcrafted, relying heavily on human intuition to dictate the rules of the model’s reasoning.  To address this bottleneck, researchers from Meta, Google, and several universities have introduced AutoTTS, a framework that automatically discovers optimal TTS strategies. This automated approach allows enterprise organizations to dynamically optimize compute allocation without manually tuning heuristics.  By implementing the optimal strategies discovered by AutoTTS, organizations can directly reduce the token usage and operational costs of deploying advanced reasoning models in production environments. In experimental trials, AutoTTS managed inference budgets efficiently, successfully reducing token consumption by up to 69.5% without sacrificing accuracy. The manual bottleneck in test-time scaling Test-time scaling enhances LLMs by granting them extra compute when generating answers. This extra compute allows the model to generate multiple reasoning paths or evaluate its intermediate steps before arriving at a final response.  The primary challenge for …

Copyediting and Philosophy, Part 3: Language, Power, and Copyediting

Copyediting and Philosophy, Part 3: Language, Power, and Copyediting

Published by Jenni Sidey

The Issues in Philosophy Beat is running a three-part mini-series called “Copyediting and Philosophy,” which focuses on issues around copyediting relevant to the philosophy profession: what it is, how to navigate it as an author, and philosophical questions it raises. This post is the third and final installment. Copyeditors are enforcers of a particular kind, even if they are not the originators of the norms they’re enforcing. Still, we might worry that the practice of copyediting could enforce standard Englishes to the exclusion of other registers and dialects in a pernicious way. Jessica Flanigan, for instance, argues that linguistic pedantry and an overemphasis on conformity with spelling and grammar perpetuates social hierarchies. And a few years ago, an essay by a former copyeditor, Helen Betya Rubinstein, argued that copyediting is a white supremacist project.  I think it’s important to pay attention to prejudices that attach to different varieties of English, whether these be registers, dialects, or “world” or “global” Englishes. At the same time, I think copyediting can be a way to facilitate communication, especially …

I stopped paying for ChatGPT and switched to a local LLM that runs on my laptop

I stopped paying for ChatGPT and switched to a local LLM that runs on my laptop

Published by skeptic

I’ve been really getting into local LLMs lately, to the point where I’ve even tried running them on my phone. My AI subscriptions are only going to get more expensive over time, and if they don’t raise prices, they’ll probably just nerf the token limits instead. So I figured it was time to actually commit to local inference and build a setup that works, rather than waiting until I’m forced to, and I think I’ve got my laptop setup in place now. Related I tried running a chatbot on my old computer hardware and it actually worked You don’t need to fork out for expensive hardware to run an AI on your PC. LM Studio runs my models faster than Ollama Faster is better, duh I already have Ollama set up and working, and I’m keeping it that way. When I need a local model connected to something external, like Claude Code, Ollama is what I use. It just integrates better with other tools and services in my experience, and I don’t see a reason …

Running a Local LLM on a 12-Year-Old Raspberry Pi

Running a Local LLM on a 12-Year-Old Raspberry Pi

Published by skeptic

Running a local AI language model on a 12-year-old Raspberry Pi might seem like an impossible task, but Better Stack demonstrates how it can be done. Using the Falcon H1 Tiny model, which features 90 million parameters and is optimized for low-resource environments, the experiment showcases how advanced techniques like 4-bit quantization and cross-compilation can overcome the severe limitations of the Raspberry Pi’s 700 MHz single-core processor and 512 MB of RAM. By pairing the lightweight Raspberry Pi OS Lite with careful memory management strategies, the setup achieved coherent AI outputs, albeit at a slow pace, proving that even outdated hardware can support modern AI frameworks under the right conditions. In this feature, you’ll explore the specific steps taken to optimize the Raspberry Pi for AI workloads, including the role of quantization in reducing memory demands and how cross-compilation enabled compatibility with the ARMv6 architecture. Gain insight into the trade-offs between model size, processing speed and output quality, as well as the practical limitations of running AI on legacy systems. Whether you’re curious about edge …