Simon Willison's Weblog: nlp

Finally, a replacement for BERT: Introducing ModernBERT

2024-12-24T06:21:29+00:00

Finally, a replacement for BERT: Introducing ModernBERT

BERT was an early language model released by Google in October 2018. Unlike modern LLMs it wasn't designed for generating text. BERT was trained for masked token prediction and was generally applied to problems like Named Entity Recognition or Sentiment Analysis. BERT also wasn't very useful on its own - most applications required you to fine-tune a model on top of it.

In exploring BERT I decided to try out dslim/distilbert-NER, a popular Named Entity Recognition model fine-tuned on top of DistilBERT (a smaller distilled version of the original BERT model). Here are my notes on running that using uv run.

Jeremy Howard's Answer.AI research group, LightOn and friends supported the development of ModernBERT, a brand new BERT-style model that applies many enhancements from the past six years of advances in this space.

While BERT was trained on 3.3 billion tokens, producing 110 million and 340 million parameter models, ModernBERT trained on 2 trillion tokens, resulting in 140 million and 395 million parameter models. The parameter count hasn't increased much because it's designed to run on lower-end hardware. It has a 8192 token context length, a significant improvement on BERT's 512.

I was able to run one of the demos from the announcement post using uv run like this (I'm not sure why I had to use numpy<2.0 but without that I got an error about cannot import name 'ComplexWarning' from 'numpy.core.numeric'):

uv run --with 'numpy<2.0' --with torch --with 'git+https://github.com/huggingface/transformers.git' python

Then this Python:

import torch
from transformers import pipeline
from pprint import pprint
pipe = pipeline(
    "fill-mask",
    model="answerdotai/ModernBERT-base",
    torch_dtype=torch.bfloat16,
)
input_text = "He walked to the [MASK]."
results = pipe(input_text)
pprint(results)

Which downloaded 573MB to ~/.cache/huggingface/hub/models--answerdotai--ModernBERT-base and output:

[{'score': 0.11669921875,
  'sequence': 'He walked to the door.',
  'token': 3369,
  'token_str': ' door'},
 {'score': 0.037841796875,
  'sequence': 'He walked to the office.',
  'token': 3906,
  'token_str': ' office'},
 {'score': 0.0277099609375,
  'sequence': 'He walked to the library.',
  'token': 6335,
  'token_str': ' library'},
 {'score': 0.0216064453125,
  'sequence': 'He walked to the gate.',
  'token': 7394,
  'token_str': ' gate'},
 {'score': 0.020263671875,
  'sequence': 'He walked to the window.',
  'token': 3497,
  'token_str': ' window'}]

I'm looking forward to trying out models that use ModernBERT as their base. The model release is accompanied by a paper (Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference) and new documentation for using it with the Transformers library.

Via @benjaminwarner.dev

Tags: bert, nlp, python, transformers, ai, jeremy-howard, hugging-face, uv

Matthew Honnibal from spaCy on why LLMs have not solved NLP

2023-09-09T21:30:18+00:00

Matthew Honnibal from spaCy on why LLMs have not solved NLP

A common trope these days is that the entire field of NLP has been effectively solved by Large Language Models. Here’s a lengthy comment from Matthew Honnibal, creator of the highly regarded spaCy Python NLP library, explaining in detail why that argument doesn’t hold up.

Tags: nlp, ai, generative-ai, llms

Closed AI Models Make Bad Baselines

2023-04-03T19:57:09+00:00

Closed AI Models Make Bad Baselines

The NLP academic research community are facing a tough challenge: the state-of-the-art in large language models, GPT-4, is entirely closed which means papers that compare it to other models lack replicability and credibility. “We make the case that as far as research and scientific publications are concerned, the “closed” models (as defined below) cannot be meaningfully studied, and they should not become a “universal baseline”, the way BERT was for some time widely considered to be.”

Anna Rogers proposes a new rule for this kind of research: “That which is not open and reasonably reproducible cannot be considered a requisite baseline.”

Via @emilymbender

Tags: nlp, ai, openai, generative-ai, gpt-4

Quoting Jeonghwan Kim

2023-03-16T05:39:58+00:00

As an NLP researcher I'm kind of worried about this field after 10-20 years. Feels like these oversized LLMs are going to eat up this field and I'm sitting in my chair thinking, "What's the point of my research when GPT-4 can do it better?"

— Jeonghwan Kim

Tags: machine-learning, nlp, ai, generative-ai, gpt-4, llms

Statistical NLP on OpenStreetMap

2018-01-08T19:33:23+00:00

Statistical NLP on OpenStreetMap

libpostal is ferociously clever: it’s a library for parsing and understanding worldwide addresses, built on top of a machine learning model trained on millions of addresses from OpenStreetMap. Al Barrentine describes how it works in this fascinating and detailed essay.

Tags: machine-learning, nlp, openstreetmap

spaCy

2017-11-08T16:43:05+00:00

spaCy

“Industrial-strength Natural Language Processing in Python”. Exciting alternative to nltk—spaCy is mostly written in Cython, makes bold performance claims and ships with a range of pre-built statistical models covering multiple different languages. The API design is clean and intuitive and spaCy even includes an SVG visualizer that works with Jupyter.

Tags: nlp, python, spacy

Oxford Deep NLP 2017 course

2017-10-31T20:39:17+00:00

Oxford Deep NLP 2017 course

Slides, course description and links to lecture videos for the 2017 Deep Natural Language Processing course at the University of Oxford presented by a team from Google DeepMind.

Via Hacker News

Tags: google, machine-learning, nlp

Which investors would consider a natural language processing startup in London?

2013-09-30T15:52:00+00:00

My answer to Which investors would consider a natural language processing startup in London? on Quora

I don't know the answer, but I know how you can find it: track down as many London-based AI/machine learning/NLP startups as you can and look at who their investors are.

Tags: london, nlp, startups, quora

topia.termextract

2009-08-10T21:26:02+00:00

topia.termextract

Impressive Python term extraction library (similar to the various term extraction web APIs but you can run it on your own hardware), incorporating a Parts-Of-Speech tagging algorithm.

Tags: nlp, python, termextraction, topia

JS-Placemaker - geolocate texts in JavaScript

2009-05-23T00:36:38+00:00

JS-Placemaker - geolocate texts in JavaScript

Chris Heilmann exposed Placemaker to JavaScript (JSONP) using a YQL execute table. Try his examples—I’m impressed that “My name is Jack London, I live in Ontario” returns just Ontario, demonstrating that Placemaker’s NLP is pretty well tuned.

Tags: christian-heilmann, geocoding, geospatial, javascript, jsonp, nlp, placemaker, yahoo, yql, yqlexecute