Natural Language Processing (NLP)

Natural language processing (NLP) is the branch of artificial intelligence (AI) that lets computers analyze, understand, and generate human language in text and speech. Its tasks range from finding the names in a sentence to translating a book or holding a conversation, and it combines computational linguistics with machine learning and deep learning. Web search, spam filters, voice assistants, phone keyboard suggestions, and chatbots built on large language models (LLMs) all run on NLP.

From Rules to Large Language Models#

The 1954 Georgetown-IBM experiment translated more than sixty Russian sentences into English with a 250-word vocabulary and six grammar rules, and Joseph Weizenbaum's ELIZA (1966) imitated a psychotherapist by matching keywords to scripted replies. Hand-written rules broke on the ambiguity and variety of real text, so from the late 1980s statistical NLP learned probabilities from annotated text instead: hidden Markov models tagged parts of speech, and statistical machine translation learned from millions of translated sentence pairs.

Neural networks took over in the 2010s. Word2vec (2013) learned embeddings that place related words near each other, and recurrent neural networks (RNNs) read sentences one word at a time. The transformer (2017) replaced recurrence with self-attention, which relates every word to every other word in parallel. Pretrained transformers such as BERT and GPT learned language from unlabeled text and then needed only small labeled datasets for each task, and since ChatGPT's release in November 2022, LLMs have handled many NLP tasks from a written instruction alone.

How NLP Systems Represent Text#

A modern NLP system first splits text into tokens, whole words or subword pieces produced by tokenization, and maps each token to an embedding vector. A neural network, usually a transformer, turns that sequence into contextual vectors in which each token's vector reflects its neighbors, and a task-specific output reads them: one label for the whole text, one tag per token, the start and end of an answer span, or newly generated tokens. Because meaning lives in the vectors, sentences with different words can land close together, as the all-MiniLM-L6-v2 sentence encoder shows:

from sentence_transformers import SentenceTransformer

# A small sentence encoder maps each text to a 384-dimensional vector
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

query = "How do I reset my password?"
candidates = ["I forgot my login credentials", "How do I reset my router?"]

# Encode all texts, then compare the query with each candidate by cosine similarity
embeddings = model.encode([query, *candidates])
scores = model.similarity(embeddings[0], embeddings[1:])[0]

for text, score in zip(candidates, scores):
    print(f"{score:.2f}  {text}")

The paraphrase "I forgot my login credentials" scores 0.68 against the question, while "How do I reset my router?", which repeats five of the question's six words, scores 0.53. Semantic search and the retrieval step of retrieval-augmented generation (RAG) rank documents with the same comparison.

Common NLP Tasks#

• Sentiment analysis: Classifies the opinion in a review, post, or survey answer as positive, negative, or neutral.
• Named entity recognition (NER): Tags each mention of a person, organization, place, or date.
• Question answering: Returns the answer to a question instead of a list of documents.
• Text summarization: Condenses a document into a shorter text that keeps its main points.
• Text generation: Writes new text one token at a time, the task behind chatbots and coding assistants.

NLP in Computer Vision#

Language is also how people tell computer vision models what to look for, and combining NLP with vision produced models that read and write about images. Vision-language models (VLMs) answer questions about photos and write image captions, and CLIP learned a shared embedding space for images and text from 400 million image-caption pairs. Open-vocabulary detectors take their class list as text: calling set_classes(["double-decker bus", "person"]) on an Ultralytics YOLOE-26 model tokenizes each name with the CLIP tokenizer and encodes it with Apple's MobileCLIP2 text encoder, and the detector matches those text embeddings against image regions to find and segment categories it was never trained on.

NLP vs. Related Terms#

• Natural language understanding (NLU): The part of NLP concerned with meaning, such as a user's intent or what a pronoun refers to. Natural language generation (NLG), which produces text, is the other main part.
• Large language models: Transformer language models with billions of parameters that perform many NLP tasks through prompting, at a higher cost per request than a small model trained for one task.
• Speech recognition: Converts spoken audio into text, which NLP models then interpret.
• Optical character recognition (OCR): Converts images of printed or handwritten text into characters, the first step before NLP can read a scanned document.