The email began "thanks so much for absolutely nothing" and the sentiment model tagged it positive. It saw "thanks", it saw "so much", it did the arithmetic it was built to do, and it never stood a chance against a customer who was furious and being sarcastic about it. The support team only found out when the same customer cancelled three weeks later, still marked green in the dashboard the whole time.
Natural language processing is the field that tries to close that gap: getting computers to work with human language rather than just storing it. Text and speech are the least structured data most organisations hold, full of ambiguity, context and habits nobody wrote down, and NLP is the collection of techniques for pulling something usable out of it: a category, a fact, a summary, a translation, a judgement about tone.
We build data and AI systems at Shipshape Data, and NLP is usually where a project's ambition meets its actual text for the first time. The plan sounds clean in a slide. Then somebody opens the real support inbox, or the real contract archive, or the real call transcripts, and the language in front of them is messier than anyone budgeted for. This guide covers what NLP actually is, the handful of core tasks that make up most real systems, how transformers changed what is possible, where businesses genuinely get value from it, and the failure modes that do not show up in a vendor demo.
Why language keeps breaking software that works fine on numbers
A spreadsheet does not misunderstand you. A number is a number, a date sorts correctly, a total either balances or it does not. Language is a different kind of problem entirely, because the same string of words can mean several things depending on who said it, when, and to whom.
"I could care less" means the opposite of what it says, and everyone understands it anyway. "Book" is a noun until it is a verb. A customer writing "not bad" might mean genuinely fine, or might mean the most damning thing they are willing to put in an email. Humans resolve all of this instantly using context we barely notice we are applying: tone, history, what came before in the conversation, what industry we are in.
The context problem
Software has none of that by default. A word taken on its own carries only a fraction of its meaning, and a huge amount of NLP's difficulty is really the difficulty of representing context cheaply enough to compute at scale. Early systems tried to shortcut this with rules and dictionaries. They worked for narrow, predictable language and fell apart the moment somebody phrased something slightly sideways, which in real customer text is most of the time.
What natural language processing actually is
NLP is a branch of AI concerned with reading, generating and reasoning over human language, spoken or written. That covers a wide span of work: a spell checker is NLP, and so is a system extracting obligations from a thousand supplier contracts overnight. The techniques underneath differ hugely, but the goal is the same. Take language, which is inherently loose and ambiguous, and produce something structured or actionable from it.
Understanding versus generating
It helps to split NLP into two directions, because they get evaluated differently and fail differently. Natural language understanding takes text in and produces structure out: a category, a sentiment score, a set of extracted entities. Natural language generation takes structure or a prompt and produces fluent text out: a summary, a reply, a translation. Most business systems need both directions working together, which is part of why building a genuinely reliable NLP feature takes more than wiring up a single model and hoping.
Nothing about this is new, either. Machine translation research goes back to the 1950s and rule-based grammar checkers to the decades after. What changed recently is not the ambition, it is what actually works, and that is a story mostly about one architecture.
The core tasks that make up most real systems
Strip away the branding and almost every NLP product in production is doing one, or a combination, of five things. Knowing which one you actually need is most of the design work.
Classification
Classification sorts text into predefined buckets: spam or not spam, this ticket goes to billing or to technical support, this review is a complaint about shipping or about the product itself. It is the oldest and most forgiving of the five tasks, because the categories are fixed in advance and the model only has to pick one. Get your categories wrong, though, and no amount of modelling saves you. A support queue with eleven overlapping tags trained the model to be confused before it saw a single example.
Extraction
Extraction pulls specific pieces out of text rather than judging the whole thing: a date, an invoice number, a clause, a person's name, a claimed diagnosis in a clinical note. This is where NLP meets a filing cabinet, and it is genuinely useful precisely because so much valuable business information sits inside documents rather than inside fields in a database. A supplier contract with a termination clause buried in paragraph fourteen is exactly the kind of thing extraction is built for, and exactly the kind of thing a tired human reviewer misses on a Friday afternoon.
Sentiment
Sentiment analysis scores text as positive, negative or neutral, and it is by far the most oversold of the five. It works reasonably well on plain, direct language: "the delivery was late and the box was damaged" scores negative without much argument. It works badly on sarcasm, mixed feeling, and anything culturally coded, which is a large share of what people actually write when they are annoyed. Treat a sentiment score as a rough signal worth spot-checking, not a verdict.
Summarisation
Summarisation compresses a long document down to its essential points. Extractive summarisation lifts existing sentences out of the source, which is safer because every word is traceable to something that was actually written. Abstractive summarisation generates new sentences that capture the meaning, which reads better and is where the risk sits, because a fluent summary can quietly drop a caveat or invent a connection that was not in the source. For anything that gets acted on, somebody needs to check what got left out as carefully as what got kept.
Translation
Machine translation converts text from one language to another, and it is the task most people already trust without thinking about it, because a decade of steady improvement in consumer tools trained everyone to expect it to just work. It mostly does now, for common language pairs and everyday text. It gets noticeably worse the moment you introduce specialist vocabulary, legal phrasing, or a language pair with less training data behind it, and a translation that reads fluently is not the same thing as a translation that is correct.
How transformers changed the field
For decades, NLP models read text roughly the way you would read with a torch in a dark room: one word at a time, carrying forward whatever you could remember of what came before. That is what the older recurrent architectures did, and the further back in a sentence the relevant information sat, the more of it got lost by the time the model reached the end. A pronoun forty words after the noun it referred to was, in practice, often invisible to the model.
The transformer architecture, introduced in 2017, changed the mechanism entirely. Instead of reading in sequence, it uses a mechanism called attention to weigh every word against every other word in the input at once, regardless of distance. The model can look at "it" near the end of a paragraph and connect it directly back to the noun at the start, without having to have remembered anything along the way. That single change is most of why NLP progress accelerated so sharply from the late 2010s onward.
From task-specific models to one model, many tasks
The practical consequence mattered more than the architecture itself. Before transformers, a business typically trained a separate model for classification, another for extraction, another for sentiment, each needing its own labelled dataset. Large language models built on transformers, trained on enormous amounts of general text, can do a reasonable version of most of these tasks out of the box, often from a well-written instruction rather than a bespoke training run. How much of that quality now comes down to how the task is framed, rather than any further training at all, is its own discipline and a fair amount of trial and error.
This did not eliminate the value of dedicated models. A narrow, well-labelled classifier trained on your own support tickets will often still beat a general-purpose model on your specific categories, and it will be cheaper to run and far easier to explain. What changed is the starting point. Teams no longer need months of labelling before they see anything work at all, which means the real design decision now is when a general model is good enough and when the task is specific enough to warrant training your own.
A fluent answer and a correct answer are not the same thing, and language models are exceptionally good at producing the first one regardless of whether they have the second.
Where businesses actually use this
The most common entry point is still the support inbox: routing tickets, drafting first-response replies, flagging the ones that need a human urgently. It is a good starting point precisely because the volume is high, the categories are usually well understood already, and a mistake is recoverable, a badly routed ticket gets rerouted rather than causing real damage.
Contract and document review is the second big one, using extraction to pull obligations, dates, and risk clauses out of documents that used to require a lawyer's afternoon to skim. Search is a third, and it has changed shape more than any other application: modern retrieval understands that "how do I get my money back" and "refund process" mean roughly the same thing, using the same underlying representations covered in our piece on embeddings, which is what lets a search box work on meaning rather than exact keyword matches.
The quieter uses
Beyond the obvious ones, plenty of NLP work happens invisibly inside other systems. Compliance teams scan communications for regulated language. Market research teams summarise thousands of open-text survey responses that nobody was ever going to read individually. Recruitment tools parse CVs into structured fields. None of this looks like an AI product from the outside. It looks like the tool doing its job slightly faster than it used to, which is usually the sign that NLP has been deployed sensibly rather than for its own sake.
The failure modes that don't show up in the demo
Every one of these tasks has a way of quietly going wrong that is different from the ways ordinary software goes wrong, because the failure is confident rather than obvious.
Fluent does not mean correct
A generated summary can be grammatically perfect and factually off, dropping a condition or inventing a relationship between two facts that were never actually linked in the source. This matters more the further the output sits from a human who will double-check it, and it is a large part of why deploying anything generative into a workflow with real consequences needs an ongoing review step. A good-looking test run tells you almost nothing about the edge cases waiting in production.
Bias rides along with the training data
Models learn from the text they were trained on, and if that text carries skewed patterns around gender, accent, dialect or region, the model reproduces them, often in ways nobody notices until a specific group of customers keeps getting misclassified or underserved. A sentiment model trained mostly on one register of English will handle a different one worse, quietly and without an error message. This is one of several places where our broader notes on AI bias apply directly, and NLP is one of the areas where the consequence lands on real people rather than staying abstract.
Domain language breaks general models
General-purpose models are trained on the internet, broadly. They handle everyday English well and specialist vocabulary considerably worse, whether that is clinical shorthand, engineering standards, or the particular jargon your industry has accumulated over twenty years. A model that performs beautifully in a generic demo can stumble badly the first time it meets your actual documents, which are rarely written in the tidy register the demo used.
Buying a tool versus building on a model
Most organisations now face a genuine choice rather than a forced one. Off-the-shelf tools handle common tasks well: general sentiment, standard classification, everyday translation. They are quick to trial and cheap to abandon if they do not work.
Building on a general-purpose language model, wired into your own documents so it can look things up rather than guess, earns its cost when the task involves your specific vocabulary, your specific documents, or a volume high enough that a small accuracy gain pays for itself many times over. It is also the option that asks the most of you, because retrieval quality depends on how well your underlying documents are organised in the first place, which is usually a data problem wearing an AI costume rather than an AI problem in its own right.
Fine-tuning a model on your own labelled examples sits between the two, and it is worth reaching for only when a general model has genuinely been tried and found short, not as the default first move. It is more expensive to set up and maintain, and it commits you to a labelling effort that off-the-shelf tools let you skip entirely.
Where to start
Pick one task, not five. Classification and extraction tend to be the safest entry points because the output is checkable: either the ticket landed in the right queue or it did not, either the date got pulled correctly or it did not. Sentiment and generation are harder to grade at a glance, so save them for once you have a feel for how the rest of the system behaves.
Look at your actual text before choosing a tool, not a sample somebody tidied up for the pitch. Most organisations' real language, the actual support tickets, the actual contracts, the actual call transcripts, is messier than whatever a vendor used in their demo, and that gap is where projects quietly go wrong. Our guide to unstructured data covers what that mess typically looks like and what it takes to get it into a state any model can work with reliably.
Then decide who checks the output, before you decide which model produces it. NLP systems fail quietly and fluently, and the safety net that catches a wrong translation or a hallucinated summary line is a person, not a confidence score. If you are looking at where language work could genuinely save your team time, or you have text sitting in your business that nobody has found a reliable way to use yet, talk to us.