24 Text Databases and Structured Data
Guiding Question: Can ordinary text files become databases?
When people hear the word database, they often imagine large servers, specialized software, and complex administration.
Yet many of the world’s most useful collections of information require nothing so elaborate.
A list of contacts.
A bibliography.
A catalogue of books.
A collection of research notes.
A financial ledger.
An inventory.
These are all structured collections of information.
Many of them can be represented perfectly well using ordinary text files.
The Unix tradition has long embraced this idea.
Sometimes, the simplest database is simply a well-structured text file.
24.1 Structure Creates Information
Earlier in this primer we learned that markup gives plain text structure.
Structured data follows the same principle.
Rather than storing isolated lines of text, we organize information into consistent records.
Once that structure exists, software can search, sort, transform, compare, and analyse the information with remarkable ease.
The distinction between documents and data becomes surprisingly small.
24.2 CSV and TSV
Perhaps the simplest structured text formats are CSV (Comma-Separated Values) and TSV (Tab-Separated Values).
Each line represents a record.
Each separator divides the record into fields.
Conceptually:
text id="80z3up" Name,Occupation,Country Alice,Writer,Zimbabwe Bob,Teacher,Kenya
Despite their simplicity, these formats power spreadsheets, scientific datasets, financial records, and countless data exchange systems.
Because they remain plain text, they work naturally with many of the Unix tools explored earlier in this part.
24.3 JSON
As software systems became increasingly interconnected, a richer format became desirable.
JSON (JavaScript Object Notation) emerged as one of the most influential solutions.
Unlike CSV, JSON can represent nested structures.
Objects.
Lists.
Hierarchies.
Relationships.
Its readability and flexibility explain why it has become one of the dominant formats for modern software communication.
Although more expressive than CSV, JSON remains fundamentally text.
24.4 YAML
Another popular structured text format is YAML.
Configuration files for modern software frequently use YAML because of its readability.
Rather than emphasizing punctuation, YAML relies largely upon indentation to express structure.
Many contemporary publishing systems, static site generators, and automation tools employ YAML to store metadata and configuration.
Readers of this primer have already encountered YAML through Quarto’s document headers.
24.5 Querying Structured Data
As structured text became increasingly common, specialized tools appeared to work with it.
For JSON, one of the most influential is jq.
It allows complex JSON documents to be searched, filtered, transformed, and reformatted using concise expressions.
For YAML, similar capabilities are provided by yq.
Rather than abandoning the Unix philosophy, these tools extend it.
They bring familiar ideas of filtering and transformation into newer structured formats.
24.6 SQLite: The Portable Database
Although SQLite is not a plain text format, it deserves mention because it shares many of the values celebrated throughout this primer.
An SQLite database resides within a single portable file.
There is no separate server.
No complex installation.
No background service.
The database moves with the file.
For many personal and embedded applications, SQLite demonstrates that sophisticated data management need not require elaborate infrastructure.
It forms a natural bridge between plain text workflows and traditional relational databases.
24.7 Choosing the Right Format
Different problems call for different forms of structured information.
CSV excels at tabular data.
JSON represents hierarchical information.
YAML emphasizes human readability.
SQLite provides efficient relational storage.
The experienced textsmith does not ask which format is universally best.
Instead, the question becomes:
“Which structure best represents the information?”
The format follows the purpose.
24.8 Text as Long-Term Storage
One of the enduring strengths of structured text lies in its longevity.
Plain text formats remain:
- searchable
- portable
- version controllable
- inspectable
- editable
- independent of particular applications
Even decades later, the information remains accessible.
That durability explains why text continues to occupy such an important place within modern computing.
24.9 Lessons for the Textsmith
This chapter reveals another important truth.
Documents and databases are not opposing worlds.
They exist along a continuum.
As information becomes increasingly structured, plain text naturally begins to resemble a database.
Likewise, portable databases such as SQLite preserve many of the same qualities that have made plain text successful for generations.
The true question is therefore not whether information belongs in a document or a database.
The question is how best to represent knowledge so that both people and computers can understand it.
24.10 Key Ideas
- Structured text formats organize information into consistent records.
- CSV and TSV provide simple yet remarkably effective tabular storage.
- JSON represents hierarchical information while remaining human-readable.
- YAML emphasizes readability and is widely used for configuration.
- Tools such as
jqandyqextend Unix text-processing ideas to structured data. - SQLite bridges the gap between plain text workflows and traditional databases through its single-file design.
- Structured text demonstrates that many databases can remain portable, searchable, and long-lived.
In the next chapter, we broaden our perspective once more.
How do we search not merely files, but entire collections of knowledge?
There we explore indexing, full-text search, desktop search tools, and the remarkable systems that allow textsmiths to navigate libraries containing millions of documents.