You don't need to install anything. There are two files (mojibake.c, mojibake.h) to add to your
C/C++ project. Download it here mojibake-amalgamation-035.zip.
Examples of normalization, characters count and NFKC casefold.
#include <stdio.h>
#include <string.h>
#include "mojibake.h"
void print_string(const char *input, size_t length);
int main(int argc, char *const argv[]) {
const char *input = "Cafe\xCC\x81";
size_t length = strlen(input);
mjb_result result;
// Normalize example: in NFC e + ◌́ -> é (U+00E9)
if(mjb_normalize(input, length, MJB_ENC_UTF_8, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8,
&result) != MJB_STATUS_OK) {
return 1;
}
// Cafe + ◌́ (U+0301, COMBINING ACUTE ACCENT) -> Café
print_string(input, length);
// Caf + é (U+00E9, LATIN SMALL LETTER E WITH ACUTE) -> Café
print_string(result.output, result.output_size);
const char *mojibake = "文字化け";
length = strlen(mojibake);
// Codepoint count example: mjb_count_codepoints counts Unicode codepoints, not bytes.
printf("\"%s\" encoded in UTF-8 is %zu bytes long, and %zu codepoints long\n",
mojibake, length, mjb_count_codepoints(mojibake, length, MJB_ENC_UTF_8));
mjb_result_free(&result);
const char *case_input = "Straße";
// NFKC casefold example: in NFKC casefold, ß -> ss
if(mjb_nfkc_casefold(case_input, MJB_NUL_TERMINATED, MJB_ENC_UTF_8, MJB_ENC_UTF_8,
&result) != MJB_STATUS_OK) {
return 1;
}
printf("%s -> %.*s\n", case_input, (int)result.output_size, result.output);
mjb_result_free(&result);
return 0;
}
void print_string(const char *input, size_t length) {
for(size_t i = 0; i < length; ++i) {
unsigned char byte = (unsigned char)input[i];
if(byte >= 0x21 && byte <= 0x7E) {
printf("%c", byte);
} else {
printf("<%02X>", byte);
}
}
printf("\n");
}
This output:
Cafe<CC><81>
Caf<C3><A9>
"文字化け" encoded in UTF-8 is 12 bytes long, and 4 characters long
Straße -> strasse
Pass an exact byte length when the input may contain U+0000. For a properly terminated input, pass
MJB_NUL_TERMINATED; Mojibake scans one-, two-, or four-byte code units according to the declared
encoding and excludes the terminator.
Mojibake aims to be:
- Small
- Easy to use
- Fast
- Self-contained
Mojibake do:
- Run in all modern OSes
- Pass the official Unicode test suites for supported algorithms
- Implement all Unicode standard algorithms
- Satisfy all Unicode Conformance Requirements
Feature highlights
All the C files, together with the Unicode data tables, are concatenated into a single large file
and header: mojibake.c and mojibake.h. Zero dependencies.
Text transformation
- Normalization: NFC/NFD/NFKC/NFKD (
mjb_normalize,mjb_normalize_into), identifier-oriented NFKC case folding (mjb_nfkc_casefold,mjb_nfkc_casefold_into), plus a fast quick-check (mjb_normalization_quick_check) (UAX #15, Unicode 18.0.0) - Case conversion: uppercase, lowercase, titlecase, and case folding with full special-casing
and conditional mappings (
mjb_map_case,mjb_map_case_into) - Internationalized domain names: strict nontransitional ToASCII and ToUnicode processing,
including Punycode and validation (
mjb_idna_to_ascii,mjb_idna_to_unicode) (UTS #46) - Filtering: strip controls, spaces, or numeric characters while normalizing
(
mjb_filter,mjb_filter_into)
Text analysis
- Character database: every Unicode Character Database property: category, script and
Script_Extensions, block, plane, numeric value, name (
mjb_codepoint_info,mjb_codepoint_script_extensions) - Segmentation: grapheme clusters, words, sentences, and line-break opportunities (UAX #29, Unicode 18.0.0, UAX #14, Unicode 18.0.0)
- Bidirectional text: full Unicode Bidirectional Algorithm: paragraph resolution, line reordering, runs (UAX #9, Unicode 18.0.0)
- Emoji: codepoint properties, sequence analysis, RGI emoji detection
- Terminal width: grapheme-aware terminal-cell estimates with explicit narrow or East Asian
profiles, emoji-sequence handling, and width-aware truncation
(
mjb_terminal_width,mjb_truncate_grapheme_width)
Sorting and comparison
- Caseless matching: Unicode default, canonical, compatibility, and identifier
case-insensitive equality (
mjb_caseless_match, Unicode 18.0.0 Section 3.13.5) - Collation: Unicode Collation Algorithm string comparison and sort keys, with primary through
quaternary strengths and shifted or non-ignorable variable weighting (
mjb_collation_compare,mjb_collation_key,mjb_collation_key_into, UTS #10, Unicode 18.0.0)
Security
- Mixed-script detection: resolve a string to its augmented Unicode script set for
identifier-policy checks (
mjb_resolved_script_set, UTS #39, Unicode 18.0.0) - Confusable detection: generate reusable skeletons and check if strings are visually
confusable (
mjb_confusable_skeleton,mjb_confusable_skeleton_into,mjb_are_confusable, UTS #39, Unicode 18.0.0) - Identifier validation: XID/ID checks for parser and compiler authors
(
mjb_is_identifier, UAX #31, Unicode 18.0.0)
Integration
- Encodings: the API accepts and outputs UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE
strings, with encoding detection and conversion (
mjb_detect_encoding,mjb_convert_encoding,mjb_convert_encoding_into) - Parsing and string functions: character-by-character iteration (
mjb_for_each_character) and codepoint counting (mjb_count_codepoints) - Locales: strict BCP 47 language tag parsing (
mjb_locale_parse) - Embeddable: custom allocators (
mjb_set_memory_functions), build-time feature flags to trim table size, a C++17 wrapper (src/cpp/mojibake.hpp), a CLI tool (src/shell), and a WASM + TypeScript API (src/api) - Tested: Mojibake uses Attractor as test suite and run 1.5M+ assertions including the official Unicode conformance suites for supported algorithms
- Fuzz Mojibake is fuzzed with libFuzzer over untrusted byte input
AddressSanitizerandUBSanclean
Minimum requirements
Mojibake requires a C11 compiler. Building the sources as C++, including the header-only C++ wrapper, requires C++17. The minimum supported compiler versions are:
| Toolchain | Minimum version |
|---|---|
| GCC | 9 |
| LLVM Clang | 10 |
| Apple Clang | 10 |
| MSVC | 19.30 (Visual Studio 2022) |
MinGW-w64 and Emscripten (WASM) are also tested as rolling compatibility targets.
Mojibake is tested on:
- Ubuntu Linux (gcc, g++, clang, clang++)
- Ubuntu Linux ARM64 (gcc, g++)
- Ubuntu Linux (Android NDK / ARM64)
- Alpine Linux (musl x86_64, musl i386, musl s390x)
- macOS (clang, clang++)
- FreeBSD 15.1 (clang, clang++)
- OpenBSD 7.9 (clang, clang++)
- NetBSD 10.1 (clang, clang++)
- Haiku R1/beta5 (clang, clang++)
- Windows 11 (MSVC, MinGW-w64)
CMake dependency
Mojibake can be embedded with either add_subdirectory or FetchContent and exposes the
Mojibake::mojibake target:
include(FetchContent)
FetchContent_Declare(
Mojibake
GIT_REPOSITORY https://github.com/zaerl/mojibake.git
GIT_TAG main
)
FetchContent_MakeAvailable(Mojibake)
target_link_libraries(your-target PRIVATE Mojibake::mojibake)
Tests, the CLI, and installation rules default to enabled for a standalone build and disabled when Mojibake is embedded. They can be controlled with:
MJB_BUILD_TESTSMJB_BUILD_CLIMJB_INSTALL
Other project options use the same MJB_ prefix: MJB_BUILD_CPP, MJB_BUILD_WASM, MJB_USE_ASAN,
MJB_USE_UBSAN, MJB_FEATURE_CHARACTER_NAMES, MJB_FEATURE_COLLATION, MJB_FEATURE_IDNA, and
MJB_FEATURE_SECURITY.
Use CMake's standard BUILD_SHARED_LIBS option to select a shared or static library.
C++17 wrapper
Mojibake also has a tiny C++ wrapper. Check here for details: src/cpp/README.md
Build-time features
Mojibake can compile out optional implementations and tables to reduce binary size. Features
default to enabled. CMake and the root Makefiles use the boolean values ON and OFF; when
invoking a C or C++ compiler directly, define the corresponding preprocessor macros as numeric
1 or 0.
MJB_FEATURE_CHARACTER_NAMEScontrols the Unicode character-name tables used bymjb_codepoint_info(...)to fillmjb_character.name. When disabled, the tables are not compiled andmjb_character.nameis reported asCodepoint U+XXXX. This will reduce the output of ~30%.MJB_FEATURE_COLLATIONcontrols the Unicode Collation Algorithm implementation and DUCET tables. Themjb_collation_*functions returnMJB_STATUS_FEATURE_NOT_ENABLEDwhen support is disabled.MJB_FEATURE_IDNAcontrols the UTS #46 implementation, Punycode implementation, and IDNA mapping tables. Themjb_idna_*functions returnMJB_STATUS_FEATURE_NOT_ENABLEDwhen support is disabled.MJB_FEATURE_SECURITYcontrols the UTS #39 resolved-script and confusable implementations and confusable mapping tables. Themjb_resolved_script_set,mjb_confusable_skeleton,mjb_confusable_skeleton_into, andmjb_are_confusablefunctions returnMJB_STATUS_FEATURE_NOT_ENABLEDwhen support is disabled.
With CMake:
cmake -S . -B build-minimal \
-DMJB_FEATURE_CHARACTER_NAMES=OFF \
-DMJB_FEATURE_COLLATION=OFF \
-DMJB_FEATURE_IDNA=OFF \
-DMJB_FEATURE_SECURITY=OFF
cmake --build build-minimal
With the provided Makefile:
make build BUILD_DIR=build-minimal FEATURE_CHARACTER_NAMES=OFF FEATURE_COLLATION=OFF \
FEATURE_IDNA=OFF FEATURE_SECURITY=OFF
make test-no-names
make test-no-collation
make test-no-idna
make test-no-security
When compiling the amalgamation directly:
cc -DMJB_FEATURE_CHARACTER_NAMES=0 -DMJB_FEATURE_COLLATION=0 -DMJB_FEATURE_IDNA=0 \
-DMJB_FEATURE_SECURITY=0 example.c mojibake.c -o example
API documentation
See API.md or the site for the detailed documentation.
CLI
The src/shell directory builds the mojibake CLI used to test the library. Example usage:
# This outputs "NFC: Café", e + ◌́ -> é
mojibake nfc $'Cafe\u0301'
# The output an emoji sequence [1] Basic, [2] Fully-qualified of two characters U+263A U+FE0F
mojibake emoji "☺️"
The amalgamation .zip also contains an amalgamation of the shell called shell.c.
Building from source and contributing
See CONTRIBUTING.md for instructions.
Licenses
Mojibake is released under the MIT License (see LICENSE).
Legalese
Here you can find the very detailed and boring informations needed to have this library conformant to the Unicode standard, or at least what I got, at CONFORMANCE_REQUIREMENTS.md.