Mojibake
Mojibake is a small, fast, self-contained Unicode library written in C11. It is released under the MIT License.
C API
API reference
Functions are organized by their metadata section. Choose Try it to see detailed behavior and use the live WASM form, or Details for C-only functions.
No installation required
Try every function in your browser
Each runnable function below includes a live form backed by the WASM build. Choose Try it, enter its arguments, and inspect the result immediately.
API section
Text transformation
Normalize, case-convert, filter, and convert Unicode text.
mjb_normalize
Normalize a string to NFC/NFKC/NFD/NFKD form.
mjb_status mjb_normalize(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_normalization form,
mjb_encoding output_encoding,
mjb_result *result,
mjb_diagnostic *diagnostic
);
Normalize a string to the requested Unicode normalization form. If the input is already normalized and no encoding conversion is needed, text->buffer is returned as-is in result->output with result->owned set to false, without allocating. Malformed subsequences follow malformed_policy, and diagnostic records the first one.
Returns
MJB_STATUS_OK- The string was normalized (or already normal)MJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_INVALID_FORM-formis not NFC, NFD, NFKC, or NFKDMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a normalized codepointMJB_STATUS_OVERFLOW- The output size would overflowMJB_STATUS_NO_MEMORY- Allocation failed
Examples
// "Cafe" + U+0301 COMBINING ACUTE ACCENT
mjb_text input = mjb_text_utf8("Cafe\xCC\x81", MJB_NUL_TERMINATED);
mjb_result result;
if(mjb_normalize(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result,
NULL) != MJB_STATUS_OK) {
return 1;
}
// NFC: Café
printf("NFC: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);// "Cafe" + U+0301 COMBINING ACUTE ACCENT
std::string input = "Cafe\xCC\x81";
std::string output = mjb::normalize(input, mjb::NormalizationForm::NFC);
// NFC: Café
std::cout << "NFC: " << output << std::endl;Related
Specifications
mjb_normalize_into
Normalize a string into a caller-provided buffer.
mjb_status mjb_normalize_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_normalization form,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Normalize a string using the same Unicode normalization forms and encoding rules as mjb_normalize. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. NFD and NFKD write without allocation. NFC and NFKC may allocate temporary composition storage, including during a size query.
Returns
MJB_STATUS_OK- The required size was returned or the normalized string was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_INVALID_FORM-formis not NFC, NFD, NFKC, or NFKDMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a normalized codepointMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_NO_MEMORY- Temporary composition allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte count
Examples
const char *cafe = "Cafe\xCC\x81"; // "Cafe" + U+0301 COMBINING ACUTE ACCENT
mjb_text input = mjb_text_utf8(cafe, strlen(cafe));
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
// Query the required size
if(mjb_normalize_into(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
char storage[5];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_normalize_into(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
// NFC payload (no terminator): Café
printf("NFC payload (no terminator): %.*s", (int)output.size, storage);// "Cafe" + U+0301 COMBINING ACUTE ACCENT
std::string input = "Cafe\xCC\x81";
std::string output;
// Replaces the content of output and reuses its capacity on later calls
mjb::normalize(input, mjb::NormalizationForm::NFC, output);
// NFC: Café
std::cout << "NFC: " << output << std::endl;Related
Specifications
mjb_filter
Filter a string with the selected mjb_filter_flags.
mjb_status mjb_filter(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_filter_flags filters,
mjb_encoding output_encoding,
mjb_result *result,
mjb_diagnostic *diagnostic
);
MJB_FILTER_LIMIT_COMBINING removes combining marks after the first MJB_FILTER_MAX_COMBINING_MARKS consecutive marks in an emitted run. Every combining mark counts, in any block: general categories Mn, Mc, and Me. An emitted character that is neither a combining mark nor a Default_Ignorable_Code_Point, such as U+200D ZERO WIDTH JOINER, starts a new run. This is useful for reducing Zalgo-style text while keeping ordinary accents and stacked marks. Some scripts, such as Biblical Hebrew and Quranic Arabic, can use more than four marks on one letter; define MJB_FILTER_MAX_COMBINING_MARKS at compile time to raise the default of 4. Malformed subsequences are stopped, replaced, or skipped according to malformed_policy; diagnostic records the first one encountered.
Returns
MJB_STATUS_OK- The filtered string was returnedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a filtered codepointMJB_STATUS_OVERFLOW- The output size would overflowMJB_STATUS_NO_MEMORY- Allocation failed
Examples
mjb_text mixed_whitespace = mjb_text_utf8("Hello\t\t\n\nworld", MJB_NUL_TERMINATED);
mjb_result result;
if(mjb_filter(&mixed_whitespace, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, MJB_ENC_UTF_8,
&result, NULL) != MJB_STATUS_OK) {
return 1;
}
// Filtered: Hello world
printf("Filtered: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);
mjb_text controls = mjb_text_utf8("\x1\x2\t\n\v\f\r\x1f", MJB_NUL_TERMINATED);
if(mjb_filter(&controls, MJB_MALFORMED_STOP, MJB_FILTER_CONTROLS, MJB_ENC_UTF_8, &result,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Filtered: \t\n\v\f\r
printf("Filtered: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);std::string mixed_whitespace = "Hello\t\t\n\nworld";
// Filtered: Hello world
std::cout << "Filtered: " << mjb::filter(mixed_whitespace, mjb::Filter::CollapseSpaces)
<< std::endl;
std::string controls = "\x1\x2\t\n\v\f\r\x1f";
// Filtered: \t\n\v\f\r
std::cout << "Filtered: " << mjb::filter(controls, mjb::Filter::Controls) << std::endl;Related
mjb_filter_into
Filter a string into a caller-provided buffer.
mjb_status mjb_filter_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_filter_flags filters,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Apply the same filters as mjb_filter without allocating the final output buffer. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Filtering itself does not allocate, but MJB_FILTER_NORMALIZE may allocate temporary normalization storage. MJB_FILTER_LIMIT_COMBINING keeps the first MJB_FILTER_MAX_COMBINING_MARKS consecutive marks in each emitted run, counting every Mn, Mc, and Me mark and ignoring Default_Ignorable_Code_Point characters as mjb_filter does. Malformed subsequences follow malformed_policy, and diagnostic records the first one.
Returns
MJB_STATUS_OK- The required size was returned or the filtered string was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a filtered codepointMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_NO_MEMORY- Temporary normalization allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte count
Examples
mjb_text input = mjb_text_utf8("Hello\t\t\nworld", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
// Query the required size
if(mjb_filter_into(&input, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
char storage[11];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_filter_into(&input, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Filtered payload (no terminator): Hello world
printf("Filtered payload (no terminator): %.*s", (int)output.size, storage);std::string input = "Hello\t\t\nworld";
std::string output;
// Replaces the content of output and reuses its capacity on later calls
mjb::filter(input, mjb::Filter::CollapseSpaces, output);
// Filtered: Hello world
std::cout << "Filtered: " << output << std::endl;Related
mjb_nfkc_casefold
Apply the Unicode NFKC_Casefold transform to a string.
mjb_status mjb_nfkc_casefold(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_encoding output_encoding,
mjb_result *result,
mjb_diagnostic *diagnostic
);
Apply the normative NFKC_Casefold mapping and normalize the result to NFC. This transform performs compatibility folding, full default case folding, and removal of default-ignorable codepoints. It is intended for identifier comparison and is not locale-sensitive. Malformed subsequences follow malformed_policy, and diagnostic records the first one.
Returns
MJB_STATUS_OK- The transformed string was returnedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- The output size would overflowMJB_STATUS_UNSUPPORTED- The transform did not stabilize, or the requested output encoding cannot represent a result codepointMJB_STATUS_NO_MEMORY- Allocation failed
Examples
mjb_text input = mjb_text_utf8("Stra\xC3\x9F" "e\xC2\xAD", MJB_NUL_TERMINATED);
mjb_result result;
if(mjb_nfkc_casefold(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_8, &result,
NULL) != MJB_STATUS_OK) {
return 1;
}
// strasse
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);// "Straße" + U+00AD SOFT HYPHEN
std::string input = "Stra\xC3\x9F" "e\xC2\xAD";
// strasse
std::cout << mjb::nfkc_casefold(input) << std::endl;Related
Specifications
mjb_nfkc_casefold_into
Apply the Unicode NFKC_Casefold transform into a caller-provided buffer.
mjb_status mjb_nfkc_casefold_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Apply the same normative NFKC_Casefold transform as mjb_nfkc_casefold. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. The final output uses caller-provided storage, but the normalization and folding passes require temporary allocations, including during a size query.
Returns
MJB_STATUS_OK- The required size was returned or the transformed string was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_UNSUPPORTED- The transform did not stabilize, or the requested output encoding cannot represent a result codepointMJB_STATUS_NO_MEMORY- Temporary allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte count
Examples
mjb_text input = mjb_text_utf8("Stra\xC3\x9F" "e\xC2\xAD", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
// Query the required size
if(mjb_nfkc_casefold_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
char storage[7];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_nfkc_casefold_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
// NFKC casefold payload (no terminator): strasse
printf("NFKC casefold payload (no terminator): %.*s", (int)output.size, storage);// "Straße" + U+00AD SOFT HYPHEN
std::string input = "Stra\xC3\x9F" "e\xC2\xAD";
std::string output;
// Replaces the content of output and reuses its capacity on later calls
mjb::nfkc_casefold(input, output);
// NFKC casefold: strasse
std::cout << "NFKC casefold: " << output << std::endl;Related
Specifications
mjb_idna_to_ascii
Convert a domain name to its UTS #46 nontransitional ASCII form.
mjb_status mjb_idna_to_ascii(
const mjb_text *text,
mjb_encoding output_encoding,
mjb_idna_info *info,
mjb_result *result
);
Apply UTS #46 nontransitional processing with STD3 ASCII, hyphen, joiner, bidi, label-length, and domain-length checks enabled. Non-ASCII labels are encoded with Punycode. A successful operational status can still set info->errors; an errored ToASCII result must not be used for DNS lookup. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- Processing completed; inspectinfo->errorsfor validation failuresMJB_STATUS_INVALID_ARGUMENT-text,info, orresultis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent the resultMJB_STATUS_OVERFLOW- An output or Punycode size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_IDNA=0
Examples
// "bücher.example"
mjb_text domain = mjb_text_utf8("b\xC3\xBC" "cher.example", MJB_NUL_TERMINATED);
mjb_idna_info info;
mjb_result result;
if(mjb_idna_to_ascii(&domain, MJB_ENC_UTF_8, &info, &result) != MJB_STATUS_OK ||
info.errors != MJB_IDNA_ERROR_NONE) {
return 1;
}
// xn--bcher-kva.example
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);// "bücher.example"
auto domain = mjb::idna_to_ascii("b\xC3\xBC" "cher.example");
if(!domain.valid()) {
return 1;
}
// xn--bcher-kva.example
std::cout << domain.text << std::endl;Related
Specifications
mjb_idna_to_ascii_into
Convert a domain name to its UTS #46 nontransitional ASCII form into a caller-provided buffer.
mjb_status mjb_idna_to_ascii_into(
const mjb_text *text,
mjb_idna_info *info,
mjb_buffer *output
);
Apply the same strict nontransitional profile as mjb_idna_to_ascii. output->encoding selects the output encoding. Set output->data to NULL to query the required byte count; otherwise output->size supplies its capacity, and on return it contains the required or written byte count. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Processing uses temporary allocations even during a size query. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The required size was returned or the ASCII domain was writtenMJB_STATUS_INVALID_ARGUMENT-text,info, oroutputis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent the resultMJB_STATUS_OVERFLOW- An output or Punycode size would overflowMJB_STATUS_NO_MEMORY- A temporary allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte countMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_IDNA=0
Examples
// "bücher.example"
mjb_text domain = mjb_text_utf8("b\xC3\xBC" "cher.example", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
mjb_idna_info info;
// Query the required size
if(mjb_idna_to_ascii_into(&domain, &info, &output) != MJB_STATUS_OK) {
return 1;
}
char storage[21];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_idna_to_ascii_into(&domain, &info, &output) != MJB_STATUS_OK ||
info.errors != MJB_IDNA_ERROR_NONE) {
return 1;
}
// ASCII payload (no terminator): xn--bcher-kva.example
printf("ASCII payload (no terminator): %.*s", (int)output.size, storage);// "bücher.example"
std::string domain = "b\xC3\xBC" "cher.example";
std::string output;
// Returns the mjb_idna_error flags
if(mjb::idna_to_ascii(domain, output) != MJB_IDNA_ERROR_NONE) {
return 1;
}
// xn--bcher-kva.example
std::cout << output << std::endl;Related
Specifications
mjb_idna_to_unicode
Convert a domain name to its UTS #46 nontransitional Unicode form.
mjb_status mjb_idna_to_unicode(
const mjb_text *text,
mjb_encoding output_encoding,
mjb_idna_info *info,
mjb_result *result
);
Apply UTS #46 nontransitional processing and decode valid xn-- labels. The function returns its best-effort converted string even when info->errors records a validation problem. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- Processing completed; inspectinfo->errorsfor validation failuresMJB_STATUS_INVALID_ARGUMENT-text,info, orresultis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent the resultMJB_STATUS_OVERFLOW- An output or Punycode size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_IDNA=0
Examples
mjb_text domain = mjb_text_utf8("xn--bcher-kva.example", MJB_NUL_TERMINATED);
mjb_idna_info info;
mjb_result result;
if(mjb_idna_to_unicode(&domain, MJB_ENC_UTF_8, &info, &result) != MJB_STATUS_OK ||
info.errors != MJB_IDNA_ERROR_NONE) {
return 1;
}
// bücher.example
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);auto domain = mjb::idna_to_unicode("xn--bcher-kva.example");
if(!domain.valid()) {
return 1;
}
// bücher.example
std::cout << domain.text << std::endl;Related
Specifications
mjb_idna_to_unicode_into
Convert a domain name to its UTS #46 nontransitional Unicode form into a caller-provided buffer.
mjb_status mjb_idna_to_unicode_into(
const mjb_text *text,
mjb_idna_info *info,
mjb_buffer *output
);
Apply the same strict nontransitional profile as mjb_idna_to_unicode. output->encoding selects the output encoding. Set output->data to NULL to query the required byte count; otherwise output->size supplies its capacity, and on return it contains the required or written byte count. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Processing uses temporary allocations even during a size query. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The required size was returned or the Unicode domain was writtenMJB_STATUS_INVALID_ARGUMENT-text,info, oroutputis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent the resultMJB_STATUS_OVERFLOW- An output or Punycode size would overflowMJB_STATUS_NO_MEMORY- A temporary allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte countMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_IDNA=0
Examples
mjb_text domain = mjb_text_utf8("xn--bcher-kva.example", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
mjb_idna_info info;
// Query the required size
if(mjb_idna_to_unicode_into(&domain, &info, &output) != MJB_STATUS_OK) {
return 1;
}
char storage[15];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_idna_to_unicode_into(&domain, &info, &output) != MJB_STATUS_OK ||
info.errors != MJB_IDNA_ERROR_NONE) {
return 1;
}
// Unicode payload (no terminator): bücher.example
printf("Unicode payload (no terminator): %.*s", (int)output.size, storage);std::string domain = "xn--bcher-kva.example";
std::string output;
// Returns the mjb_idna_error flags
if(mjb::idna_to_unicode(domain, output) != MJB_IDNA_ERROR_NONE) {
return 1;
}
// bücher.example
std::cout << output << std::endl;Related
Specifications
mjb_codepoint_encode
Encode a codepoint to a string.
unsigned int mjb_codepoint_encode(
mjb_codepoint codepoint,
char *buffer,
size_t byte_length,
mjb_encoding encoding
);
Example
char encoded[4];
unsigned int size = mjb_codepoint_encode(0x20AC, encoded, sizeof(encoded), MJB_ENC_UTF_8);
// € sign uses 3 UTF-8 bytes
printf("%.*s sign uses %u UTF-8 bytes", (int)size, encoded, size);
mjb_convert_encoding
Convert from one encoding to another.
mjb_status mjb_convert_encoding(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_encoding output_encoding,
mjb_result *result,
mjb_diagnostic *diagnostic
);
Convert a string between the supported encodings (UTF-8, UTF-16LE/BE, UTF-32LE/BE). Generic UTF-16/UTF-32 input consumes a leading BOM as the encoding scheme signature and uses it to resolve byte order. Explicit-endian input preserves an initial U+FEFF as text. Generic UTF-16/UTF-32 without a BOM, and generic UTF-16/UTF-32 output, are rejected because the byte order is not specified. Malformed source subsequences are stopped, replaced, or skipped according to malformed_policy.
Returns
MJB_STATUS_OK- The string was convertedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent an input codepointMJB_STATUS_OVERFLOW- The output size would overflowMJB_STATUS_NO_MEMORY- Allocation failed
Example
mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_result result;
if(mjb_convert_encoding(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_16LE, &result,
NULL) != MJB_STATUS_OK) {
return 1;
}
// UTF-16LE bytes: 8
printf("UTF-16LE bytes: %zu", result.output_size);
mjb_result_free(&result);
Related
mjb_convert_encoding_into
Convert from one encoding to another into a caller-provided buffer.
mjb_status mjb_convert_encoding_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Convert a string using the same encoding and BOM rules as mjb_convert_encoding, without allocating memory. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. The size is the encoded payload byte count: terminators are excluded, and this function does not write a terminator. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity.
Returns
MJB_STATUS_OK- The required size was returned or the string was convertedMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent an input codepointMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte count
Example
mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_16LE);
// Query the required size
if(mjb_convert_encoding_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
unsigned char storage[8];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_16LE);
if(mjb_convert_encoding_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
// UTF-16LE payload bytes (no terminator): 8
printf("UTF-16LE payload bytes (no terminator): %zu", output.size);
Related
mjb_map_case
Change string case.
mjb_status mjb_map_case(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_map_case_type type,
mjb_locale locale,
mjb_encoding output_encoding,
mjb_result *result,
mjb_diagnostic *diagnostic
);
Convert a string to uppercase, lowercase, titlecase, or its case-folded form. Full case mappings are applied, including special casing and conditional mappings, so the output may have a different length than the input. Titlecase uses UAX #29 word boundaries: the first cased character in each word segment is titlecased, and subsequent characters in that segment are lowercased. locale selects the language-sensitive rules of SpecialCasing.txt for this call only; there is no global locale, so concurrent calls with different locales do not interfere. MJB_LOCALE_TR and MJB_LOCALE_AZ apply Turkish/Azerbaijani dotted-I casing and Turkic T case-folding mappings. MJB_LOCALE_LT applies Lithuanian dot-above casing rules, while case folding remains the default non-Turkic mapping. MJB_LOCALE_NOT_SET and every other valid locale use the default mappings. Malformed subsequences follow malformed_policy, and diagnostic records the first one.
Returns
MJB_STATUS_OK- The case conversion succeededMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, the malformed policy is invalid,typeis not a valid case type, orlocaleis not a validmjb_localevalueMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a mapped codepointMJB_STATUS_NO_MEMORY- Allocation failed
Example
mjb_text input = mjb_text_utf8("Stra\xC3\x9F""e", MJB_NUL_TERMINATED); // "Straße"
mjb_result result;
if(mjb_map_case(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, MJB_ENC_UTF_8,
&result, NULL) != MJB_STATUS_OK) {
return 1;
}
// Upper: STRASSE
printf("Upper: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);
mjb_text istanbul = mjb_text_utf8("istanbul", MJB_NUL_TERMINATED);
if(mjb_map_case(&istanbul, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_TR, MJB_ENC_UTF_8,
&result, NULL) != MJB_STATUS_OK) {
return 1;
}
// Turkish upper: \xC4\xB0STANBUL
printf("Turkish upper: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);
Related
Specifications
mjb_map_case_into
Change string case into a caller-provided buffer.
mjb_status mjb_map_case_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_map_case_type type,
mjb_locale locale,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Apply the same full, special, conditional, titlecase, locale-sensitive, and case folding mappings as mjb_map_case without allocating memory. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity.
Returns
MJB_STATUS_OK- The required size was returned or the case-mapped string was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, the malformed policy is invalid,typeis invalid, orlocaleis not a validmjb_localevalueMJB_STATUS_INVALID_ENCODING- An encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_UNSUPPORTED- The requested output encoding cannot represent a mapped codepointMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte count
Example
mjb_text input = mjb_text_utf8("Stra\xC3\x9F""e", MJB_NUL_TERMINATED); // "Straße"
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
// Query the required size
if(mjb_map_case_into(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
char storage[7];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_map_case_into(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, &output,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Upper payload (no terminator): STRASSE
printf("Upper payload (no terminator): %.*s", (int)output.size, storage);
Related
Specifications
API section
Text analysis
Inspect codepoints, properties, boundaries, width, emoji, and bidirectional text.
mjb_codepoint_info
Return the codepoint character.
mjb_status mjb_codepoint_info(
mjb_codepoint codepoint,
mjb_character *character
);
Fill character with the Unicode Character Database record of a codepoint: name, category, combining class, bidirectional category, decomposition, numeric values, mirrored flag, and simple case mappings. When the library is compiled with MJB_FEATURE_CHARACTER_NAMES=0 the name field is reported as Codepoint U+XXXX.
Returns
MJB_STATUS_OK- The character was found and filledMJB_STATUS_INVALID_ARGUMENT-characteris NULL or the codepoint is not validMJB_STATUS_NOT_FOUND- The codepoint is not assigned
Examples
mjb_character character;
// U+0665 ARABIC-INDIC DIGIT FIVE
if(mjb_codepoint_info(0x0665, &character) != MJB_STATUS_OK) {
return 1;
}
bool is_digit = character.category == MJB_CATEGORY_ND;
// Decimal digit: yes, value: 5
printf("Decimal digit: %s, value: %d", is_digit ? "yes" : "no", character.decimal);// U+0665 ARABIC-INDIC DIGIT FIVE
auto character = mjb::Character::from(0x0665);
if(!character) {
return 1;
}
bool is_digit = character->category() == MJB_CATEGORY_ND;
// Decimal digit: yes, value: 5
std::cout << "Decimal digit: " << (is_digit ? "yes" : "no") << ", value: "
<< character->decimal_value().value_or(-1) << std::endl;Related
Specifications
mjb_normalization_quick_check
Check if a string is normalized to NFC/NFKC/NFD/NFKD form.
mjb_status mjb_normalization_quick_check(
const mjb_text *text,
mjb_normalization form,
mjb_quick_check_result *quick_check
);
Run the normalization quick-check on a string without allocating. MJB_QC_MAYBE means the string may still be normalized, and only a full normalization pass with mjb_normalize can decide.
Returns
MJB_STATUS_OK-quick_checkcontainsMJB_QC_YES,MJB_QC_NO, orMJB_QC_MAYBEMJB_STATUS_INVALID_ARGUMENT-quick_checkortextis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- The text encoding is invalid or a generic UTF-16/UTF-32 encoding has no byte-order informationMJB_STATUS_INVALID_FORM-formis not NFC, NFD, NFKC, or NFKDMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequence
Example
mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_quick_check_result check;
if(mjb_normalization_quick_check(&input, MJB_NORMALIZATION_NFC, &check) != MJB_STATUS_OK) {
return 1;
}
// NFC normalized: yes
printf("NFC normalized: %s", check == MJB_QC_YES ? "yes" : "no");
Related
Specifications
mjb_detect_encoding
Return the string encoding (the most probable).
mjb_encoding mjb_detect_encoding(
const char *buffer,
size_t byte_length
);
mjb_detect_encoding reports BOM-derived UTF-16/UTF-32 schemes with the generic family bit plus the resolved endian bit. Passing that detected value consumes the leading BOM as a signature. Passing an explicit-endian encoding such as MJB_ENC_UTF_16BE preserves an initial U+FEFF as text. When flags overlap, as with a UTF-32LE BOM that also has the UTF-16LE BOM prefix, decoding gives UTF-32 precedence. MJB_NUL_TERMINATED is not accepted because the encoding, and therefore the NUL code-unit width, is unknown.
Example
const char utf16le[] = "\xFF\xFEH\0i\0";
mjb_encoding detected = mjb_detect_encoding(utf16le, sizeof(utf16le) - 1);
bool is_utf16le = detected == (MJB_ENC_UTF_16 | MJB_ENC_UTF_16LE);
// UTF-16LE detected: yes
printf("UTF-16LE detected: %s", is_utf16le ? "yes" : "no");
mjb_is_ascii
Return true if the string is encoded in ASCII.
bool mjb_is_ascii(
const char *buffer,
size_t byte_length
);
Examples
const char *input = "Plain ASCII";
// ASCII: yes
printf("ASCII: %s", mjb_is_ascii(input, strlen(input)) ? "yes" : "no");std::string input = "Plain ASCII";
// ASCII: yes
std::cout << "ASCII: " << (mjb::is_ascii(input) ? "yes" : "no") << std::endl;mjb_is_utf8
Return true if the string is encoded in UTF-8.
bool mjb_is_utf8(
const char *buffer,
size_t byte_length
);
Examples
const char *input = "caf\xC3\xA9";
// Valid UTF-8: yes
printf("Valid UTF-8: %s", mjb_is_utf8(input, strlen(input)) ? "yes" : "no");std::string input = "caf\xC3\xA9";
// Valid UTF-8: yes
std::cout << "Valid UTF-8: " << (mjb::is_utf8(input) ? "yes" : "no") << std::endl;mjb_is_utf16
Return true if the string is encoded in UTF-16BE or UTF-16LE.
bool mjb_is_utf16(
const char *buffer,
size_t byte_length
);
Examples
const char utf16be[] = "\xFE\xFF\0H\0i"; // BOM + "Hi" in UTF-16BE
// UTF-16: yes
printf("UTF-16: %s", mjb_is_utf16(utf16be, sizeof(utf16be) - 1) ? "yes" : "no");using namespace std::string_view_literals;
// BOM + "Hi" in UTF-16BE. The sv suffix keeps the embedded NUL bytes.
auto utf16be = "\xFE\xFF\0H\0i"sv;
// UTF-16: yes
std::cout << "UTF-16: " << (mjb::is_utf16(utf16be) ? "yes" : "no") << std::endl;mjb_validate_encoding
Validate a complete Unicode code-unit sequence.
mjb_status mjb_validate_encoding(
const mjb_text *text,
mjb_diagnostic *diagnostic
);
Validate the complete input without producing output. Empty input is well-formed. On malformed input, diagnostic identifies the first maximal ill-formed subsequence. Generic UTF-16 and UTF-32 require a byte-order mark.
Returns
MJB_STATUS_OK- The complete input is well-formedMJB_STATUS_INVALID_ARGUMENT-textis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- The encoding is unsupported, or generic UTF-16/UTF-32 has no byte-order markMJB_STATUS_MALFORMED_INPUT- The first malformed subsequence is described bydiagnostic
Examples
const char invalid[] = "\xE2\x82";
mjb_text text = mjb_text_utf8(invalid, sizeof(invalid) - 1);
mjb_diagnostic diagnostic;
if(mjb_validate_encoding(&text, &diagnostic) != MJB_STATUS_MALFORMED_INPUT ||
diagnostic.byte_offset != 0) {
return 1;
}std::string invalid = "\xE2\x82"; // A truncated three-byte sequence
mjb_diagnostic diagnostic{};
if(mjb::validate_encoding(invalid, &diagnostic) != MJB_STATUS_MALFORMED_INPUT) {
return 1;
}
// Malformed input at byte 0
std::cout << "Malformed input at byte " << diagnostic.byte_offset << std::endl;Related
mjb_nul_terminated_length
Return the byte length of a string that ends at its first NUL code unit.
mjb_status mjb_nul_terminated_length(
const char *buffer,
mjb_encoding encoding,
size_t *byte_length
);
Scan for the terminator of the encoding: a zero byte for ASCII and UTF-8, a zero 16-bit code unit for UTF-16, and a zero 32-bit code unit for UTF-32. Code units are aligned to the start of buffer.
Returns
MJB_STATUS_OK- The length was computedMJB_STATUS_INVALID_ARGUMENT-bufferorbyte_lengthis NULLMJB_STATUS_INVALID_ENCODING- The encoding is not a supported input encoding
Example
const char utf16le[] = "H\0i\0\0\0"; // "Hi" in UTF-16LE
size_t byte_length;
if(mjb_nul_terminated_length(utf16le, MJB_ENC_UTF_16LE, &byte_length) != MJB_STATUS_OK) {
return 1;
}
// Byte length: 4
printf("Byte length: %zu", byte_length);
Related
mjb_decode_next
Decode the next codepoint from a string.
mjb_status mjb_decode_next(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *offset,
mjb_codepoint *codepoint,
mjb_diagnostic *diagnostic
);
Decode one codepoint and advance offset. With MJB_MALFORMED_STOP, malformed input returns MJB_STATUS_MALFORMED_INPUT; offset still advances over the malformed subsequence so decoding can resume. MJB_MALFORMED_REPLACE returns U+FFFD, while MJB_MALFORMED_SKIP advances until a valid codepoint or end of input. A diagnostic is reported for replacement and skipping even when the function returns success.
Returns
MJB_STATUS_OK- A codepoint was decodedMJB_STATUS_END_OF_INPUT- No codepoint remainsMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_INVALID_ARGUMENT- An argument or policy is invalid, or the text isMJB_NUL_TERMINATEDMJB_STATUS_INVALID_ENCODING- The encoding cannot be decoded
Example
// "été"
const char *word = "\xC3\xA9t\xC3\xA9";
mjb_text input = mjb_text_utf8(word, strlen(word));
size_t offset = 0;
size_t count = 0;
mjb_codepoint codepoint;
mjb_status status;
while((status = mjb_decode_next(&input, MJB_MALFORMED_STOP, &offset, &codepoint, NULL)) ==
MJB_STATUS_OK) {
++count;
}
if(status != MJB_STATUS_END_OF_INPUT) {
return 1;
}
// 3 codepoints in 5 bytes
printf("%zu codepoints in %zu bytes", count, offset);
Related
mjb_decode_previous
Decode the previous codepoint from a string.
mjb_status mjb_decode_previous(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *offset,
mjb_codepoint *codepoint,
mjb_diagnostic *diagnostic
);
Decode backward from offset, using the same malformed-input policies and diagnostic contract as mjb_decode_next. On success, offset is the first byte of the decoded codepoint. Start with the input byte length to iterate from the end.
Returns
MJB_STATUS_OK- A codepoint was decodedMJB_STATUS_END_OF_INPUT- No codepoint precedesoffsetMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_INVALID_ARGUMENT- An argument or policy is invalid, or the text isMJB_NUL_TERMINATEDMJB_STATUS_INVALID_ENCODING- The encoding cannot be decoded
Example
// "été"
const char *word = "\xC3\xA9t\xC3\xA9";
mjb_text input = mjb_text_utf8(word, strlen(word));
size_t offset = strlen(word); // Start at the end
mjb_codepoint codepoint;
if(mjb_decode_previous(&input, MJB_MALFORMED_STOP, &offset, &codepoint,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Last codepoint: U+00E9 at byte 3
printf("Last codepoint: U+%04X at byte %zu", (unsigned)codepoint, offset);
Related
mjb_codepoint_count
Count the codepoints in a string.
mjb_status mjb_codepoint_count(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *count,
mjb_diagnostic *diagnostic
);
Count the number of decoded Unicode codepoints in a string. Malformed subsequences are stopped, replaced, or skipped according to malformed_policy. A replacement counts as one codepoint. On failure, count is set to zero.
Returns
MJB_STATUS_OK- The count was computedMJB_STATUS_INVALID_ARGUMENT-countortextis NULL,text->bufferis NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- The text encoding is not a supported input encodingMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOP
Example
// The "Héllö" string is five Unicode characters, but has different byte lengths in different encodings.
mjb_text utf8 = mjb_text_utf8("H\xC3\xA9ll\xC3\xB6", 7); // 7 bytes
mjb_text utf16le = mjb_text_utf16le("H\0\xE9\0l\0l\0\xF6\0", 10); // 10 bytes
size_t count;
if(mjb_codepoint_count(&utf8, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
return 1;
}
// 5 UTF-8 characters
printf("%zu UTF-8 characters", count);
if(mjb_codepoint_count(&utf16le, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
return 1;
}
// 5 UTF-16LE characters
printf("%zu UTF-16LE characters", count);
Related
mjb_for_each_codepoint
Run a callback for each codepoint of a string.
mjb_status mjb_for_each_codepoint(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_for_each_codepoint_fn callback,
void *context,
mjb_diagnostic *diagnostic
);
Decode the string according to malformed_policy and call the callback for every resulting codepoint, with its position in the string and context. Use context to give the callback its own state, such as a counter or an output buffer, without global variables. The first malformed subsequence is reported in diagnostic even when it is replaced or skipped.
Returns
MJB_STATUS_OK- Every decoded codepoint was visitedMJB_STATUS_INVALID_ARGUMENT- The text, callback, or malformed policy is invalidMJB_STATUS_INVALID_ENCODING- The encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_CALLBACK_STOPPED- The callback returned false
Example
mjb_text input = mjb_text_utf8("ABC", 3);
mjb_status status = mjb_for_each_codepoint(&input, MJB_MALFORMED_STOP, NULL, NULL, NULL);
// A callback is required: yes
bool callback_required = status == MJB_STATUS_INVALID_ARGUMENT;
// A callback is required: yes
printf("A callback is required: %s", callback_required ? "yes" : "no");
mjb_codepoint_property_binary
Return the value of a binary Unicode property.
mjb_status mjb_codepoint_property_binary(
mjb_codepoint codepoint,
mjb_property property,
bool *value
);
Return true when the codepoint has the binary property and false when it does not. Passing an enumerated property is a type mismatch and returns MJB_STATUS_INVALID_ARGUMENT.
Example
bool is_alphabetic;
if(mjb_codepoint_property_binary('A', MJB_PR_ALPHABETIC,
&is_alphabetic) != MJB_STATUS_OK) {
return 1;
}
// U+0041 is alphabetic: yes
printf("U+0041 is alphabetic: %s", is_alphabetic ? "yes" : "no");
Related
Specifications
mjb_codepoint_property_int
Return the value of an enumerated or integer Unicode property.
mjb_status mjb_codepoint_property_int(
mjb_codepoint codepoint,
mjb_property property,
int32_t *value
);
Passing a binary property is a type mismatch and returns MJB_STATUS_INVALID_ARGUMENT. MJB_STATUS_NOT_FOUND means that the codepoint has no stored value for the requested property.
Example
int32_t script;
if(mjb_codepoint_property_int('A', MJB_PR_SCRIPT, &script) != MJB_STATUS_OK) {
return 1;
}
// U+0041 uses the Latin script: yes
printf("U+0041 uses the Latin script: %s", script == MJB_SC_LATN ? "yes" : "no");
Related
Specifications
mjb_codepoint_numeric_value
Return the numeric value of a codepoint.
mjb_status mjb_codepoint_numeric_value(
mjb_codepoint codepoint,
mjb_numeric_value *value
);
Return the numeric value of a codepoint, if any. If the codepoint has no numeric value, value->decimal and value->digit are set to MJB_NUMBER_NOT_VALID (-1).
Returns
MJB_STATUS_OK- The character was found and filledMJB_STATUS_INVALID_ARGUMENT-valueis NULL or the codepoint is not valid
Example
mjb_numeric_value num;
if(mjb_codepoint_numeric_value(0x0031, &num) != MJB_STATUS_OK) { // U+0031 = 1
return 1;
}
// decimal=1, digit=1, numeric=1
printf("decimal=%d, digit=%d, numeric=%s", num.decimal, num.digit, num.numeric);
if(mjb_codepoint_numeric_value(0x00BD, &num) != MJB_STATUS_OK) { // U+00BD = '½'
return 1;
}
// decimal=-1, digit=-1, numeric=1/2
printf("decimal=%d, digit=%d, numeric=%s", num.decimal, num.digit, num.numeric);
Specifications
mjb_codepoint_block
Return the character block.
mjb_status mjb_codepoint_block(
mjb_codepoint codepoint,
mjb_block_info *block
);
Examples
mjb_block_info block;
if(mjb_codepoint_block('A', &block) != MJB_STATUS_OK) {
return 1;
}
// Block: Basic Latin
printf("Block: %s", block.name);auto block = mjb::block_info('A');
if(!block) {
return 1;
}
// Block: Basic Latin
std::cout << "Block: " << block->name() << std::endl;Specifications
mjb_codepoint_script
Return the script of a codepoint.
mjb_script mjb_codepoint_script(
mjb_codepoint codepoint
);
Examples
mjb_script script = mjb_codepoint_script(0x03A9); // Greek capital omega
// Greek script: yes
printf("Greek script: %s", script == MJB_SC_GREK ? "yes" : "no");mjb_script script = mjb::script(0x03A9); // Greek capital omega
// Greek script: yes
std::cout << "Greek script: " << (script == MJB_SC_GREK ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_script_extensions
Return the Script_Extensions set of a codepoint.
mjb_status mjb_codepoint_script_extensions(
mjb_codepoint codepoint,
mjb_script *scripts,
size_t *count
);
Return the explicit Script_Extensions set, or the ordinary Script value when the codepoint has no explicit Script_Extensions entry. Call first with scripts set to NULL to obtain the required count.
Examples
size_t count = 0;
if(mjb_codepoint_script_extensions(0x30FC, NULL, &count) != MJB_STATUS_OK) {
return 1;
}
mjb_script scripts[3];
if(count > 3 || mjb_codepoint_script_extensions(0x30FC, scripts,
&count) != MJB_STATUS_OK) {
return 1;
}
// U+30FC has 2 Script_Extensions
printf("U+30FC has %zu Script_Extensions", count);// Throws mjb::LibraryError on failure
std::vector<mjb_script> scripts = mjb::script_extensions(0x30FC);
// U+30FC has 2 Script_Extensions
std::cout << "U+30FC has " << scripts.size() << " Script_Extensions" << std::endl;Related
Specifications
mjb_codepoint_is_valid
Return true if the codepoint is valid.
bool mjb_codepoint_is_valid(
mjb_codepoint codepoint
);
Examples
// U+10FFFD valid: yes
printf("U+10FFFD valid: %s", mjb_codepoint_is_valid(0x10FFFD) ? "yes" : "no");// U+10FFFD valid: yes
std::cout << "U+10FFFD valid: " << (mjb::is_valid(0x10FFFD) ? "yes" : "no") << std::endl;mjb_codepoint_is_graphic
Return true if the codepoint is graphic.
bool mjb_codepoint_is_graphic(
mjb_codepoint codepoint
);
Examples
// Letter A is graphic: yes
printf("Letter A is graphic: %s", mjb_codepoint_is_graphic('A') ? "yes" : "no");// Letter A is graphic: yes
std::cout << "Letter A is graphic: " << (mjb::is_graphic('A') ? "yes" : "no") << std::endl;mjb_codepoint_is_combining
Return true if the codepoint is combining.
bool mjb_codepoint_is_combining(
mjb_codepoint codepoint
);
Examples
// U+0301 is combining: yes
printf("U+0301 is combining: %s", mjb_codepoint_is_combining(0x0301) ? "yes" : "no");// U+0301 is combining: yes
std::cout << "U+0301 is combining: " << (mjb::is_combining(0x0301) ? "yes" : "no") << std::endl;mjb_codepoint_is_cjk_ideograph
Return if the codepoint is CJK ideograph.
bool mjb_codepoint_is_cjk_ideograph(
mjb_codepoint codepoint
);
Examples
// U+4E00 is a CJK ideograph: yes
printf("U+4E00 is a CJK ideograph: %s", mjb_codepoint_is_cjk_ideograph(0x4E00) ? "yes" : "no");// U+4E00 is a CJK ideograph: yes
std::cout << "U+4E00 is a CJK ideograph: "
<< (mjb::is_cjk_ideograph(0x4E00) ? "yes" : "no") << std::endl;mjb_codepoint_is_cjk_extension_ideograph
Return if the codepoint is CJK extension.
bool mjb_codepoint_is_cjk_extension_ideograph(
mjb_codepoint codepoint
);
Examples
// U+20000 is a CJK extension ideograph: yes
printf("U+20000 is a CJK extension ideograph: %s", mjb_codepoint_is_cjk_extension_ideograph(0x20000) ? "yes" : "no");// U+20000 is a CJK extension ideograph: yes
std::cout << "U+20000 is a CJK extension ideograph: "
<< (mjb::is_cjk_extension_ideograph(0x20000) ? "yes" : "no") << std::endl;mjb_category_is_graphic
Return true if the category is graphic.
bool mjb_category_is_graphic(
mjb_category category
);
Example
// Uppercase letters are graphic: yes
bool graphic = mjb_category_is_graphic(MJB_CATEGORY_LU);
// Uppercase letters are graphic: yes
printf("Uppercase letters are graphic: %s", graphic ? "yes" : "no");
mjb_category_is_combining
Return true if the category is combining.
bool mjb_category_is_combining(
mjb_category category
);
Example
// Nonspacing marks are combining: yes
bool combining = mjb_category_is_combining(MJB_CATEGORY_MN);
// Nonspacing marks are combining: yes
printf("Nonspacing marks are combining: %s", combining ? "yes" : "no");
mjb_codepoint_is_id_start
Return true if the codepoint is a valid Unicode identifier start (Unicode 18.0.0 UAX #31 ID_Start).
bool mjb_codepoint_is_id_start(
mjb_codepoint codepoint
);
Examples
// Greek alpha starts an identifier: yes
bool starts = mjb_codepoint_is_id_start(0x03B1);
// Greek alpha starts an identifier: yes
printf("Greek alpha starts an identifier: %s", starts ? "yes" : "no");// Greek alpha starts an identifier: yes
std::cout << "Greek alpha starts an identifier: "
<< (mjb::is_id_start(0x03B1) ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_id_continue
Return true if the codepoint is a valid Unicode identifier continuation (Unicode 18.0.0 UAX #31 ID_Continue).
bool mjb_codepoint_is_id_continue(
mjb_codepoint codepoint
);
Examples
// Digit 7 continues an identifier: yes
bool continues = mjb_codepoint_is_id_continue('7');
// Digit 7 continues an identifier: yes
printf("Digit 7 continues an identifier: %s", continues ? "yes" : "no");// Digit 7 continues an identifier: yes
std::cout << "Digit 7 continues an identifier: "
<< (mjb::is_id_continue('7') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_xid_start
Return true if the codepoint is a valid NFKC identifier start (Unicode 18.0.0 UAX #31 XID_Start).
bool mjb_codepoint_is_xid_start(
mjb_codepoint codepoint
);
Examples
// Letter A is XID_Start: yes
printf("Letter A is XID_Start: %s", mjb_codepoint_is_xid_start('A') ? "yes" : "no");// Letter A is XID_Start: yes
std::cout << "Letter A is XID_Start: " << (mjb::is_xid_start('A') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_xid_continue
Return true if the codepoint is a valid NFKC identifier continuation (Unicode 18.0.0 UAX #31 XID_Continue).
bool mjb_codepoint_is_xid_continue(
mjb_codepoint codepoint
);
Examples
// Underscore is XID_Continue: yes
bool continues = mjb_codepoint_is_xid_continue('_');
// Underscore is XID_Continue: yes
printf("Underscore is XID_Continue: %s", continues ? "yes" : "no");// Underscore is XID_Continue: yes
std::cout << "Underscore is XID_Continue: "
<< (mjb::is_xid_continue('_') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_pattern_syntax
Return true if the codepoint is reserved for use in patterns (Unicode 18.0.0 UAX #31 Pattern_Syntax).
bool mjb_codepoint_is_pattern_syntax(
mjb_codepoint codepoint
);
Examples
// Plus sign is Pattern_Syntax: yes
bool syntax = mjb_codepoint_is_pattern_syntax('+');
// Plus sign is Pattern_Syntax: yes
printf("Plus sign is Pattern_Syntax: %s", syntax ? "yes" : "no");// Plus sign is Pattern_Syntax: yes
std::cout << "Plus sign is Pattern_Syntax: "
<< (mjb::is_pattern_syntax('+') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_pattern_white_space
Return true if the codepoint is pattern whitespace (Unicode 18.0.0 UAX #31 Pattern_White_Space).
bool mjb_codepoint_is_pattern_white_space(
mjb_codepoint codepoint
);
Examples
// Space is Pattern_White_Space: yes
bool whitespace = mjb_codepoint_is_pattern_white_space(' ');
// Space is Pattern_White_Space: yes
printf("Space is Pattern_White_Space: %s", whitespace ? "yes" : "no");// Space is Pattern_White_Space: yes
std::cout << "Space is Pattern_White_Space: "
<< (mjb::is_pattern_white_space(' ') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_extended_pictographic
Return true if the codepoint has the Unicode Extended_Pictographic property.
bool mjb_codepoint_is_extended_pictographic(
mjb_codepoint codepoint
);
Examples
// Red heart is Extended_Pictographic: yes
bool pictographic = mjb_codepoint_is_extended_pictographic(0x2764);
// Red heart is Extended_Pictographic: yes
printf("Red heart is Extended_Pictographic: %s", pictographic ? "yes" : "no");// Red heart is Extended_Pictographic: yes
std::cout << "Red heart is Extended_Pictographic: "
<< (mjb::is_extended_pictographic(0x2764) ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_plane
Return the plane of the codepoint.
mjb_plane mjb_codepoint_plane(
mjb_codepoint codepoint
);
Example
mjb_plane plane = mjb_codepoint_plane(0x1F600);
// U+1F600 is in the SMP: yes
printf("U+1F600 is in the SMP: %s", plane == MJB_PLANE_SMP ? "yes" : "no");
mjb_plane_is_valid
Return true if the plane is valid.
bool mjb_plane_is_valid(
mjb_plane plane
);
Example
// Plane 16 is valid: yes
printf("Plane 16 is valid: %s", mjb_plane_is_valid(MJB_PLANE_PUA_B) ? "yes" : "no");
mjb_plane_name
Return the name of a plane, NULL if the plane specified is not valid.
const char *mjb_plane_name(
mjb_plane plane,
bool abbreviation
);
Example
// Plane: Basic Multilingual Plane
printf("Plane: %s", mjb_plane_name(MJB_PLANE_BMP, false));
mjb_script_code
Return the ISO 15924 code of a script, NULL if the script is not valid.
const char *mjb_script_code(
mjb_script script
);
Return the four-letter ISO 15924 code that names script, such as Latn for MJB_SC_LATN. MJB_SC_NOT_SET yields an empty string.
Returns
const char *- The ISO 15924 code, or NULL ifscriptis not a validmjb_scriptvalue
Example
// Script code: Latn
printf("Script code: %s", mjb_script_code(MJB_SC_LATN));
Related
API section
Sorting and comparison
Compare and sort strings with the Unicode Collation Algorithm.
mjb_caseless_match
Compare two strings using a Unicode caseless matching relation.
mjb_status mjb_caseless_match(
const mjb_text *first,
const mjb_text *second,
mjb_malformed_policy malformed_policy,
mjb_caseless_mode mode,
bool *matches,
mjb_diagnostic *first_diagnostic,
mjb_diagnostic *second_diagnostic
);
Compare two strings for case-insensitive equality using Unicode full default case folding. Canonical mode implements D145 and is recommended for ordinary text. Unnormalized mode implements D144 without normalization. Compatibility mode implements the iterated folding and compatibility normalization required by D146. Identifier mode implements D147 by applying NFD before mjb_nfkc_casefold; it removes default-ignorable codepoints but does not validate identifier syntax. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not examined and second_diagnostic stays clear.
Returns
MJB_STATUS_OK-matchescontains the comparison resultMJB_STATUS_INVALID_ARGUMENT-matches,first, orsecondis NULL, a text buffer is NULL with a non-zero size,modeis invalid, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An input encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- An intermediate size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_UNSUPPORTED- The identifier case-folding transform did not converge
Example
mjb_text left = mjb_text_utf8("Stra\xC3\x9F" "e", MJB_NUL_TERMINATED);
mjb_text right = mjb_text_utf8("STRASSE", MJB_NUL_TERMINATED);
bool matches;
if(mjb_caseless_match(&left, &right, MJB_MALFORMED_STOP, MJB_CASELESS_CANONICAL, &matches, NULL,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Canonical caseless match: yes
printf("Canonical caseless match: %s", matches ? "yes" : "no");
Related
Specifications
mjb_collation_compare
Compare two strings using UCA.
mjb_status mjb_collation_compare(
const mjb_text *first,
const mjb_text *second,
mjb_malformed_policy malformed_policy,
mjb_collation_variable_weighting variable_weighting,
mjb_collation_strength strength,
int *order,
mjb_diagnostic *first_diagnostic,
mjb_diagnostic *second_diagnostic
);
Compare two strings using the Unicode Collation Algorithm and the default collation element table (DUCET), with strcmp-style semantics. Primary strength compares base-character weights. Secondary strength also compares accents while ignoring tertiary case differences. Tertiary strength adds case and variant differences. Quaternary strength also compares variable elements moved to level 4 by MJB_COLLATION_SHIFTED; with MJB_COLLATION_NON_IGNORABLE, it is equivalent to tertiary strength. Collation equality at a selected strength is not Unicode caseless matching; use mjb_caseless_match when case-insensitive equality is the intended operation. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not read and second_diagnostic stays clear. See [Unicode collation](#unicode-collation) for the comparison process and configuration guidance. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK-orderis negative, zero, or positive according to the collation orderMJB_STATUS_INVALID_ARGUMENT-order,first, orsecondis NULL, a text buffer is NULL with a non-zero size, the malformed policy is invalid, or an option is invalidMJB_STATUS_INVALID_ENCODING- An input encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- An intermediate size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_COLLATION=0
Example
mjb_text apple = mjb_text_utf8("apple", 5);
mjb_text banana = mjb_text_utf8("banana", 6);
int order;
if(mjb_collation_compare(&apple, &banana, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
MJB_COLLATION_TERTIARY, &order, NULL, NULL) != MJB_STATUS_OK) {
return 1;
}
// apple sorts before banana: yes
printf("apple sorts before banana: %s", order < 0 ? "yes" : "no");
Related
Specifications
mjb_collation_key
Generate a UCA sort key for a string.
mjb_status mjb_collation_key(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_collation_variable_weighting variable_weighting,
mjb_collation_strength strength,
mjb_result *result,
mjb_diagnostic *diagnostic
);
Generate a binary sort key for a string. Sort keys of different strings can be compared with memcmp and yield the same order as mjb_collation_compare when both use the same variable weighting and strength. Useful when the same strings are compared many times, such as sorting or database indexing. Empty input and non-empty input with no effective weights at the selected strength both produce a zero-length key. Malformed subsequences follow malformed_policy, and diagnostic records the first one. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The sort key was generatedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL,text->bufferis NULL with a non-zero size, the malformed policy is invalid, or a collation option is invalidMJB_STATUS_INVALID_ENCODING- The input encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- The sort key size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_COLLATION=0
Example
mjb_text input = mjb_text_utf8("r\xC3\xA9sum\xC3\xA9", 8);
mjb_result key;
if(mjb_collation_key(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
MJB_COLLATION_TERTIARY, &key, NULL) != MJB_STATUS_OK) {
return 1;
}
// Sort key is non-empty: yes
printf("Sort key is non-empty: %s", key.output_size > 0 ? "yes" : "no");
mjb_result_free(&key);
Related
Specifications
mjb_collation_key_into
Generate a binary collation key into a caller-provided buffer.
mjb_status mjb_collation_key_into(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_collation_variable_weighting variable_weighting,
mjb_collation_strength strength,
mjb_buffer *output,
mjb_diagnostic *diagnostic
);
Generate the same binary sort key as mjb_collation_key without allocating the final key buffer. Set output->data to NULL to query the required byte count. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. A collation key is binary: output->encoding is ignored, and no terminator is included or written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the key, but nothing is written past its capacity. Collation processing still uses temporary allocations, including during a size query. Empty input and non-empty input with no effective weights at the selected strength both require zero bytes. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The required size was returned or the binary sort key was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL,text->bufferis NULL with a non-zero size, the malformed policy is invalid, or a collation option is invalidMJB_STATUS_INVALID_ENCODING- The input encoding is invalidMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- The required key size would overflowMJB_STATUS_NO_MEMORY- Temporary allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte countMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_COLLATION=0
Example
mjb_text input = mjb_text_utf8("r\xC3\xA9sum\xC3\xA9", 8);
// A collation key is binary, so the buffer encoding is ignored
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UNKNOWN);
// Query the required size
if(mjb_collation_key_into(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
MJB_COLLATION_TERTIARY, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
unsigned char storage[64];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UNKNOWN);
if(mjb_collation_key_into(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
MJB_COLLATION_TERTIARY, &output, NULL) != MJB_STATUS_OK) {
return 1;
}
// Sort key is non-empty: yes
printf("Sort key is non-empty: %s", output.size > 0 ? "yes" : "no");
Related
Specifications
API section
Formatting
Format UTF-8 text with codepoint or grapheme-safe truncation.
mjb_utf8_snprintf
Format a UTF-8 string without leaving an incomplete trailing codepoint.
int mjb_utf8_snprintf(
char *buffer,
size_t buffer_size,
const char *format,
...
);
Use the C library formatting rules and return semantics of snprintf. If the destination buffer truncates a well-formed UTF-8 result, any incomplete trailing codepoint is removed before the terminating NULL. Truncation is at a codepoint boundary, not a grapheme-cluster boundary.
Returns
A nonnegative value- The number of bytes the complete result requires, excluding the terminating NULLA negative value- The underlyingvsnprintfreported an encoding error
Example
char buffer[4];
int required = mjb_utf8_snprintf(buffer, sizeof(buffer), "%s",
"\xC3\xA9\xC3\xA9"); // éé
// 4: é
printf("%d: %s", required, buffer);
Related
mjb_utf8_vsnprintf
Format a UTF-8 string from a va_list without leaving an incomplete trailing codepoint.
int mjb_utf8_vsnprintf(
char *buffer,
size_t buffer_size,
const char *format,
va_list args
);
The va_list counterpart of mjb_utf8_snprintf. It has the same UTF-8 input requirements, clipping behavior, and return semantics as mjb_utf8_snprintf.
Returns
A nonnegative value- The number of bytes the complete result requires, excluding the terminating NULLA negative value- The underlyingvsnprintfreported an encoding error
Example
// A printf-style function of your own that never ends with a partial codepoint
static int format_utf8(char *buffer, size_t buffer_size, const char *format, ...) {
va_list args;
va_start(args, format);
int required = mjb_utf8_vsnprintf(buffer, buffer_size, format, args);
va_end(args);
return required;
}
char buffer[4];
int required = format_utf8(buffer, sizeof(buffer), "%s", "\xC3\xA9\xC3\xA9"); // éé
// 4: é
printf("%d: %s", required, buffer);
Related
mjb_utf8_grapheme_snprintf
Format UTF-8 without truncating an extended grapheme cluster.
int mjb_utf8_grapheme_snprintf(
char *buffer,
size_t buffer_size,
const char *format,
...
);
Use the C library formatting rules and return semantics of snprintf. If the destination buffer truncates a well-formed UTF-8 result, the output is shortened to the largest prefix that ends at an extended grapheme-cluster boundary in the complete result. Unlike mjb_utf8_snprintf, truncation can require temporary allocation and a second evaluation of the format. Do not use %n or arguments whose values can change as a formatting side effect. On allocation failure the destination is set to an empty string, the function returns a negative value, and errno is set to ENOMEM.
Returns
A nonnegative value- The number of bytes the complete result requires, excluding the terminating NULLA negative value- Formatting failed or the complete result could not be obtained
Example
char buffer[4];
int required = mjb_utf8_grapheme_snprintf(buffer, sizeof(buffer), "%s",
"Ae\xCC\x81" "B"); // A, e + combining acute accent, B
// 5: A
printf("%d: %s", required, buffer);
Related
Specifications
mjb_utf8_grapheme_vsnprintf
Format UTF-8 from a va_list without truncating an extended grapheme cluster.
int mjb_utf8_grapheme_vsnprintf(
char *buffer,
size_t buffer_size,
const char *format,
va_list args
);
The va_list counterpart of mjb_utf8_grapheme_snprintf. It has the same grapheme-safe clipping behavior, allocation requirements, and return semantics as mjb_utf8_grapheme_snprintf.
Returns
A nonnegative value- The number of bytes the complete result requires, excluding the terminating NULLA negative value- Formatting failed or the complete result could not be obtained
Example
// A printf-style function of your own that never splits a grapheme cluster
static int format_graphemes(char *buffer, size_t buffer_size, const char *format, ...) {
va_list args;
va_start(args, format);
int required = mjb_utf8_grapheme_vsnprintf(buffer, buffer_size, format, args);
va_end(args);
return required;
}
char buffer[4];
int required = format_graphemes(buffer, sizeof(buffer), "%s",
"Ae\xCC\x81" "B"); // A, e + combining acute accent, B
// 5: A
printf("%d: %s", required, buffer);
Related
Specifications
API section
Security
Detect visually confusable Unicode text.
mjb_is_identifier
Return true if the string is a valid Unicode identifier (Unicode 18.0.0 UAX #31).
bool mjb_is_identifier(
const mjb_text *text,
mjb_identifier_profile profile
);
Validate a string as a Unicode identifier: the first character must be a valid identifier start and the following ones valid identifier continuations, using ID_Start/ID_Continue for the DEFAULT profile or XID_Start/XID_Continue for the NFKC profile.
Example
mjb_text identifier = mjb_text_utf8("delta_2", MJB_NUL_TERMINATED);
bool valid = mjb_is_identifier(&identifier, MJB_IDENTIFIER_NFKC);
// Valid identifier: yes
printf("Valid identifier: %s", valid ? "yes" : "no");
Related
mjb_codepoint_is_id_startmjb_codepoint_is_id_continuemjb_codepoint_is_xid_startmjb_codepoint_is_xid_continue
Specifications
mjb_resolved_script_set
Return the UTS #39 resolved script set of a string.
mjb_status mjb_resolved_script_set(
const mjb_text *text,
mjb_script *scripts,
size_t *count,
mjb_script_set_kind *kind
);
Intersect the augmented Script_Extensions sets of every codepoint as specified by UTS #39. Common and Inherited resolve to ALL. Han, Hiragana, Katakana, Hangul, and Bopomofo are augmented with the Hanb, Jpan, and Kore writing-system values. Set scripts to NULL to query the required count. A mixed-script string returns EMPTY with a zero count; an empty string or a string containing only Common or Inherited characters returns ALL with a zero count. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The required count and set kind were returned, or the resolved scripts were writtenMJB_STATUS_INVALID_ARGUMENT-count,kind, ortextis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- The input encoding is invalid or lacks required byte-order informationMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_OUTPUT_TOO_SMALL- The script buffer capacity is smaller than the required countMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_SECURITY=0
Example
mjb_text input = mjb_text_utf8("\xE3\x81\xAD\xE3\x82\xAC", MJB_NUL_TERMINATED); // Hiragana + Katakana
mjb_script scripts[1];
size_t count = 1;
mjb_script_set_kind kind;
if(mjb_resolved_script_set(&input, scripts, &count, &kind) != MJB_STATUS_OK) {
return 1;
}
bool japanese = kind == MJB_SCRIPT_SET_RESOLVED && count == 1 && scripts[0] == MJB_SC_JPAN;
// Japanese writing system: yes
printf("Japanese writing system: %s", japanese ? "yes" : "no");
Related
Specifications
mjb_confusable_skeleton
Compute a Unicode confusable skeleton (Unicode 18.0.0 UTS #39 Section 4).
mjb_status mjb_confusable_skeleton(
const mjb_text *text,
mjb_encoding output_encoding,
mjb_result *result
);
Compute the UTS #39 bidiSkeleton(LTR, input): apply the Unicode Bidirectional Algorithm through L4, then NFD, remove default-ignorables, substitute prototypes from confusables.txt, and reapply NFD. Skeletons can be stored or indexed so future confusable checks can compare them directly. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The confusable skeleton was returnedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_UNSUPPORTED- The output encoding cannot represent a skeleton codepointMJB_STATUS_OVERFLOW- The output size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_SECURITY=0
Example
// Cyrillic U+0435 in place of e
mjb_text input = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED);
mjb_result result;
if(mjb_confusable_skeleton(&input, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) {
return 1;
}
// hello
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);
Related
Specifications
mjb_confusable_skeleton_into
Compute a Unicode confusable skeleton into a caller-provided buffer.
mjb_status mjb_confusable_skeleton_into(
const mjb_text *text,
mjb_buffer *output
);
Compute the same UTS #39 bidiSkeleton(LTR, input) as mjb_confusable_skeleton without allocating the final output buffer. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Bidirectional resolution, normalization, and skeleton mapping still require temporary allocations, including during a size query. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.
Returns
MJB_STATUS_OK- The required size was returned or the skeleton was writtenMJB_STATUS_INVALID_ARGUMENT-outputortextis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- An encoding is invalidMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_UNSUPPORTED- The output encoding cannot represent a skeleton codepointMJB_STATUS_OVERFLOW- The required output size would overflowMJB_STATUS_NO_MEMORY- Temporary allocation failedMJB_STATUS_OUTPUT_TOO_SMALL- The output capacity is smaller than the required byte countMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_SECURITY=0
Example
// Cyrillic U+0435 in place of e
mjb_text input = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
// Query the required size
if(mjb_confusable_skeleton_into(&input, &output) != MJB_STATUS_OK) {
return 1;
}
char storage[5];
if(output.size > sizeof(storage)) {
return 1;
}
output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);
if(mjb_confusable_skeleton_into(&input, &output) != MJB_STATUS_OK) {
return 1;
}
// Skeleton payload (no terminator): hello
printf("Skeleton payload (no terminator): %.*s", (int)output.size, storage);
Related
Specifications
mjb_confusable_match
Determine whether two strings are visually confusable (Unicode 18.0.0 UTS #39 Section 4): skeleton(s1) == skeleton(s2).
mjb_status mjb_confusable_match(
const mjb_text *first,
const mjb_text *second,
mjb_malformed_policy malformed_policy,
bool *confusable,
mjb_diagnostic *first_diagnostic,
mjb_diagnostic *second_diagnostic
);
Compute the confusable skeleton of both strings and store true when the skeletons are equal, meaning the two strings are visually confusable, such as "good" and "gооd" with Cyrillic о. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not examined and second_diagnostic stays clear.
Returns
MJB_STATUS_OK-confusablecontains the comparison resultMJB_STATUS_INVALID_ARGUMENT-confusable,first, orsecondis NULL, a text buffer is NULL with a non-zero size, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- An input encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOPMJB_STATUS_OVERFLOW- An intermediate size would overflowMJB_STATUS_NO_MEMORY- Allocation failedMJB_STATUS_FEATURE_NOT_ENABLED- The library was compiled withMJB_FEATURE_SECURITY=0
Example
mjb_text latin = mjb_text_utf8("hello", MJB_NUL_TERMINATED);
mjb_text mixed = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED); // Cyrillic е
bool confusable;
if(mjb_confusable_match(&latin, &mixed, MJB_MALFORMED_STOP, &confusable, NULL,
NULL) != MJB_STATUS_OK) {
return 1;
}
// Visually confusable: yes
printf("Visually confusable: %s", confusable ? "yes" : "no");
Related
Specifications
API section
Segmentation
Segment text into grapheme clusters, words, and sentences.
mjb_line_break_next
Unicode line break algorithm.
mjb_break_type mjb_line_break_next(
const mjb_text *text,
mjb_line_break_state *state
);
Iterate the line break opportunities of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it.
Example
mjb_text input = mjb_text_utf8("Hello world", 11);
mjb_line_break_state state;
state.index = 0;
mjb_break_type type = mjb_line_break_next(&input, &state);
// First line-break result is set: yes
printf("First line-break result is set: %s", type != MJB_BT_NOT_SET ? "yes" : "no");
Related
Specifications
mjb_word_break_next
Word cluster breaking.
mjb_break_type mjb_word_break_next(
const mjb_text *text,
mjb_word_break_state *state
);
Iterate the word boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it.
Example
mjb_text input = mjb_text_utf8("Hello world", 11);
mjb_word_break_state state;
state.index = 0;
size_t boundaries = 0;
while(mjb_word_break_next(&input, &state) != MJB_BT_NOT_SET) {
++boundaries;
}
// Word-break positions: 11
printf("Word-break positions: %zu", boundaries);
Related
Specifications
mjb_sentence_break_next
Sentence boundaries breaking.
mjb_break_type mjb_sentence_break_next(
const mjb_text *text,
mjb_sentence_break_state *state
);
Iterate the sentence boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it.
Example
mjb_sentence_break_state state;
state.index = 0;
size_t boundaries = 0;
mjb_text input = mjb_text_utf8("Hello. Goodbye.", 15);
while(mjb_sentence_break_next(&input, &state) != MJB_BT_NOT_SET) {
++boundaries;
}
// Sentence-break positions: 15
printf("Sentence-break positions: %zu", boundaries);
Related
Specifications
mjb_sentence_count
Count the sentence segments in a string.
mjb_status mjb_sentence_count(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *count,
mjb_diagnostic *diagnostic
);
Count the sentence segments produced by the default Unicode sentence-boundary rules. The default rules carry no abbreviation list, so text such as Dr. Smith counts as two sentences. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.
Returns
MJB_STATUS_OK- The count was computedMJB_STATUS_INVALID_ARGUMENT-countis NULL, the text is invalid, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- The encoding is not a supported input encodingMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOP
Example
mjb_text input = mjb_text_utf8("Hello. How are you? Fine!", MJB_NUL_TERMINATED);
size_t count;
if(mjb_sentence_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
return 1;
}
// Sentences: 3
printf("Sentences: %zu", count);
Related
Specifications
mjb_grapheme_break_next
Grapheme cluster breaking.
mjb_break_type mjb_grapheme_break_next(
const mjb_text *text,
mjb_grapheme_break_state *state
);
Iterate the grapheme cluster (user-perceived character) boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it.
Example
mjb_text input = mjb_text_utf8("e\xCC\x81", 3); // e + combining acute accent
mjb_grapheme_break_state state;
state.index = 0;
size_t codepoints = 0;
while(mjb_grapheme_break_next(&input, &state) != MJB_BT_NOT_SET) {
++codepoints;
}
// Codepoints examined: 2
printf("Codepoints examined: %zu", codepoints);
Related
Specifications
mjb_grapheme_prefix_length
Return the number of bytes that form the first `max_graphemes` grapheme cluster segments.
size_t mjb_grapheme_prefix_length(
const mjb_text *text,
size_t max_graphemes
);
Example
mjb_text input = mjb_text_utf8("A\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9Z", MJB_NUL_TERMINATED); // A🇮🇹Z
size_t bytes = mjb_grapheme_prefix_length(&input, 2);
// First two graphemes use 9 bytes
printf("First two graphemes use %zu bytes", bytes);
mjb_grapheme_count
Count the extended grapheme clusters in a string.
mjb_status mjb_grapheme_count(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *count,
mjb_diagnostic *diagnostic
);
Count user-perceived characters: the number of extended grapheme cluster segments in the string. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.
Returns
MJB_STATUS_OK- The count was computedMJB_STATUS_INVALID_ARGUMENT-countis NULL, the text is invalid, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- The encoding is not a supported input encodingMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOP
Example
mjb_text input = mjb_text_utf8("A\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // A🇮🇹
size_t count;
if(mjb_grapheme_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
return 1;
}
// Grapheme clusters: 2
printf("Grapheme clusters: %zu", count);
Related
Specifications
mjb_word_prefix_length
Return the number of bytes that form the first max_segments word-break segments.
size_t mjb_word_prefix_length(
const mjb_text *text,
size_t max_segments
);
Example
mjb_text input = mjb_text_utf8("Hello world", MJB_NUL_TERMINATED);
size_t bytes = mjb_word_prefix_length(&input, 1);
// First word segment uses 5 bytes
printf("First word segment uses %zu bytes", bytes);
mjb_word_count
Count the word-like segments in a string.
mjb_status mjb_word_count(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
size_t *count,
mjb_diagnostic *diagnostic
);
Count the words in a string: the word-break segments that contain at least one alphabetic or numeric character. Punctuation, whitespace, and symbol segments are not counted, so Hello, world! counts as two words. Hyphenated compounds count each part, matching the default Unicode word-boundary rules. Unlike mjb_word_prefix_length, whose max_segments counts every raw segment, this function skips non-word segments. For scripts segmented by dictionary lookup in other implementations, such as Chinese, Japanese, Thai, Lao, Khmer, and Burmese, the count approximates one word per character: Mojibake does not use frequency dictionaries to keep the size of the library small. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.
Returns
MJB_STATUS_OK- The count was computedMJB_STATUS_INVALID_ARGUMENT-countis NULL, the text is invalid, or the malformed policy is invalidMJB_STATUS_INVALID_ENCODING- The encoding is not a supported input encodingMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOP
Example
mjb_text input = mjb_text_utf8("Hello, world! It works.", MJB_NUL_TERMINATED);
size_t count;
if(mjb_word_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
return 1;
}
// Words: 4
printf("Words: %zu", count);
Related
Specifications
API section
Bidirectional text
Resolve embedding levels, reorder lines, and get line runs for bidirectional text.
mjb_bidi_resolve
Resolve bidirectional text (TR9) for a paragraph.
mjb_status mjb_bidi_resolve(
const mjb_text *text,
mjb_direction direction,
mjb_bidi_paragraph *result
);
Resolve the embedding levels of a paragraph following the Unicode Bidirectional Algorithm. The resolved paragraph can then be split into lines and reordered visually with mjb_bidi_reorder_line and mjb_bidi_line_runs.
Returns
MJB_STATUS_OK- The paragraph was resolvedMJB_STATUS_INVALID_ARGUMENT-resultortextis NULL, ortext->bufferis NULL with a non-zero sizeMJB_STATUS_INVALID_ENCODING- The input encoding is invalid or lacks byte-order informationMJB_STATUS_MALFORMED_INPUT- The input contains an ill-formed code-unit sequenceMJB_STATUS_OVERFLOW- The paragraph size would overflowMJB_STATUS_NO_MEMORY- Allocation failed
Example
mjb_text input = mjb_text_utf8("abc \xD7\x90\xD7\x91\xD7\x92", MJB_NUL_TERMINATED); // abc אבג
mjb_bidi_paragraph paragraph;
if(mjb_bidi_resolve(&input, MJB_DIRECTION_AUTO, ¶graph) != MJB_STATUS_OK) {
return 1;
}
// Paragraph codepoints: 7
printf("Paragraph codepoints: %zu", paragraph.count);
mjb_bidi_paragraph_free(¶graph);
Related
Specifications
mjb_bidi_reorder_line
Reorder a line visually (L1-L4); visual_order is caller-allocated.
mjb_status mjb_bidi_reorder_line(
const mjb_bidi_paragraph *paragraph,
size_t line_start,
size_t line_end,
size_t *visual_order
);
The levels stored in the paragraph already have rule L1 applied as if the whole paragraph were one line. For this line, whitespace and isolate formatting characters at its end are also reset to the paragraph level (L1) on a private copy of the levels before the L2 reversal; the paragraph itself is not modified.
Example
mjb_text input = mjb_text_utf8("\xD7\x90\xD7\x91\xD7\x92", MJB_NUL_TERMINATED); // אבג
mjb_bidi_paragraph paragraph;
size_t visual_order[3];
if(mjb_bidi_resolve(&input, MJB_DIRECTION_AUTO, ¶graph) != MJB_STATUS_OK ||
mjb_bidi_reorder_line(¶graph, 0, paragraph.count,
visual_order) != MJB_STATUS_OK) {
return 1;
}
// First visual index: 2
printf("First visual index: %zu", visual_order[0]);
mjb_bidi_paragraph_free(¶graph);
Related
Specifications
mjb_bidi_line_runs
Compute visual level runs; pass runs=NULL to count first.
mjb_status mjb_bidi_line_runs(
const mjb_bidi_paragraph *paragraph,
const size_t *visual_order,
size_t count,
mjb_bidi_run *runs,
size_t *run_count
);
Run levels use the same line levels as mjb_bidi_reorder_line: visual_order must be a permutation of one line, and whitespace and isolate formatting characters at the end of that line report the paragraph level (rule L1) even though the paragraph stores their paragraph-wide level.
Returns
MJB_STATUS_OK- The runs were counted or writtenMJB_STATUS_INVALID_ARGUMENT-paragraphorrun_countis NULL,visual_orderis NULL with a non-zerocount, orvisual_orderholds an index outside the paragraph
Example
mjb_text input = mjb_text_utf8("abc", 3);
mjb_bidi_paragraph paragraph;
size_t visual_order[3];
size_t run_count = 0;
if(mjb_bidi_resolve(&input, MJB_DIRECTION_LTR, ¶graph) != MJB_STATUS_OK ||
mjb_bidi_reorder_line(¶graph, 0, 3, visual_order) != MJB_STATUS_OK ||
mjb_bidi_line_runs(¶graph, visual_order, 3, NULL,
&run_count) != MJB_STATUS_OK) {
return 1;
}
// Visual runs: 1
printf("Visual runs: %zu", run_count);
mjb_bidi_paragraph_free(¶graph);
Related
Specifications
mjb_bidi_paragraph_free
Free a bidi paragraph allocated by mjb_bidi_resolve.
void mjb_bidi_paragraph_free(
mjb_bidi_paragraph *paragraph
);
Example
mjb_text input = mjb_text_utf8("abc", 3);
mjb_bidi_paragraph paragraph;
if(mjb_bidi_resolve(&input, MJB_DIRECTION_LTR, ¶graph) != MJB_STATUS_OK) {
return 1;
}
mjb_bidi_paragraph_free(¶graph);
// Paragraph released: yes
printf("Paragraph released: %s", paragraph.chars == NULL ? "yes" : "no");
Related
API section
Emoji
Inspect emoji sequences and properties.
mjb_codepoint_emoji_properties
Return the emoji properties.
mjb_status mjb_codepoint_emoji_properties(
mjb_codepoint codepoint,
mjb_emoji_properties *emoji
);
Example
mjb_emoji_properties emoji;
if(mjb_codepoint_emoji_properties(0x1F600, &emoji) != MJB_STATUS_OK) {
return 1;
}
// U+1F600 has Emoji_Presentation: yes
printf("U+1F600 has Emoji_Presentation: %s", emoji.presentation ? "yes" : "no");
Related
Specifications
mjb_codepoint_is_emoji
Return true if the codepoint has the Unicode Emoji property.
bool mjb_codepoint_is_emoji(
mjb_codepoint codepoint
);
Examples
// Number sign has the Emoji property: yes
bool emoji = mjb_codepoint_is_emoji('#');
// Number sign has the Emoji property: yes
printf("Number sign has the Emoji property: %s", emoji ? "yes" : "no");// Number sign has the Emoji property: yes
std::cout << "Number sign has the Emoji property: "
<< (mjb::is_emoji('#') ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_emoji_presentation
Return true if the codepoint has the Unicode Emoji_Presentation property.
bool mjb_codepoint_is_emoji_presentation(
mjb_codepoint codepoint
);
Examples
// Grinning face defaults to emoji presentation: yes
bool presentation = mjb_codepoint_is_emoji_presentation(0x1F600);
// Grinning face defaults to emoji presentation: yes
printf("Grinning face defaults to emoji presentation: %s", presentation ? "yes" : "no");// Grinning face defaults to emoji presentation: yes
std::cout << "Grinning face defaults to emoji presentation: "
<< (mjb::is_emoji_presentation(0x1F600) ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_emoji_modifier
Return true if the codepoint has the Unicode Emoji_Modifier property.
bool mjb_codepoint_is_emoji_modifier(
mjb_codepoint codepoint
);
Examples
// Medium skin tone is an emoji modifier: yes
bool modifier = mjb_codepoint_is_emoji_modifier(0x1F3FD);
// Medium skin tone is an emoji modifier: yes
printf("Medium skin tone is an emoji modifier: %s", modifier ? "yes" : "no");// Medium skin tone is an emoji modifier: yes
std::cout << "Medium skin tone is an emoji modifier: "
<< (mjb::is_emoji_modifier(0x1F3FD) ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_emoji_modifier_base
Return true if the codepoint has the Unicode Emoji_Modifier_Base property.
bool mjb_codepoint_is_emoji_modifier_base(
mjb_codepoint codepoint
);
Examples
// Waving hand accepts an emoji modifier: yes
bool modifier_base = mjb_codepoint_is_emoji_modifier_base(0x1F44B);
// Waving hand accepts an emoji modifier: yes
printf("Waving hand accepts an emoji modifier: %s", modifier_base ? "yes" : "no");// Waving hand accepts an emoji modifier: yes
std::cout << "Waving hand accepts an emoji modifier: "
<< (mjb::is_emoji_modifier_base(0x1F44B) ? "yes" : "no") << std::endl;Specifications
mjb_codepoint_is_emoji_component
Return true if the codepoint has the Unicode Emoji_Component property.
bool mjb_codepoint_is_emoji_component(
mjb_codepoint codepoint
);
Examples
// Zero-width joiner is an emoji component: yes
bool component = mjb_codepoint_is_emoji_component(0x200D);
// Zero-width joiner is an emoji component: yes
printf("Zero-width joiner is an emoji component: %s", component ? "yes" : "no");// Zero-width joiner is an emoji component: yes
std::cout << "Zero-width joiner is an emoji component: "
<< (mjb::is_emoji_component(0x200D) ? "yes" : "no") << std::endl;Specifications
mjb_emoji_sequence_info
Return emoji sequence metadata for a complete string.
mjb_status mjb_emoji_sequence_info(
const mjb_text *text,
mjb_emoji_sequence *emoji
);
Example
mjb_text flag = mjb_text_utf8("\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // 🇮🇹
mjb_emoji_sequence emoji;
if(mjb_emoji_sequence_info(&flag, &emoji) != MJB_STATUS_OK) {
return 1;
}
// Sequence codepoints: 2
printf("Sequence codepoints: %zu", emoji.codepoint_count);
Related
Specifications
mjb_is_emoji_sequence
Return true if the complete string is an emoji sequence listed by Unicode, including standardized emoji variation sequences.
bool mjb_is_emoji_sequence(
const mjb_text *text
);
Example
mjb_text keycap = mjb_text_utf8("1\xEF\xB8\x8F\xE2\x83\xA3", MJB_NUL_TERMINATED); // 1️⃣
bool listed = mjb_is_emoji_sequence(&keycap);
// Listed emoji sequence: yes
printf("Listed emoji sequence: %s", listed ? "yes" : "no");
Related
Specifications
mjb_is_rgi_emoji
Return true if the complete string is an RGI emoji sequence, excluding plain standardized variation sequences.
bool mjb_is_rgi_emoji(
const mjb_text *text
);
Example
mjb_text flag = mjb_text_utf8("\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // 🇮🇹
bool rgi = mjb_is_rgi_emoji(&flag);
// RGI emoji: yes
printf("RGI emoji: %s", rgi ? "yes" : "no");
Related
Specifications
API section
Terminal width
Estimate the terminal cells occupied by printable, single-line text.
mjb_grapheme_prefix_length_for_width
Return the number of bytes whose grapheme clusters fit within max_columns terminal cells.
size_t mjb_grapheme_prefix_length_for_width(
const mjb_text *text,
mjb_terminal_width_profile profile,
size_t max_columns
);
Truncate only at extended grapheme-cluster boundaries, using the same terminal-cell policy as mjb_terminal_width. The returned prefix stops before a cluster that would exceed the budget or cannot be measured as printable, single-line terminal text.
Example
mjb_text input = mjb_text_utf8("A\xE7\x95\x8C", MJB_NUL_TERMINATED); // A界
size_t bytes = mjb_grapheme_prefix_length_for_width(&input, MJB_TERMINAL_WIDTH_NARROW, 2);
// Two columns include 1 byte
printf("Two columns include %zu byte", bytes);
mjb_word_prefix_length_for_width
Return the number of bytes whose word-break segments fit within max_columns terminal cells.
size_t mjb_word_prefix_length_for_width(
const mjb_text *text,
mjb_terminal_width_profile profile,
size_t max_columns
);
Truncate only at word-break boundaries, using the same terminal-cell policy as mjb_terminal_width. The returned prefix stops before a segment that would exceed the budget or cannot be measured as printable, single-line terminal text.
Example
mjb_text input = mjb_text_utf8("Hello world", MJB_NUL_TERMINATED);
size_t bytes = mjb_word_prefix_length_for_width(&input, MJB_TERMINAL_WIDTH_NARROW, 6);
// Six columns include 6 bytes
printf("Six columns include %zu bytes", bytes);
mjb_codepoint_east_asian_width
Return the east asian width of a codepoint.
mjb_status mjb_codepoint_east_asian_width(
mjb_codepoint codepoint,
mjb_east_asian_width *width
);
Example
mjb_east_asian_width width;
if(mjb_codepoint_east_asian_width(0x754C, &width) != MJB_STATUS_OK) { // 界
return 1;
}
// U+754C is wide: yes
printf("U+754C is wide: %s", width == MJB_EAW_WIDE ? "yes" : "no");
Related
Specifications
mjb_terminal_width
Return the estimated terminal-cell width of printable, single-line text.
mjb_status mjb_terminal_width(
const mjb_text *text,
mjb_malformed_policy malformed_policy,
mjb_terminal_width_profile profile,
size_t *width,
mjb_diagnostic *diagnostic
);
Estimate the number of fixed terminal cells occupied by printable, single-line text. The input is normalized to NFC so canonically equivalent text has the same width. Combining and format characters occupy no additional cells, listed emoji-presentation sequences occupy two cells, and text-presentation sequences retain their East Asian Width. This is a deterministic terminal policy, not a measurement of proportional glyph advances. Use grapheme boundaries for cursor movement, selection, deletion, and user-perceived character counts. Controls, line separators, and paragraph separators are rejected because their effect depends on terminal state. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, width is set to zero.
Returns
MJB_STATUS_OK- The width was computedMJB_STATUS_INVALID_ARGUMENT-widthis NULL, the profile or malformed policy is invalid, or the text is invalidMJB_STATUS_INVALID_ENCODING- The encoding is invalid or lacks required byte-order informationMJB_STATUS_UNSUPPORTED- The input contains a control, line separator, or paragraph separatorMJB_STATUS_NO_MEMORY- NFC normalization could not allocate memoryMJB_STATUS_OVERFLOW- The width would overflowMJB_STATUS_MALFORMED_INPUT- Malformed input was encountered withMJB_MALFORMED_STOP
Example
mjb_text input = mjb_text_utf8("A\xE7\x95\x8C", MJB_NUL_TERMINATED); // A界
size_t width;
if(mjb_terminal_width(&input, MJB_MALFORMED_STOP,
MJB_TERMINAL_WIDTH_NARROW, &width, NULL) != MJB_STATUS_OK) {
return 1;
}
// Terminal cells: 3
printf("Terminal cells: %zu", width);
Related
mjb_codepoint_east_asian_widthmjb_grapheme_prefix_length_for_widthmjb_grapheme_break_nextmjb_is_rgi_emoji
Specifications
API section
Hangul language
Inspect Hangul syllables and perform Hangul syllable decomposition and composition.
mjb_codepoint_is_hangul_leading_jamo
Return if the codepoint is a hangul L.
bool mjb_codepoint_is_hangul_leading_jamo(
mjb_codepoint codepoint
);
Examples
// U+1100 is a leading Jamo: yes
printf("U+1100 is a leading Jamo: %s", mjb_codepoint_is_hangul_leading_jamo(0x1100) ? "yes" : "no");// U+1100 is a leading Jamo: yes
std::cout << "U+1100 is a leading Jamo: "
<< (mjb::is_hangul_leading_jamo(0x1100) ? "yes" : "no") << std::endl;mjb_codepoint_is_hangul_vowel_jamo
Return if the codepoint is a hangul V.
bool mjb_codepoint_is_hangul_vowel_jamo(
mjb_codepoint codepoint
);
Examples
// U+1161 is a vowel Jamo: yes
printf("U+1161 is a vowel Jamo: %s", mjb_codepoint_is_hangul_vowel_jamo(0x1161) ? "yes" : "no");// U+1161 is a vowel Jamo: yes
std::cout << "U+1161 is a vowel Jamo: "
<< (mjb::is_hangul_vowel_jamo(0x1161) ? "yes" : "no") << std::endl;mjb_codepoint_is_hangul_trailing_jamo
Return if the codepoint is a hangul T.
bool mjb_codepoint_is_hangul_trailing_jamo(
mjb_codepoint codepoint
);
Examples
// U+11A8 is a trailing Jamo: yes
printf("U+11A8 is a trailing Jamo: %s", mjb_codepoint_is_hangul_trailing_jamo(0x11A8) ? "yes" : "no");// U+11A8 is a trailing Jamo: yes
std::cout << "U+11A8 is a trailing Jamo: "
<< (mjb::is_hangul_trailing_jamo(0x11A8) ? "yes" : "no") << std::endl;mjb_codepoint_is_hangul_jamo
Return if the codepoint is a hangul jamo.
bool mjb_codepoint_is_hangul_jamo(
mjb_codepoint codepoint
);
Examples
// U+1100 is Hangul Jamo: yes
printf("U+1100 is Hangul Jamo: %s", mjb_codepoint_is_hangul_jamo(0x1100) ? "yes" : "no");// U+1100 is Hangul Jamo: yes
std::cout << "U+1100 is Hangul Jamo: " << (mjb::is_hangul_jamo(0x1100) ? "yes" : "no") << std::endl;mjb_codepoint_is_hangul_syllable
Return if the codepoint is a hangul syllable.
bool mjb_codepoint_is_hangul_syllable(
mjb_codepoint codepoint
);
Examples
// U+AC00 is a Hangul syllable: yes
printf("U+AC00 is a Hangul syllable: %s", mjb_codepoint_is_hangul_syllable(0xAC00) ? "yes" : "no");// U+AC00 is a Hangul syllable: yes
std::cout << "U+AC00 is a Hangul syllable: "
<< (mjb::is_hangul_syllable(0xAC00) ? "yes" : "no") << std::endl;mjb_hangul_syllable_name
Return hangul syllable name.
mjb_status mjb_hangul_syllable_name(
mjb_codepoint codepoint,
char *buffer,
size_t byte_length
);
Example
char name[32];
if(mjb_hangul_syllable_name(0xAC01, name, sizeof(name)) != MJB_STATUS_OK) {
return 1;
}
// Name: HANGUL SYLLABLE GAG
printf("Name: %s", name);
mjb_hangul_syllable_decompose
Decompose a Hangul syllable into its jamo.
mjb_status mjb_hangul_syllable_decompose(
mjb_codepoint codepoint,
mjb_codepoint *codepoints
);
Apply the arithmetic Hangul syllable decomposition of Unicode Section 3.12, the inverse of mjb_hangul_syllable_compose.
Returns
MJB_STATUS_OK- The syllable was decomposedMJB_STATUS_INVALID_ARGUMENT-codepointsis NULLMJB_STATUS_NOT_FOUND-codepointis not a Hangul syllable
Example
mjb_codepoint decomposition[3];
if(mjb_hangul_syllable_decompose(0xAC01, decomposition) != MJB_STATUS_OK) {
return 1;
}
// Decomposition starts with: U+1100
printf("Decomposition starts with: U+%04X", decomposition[0]);
Related
mjb_hangul_syllable_compose
Compose Hangul jamo into a syllable.
mjb_status mjb_hangul_syllable_compose(
mjb_codepoint leading,
mjb_codepoint vowel,
mjb_codepoint trailing,
mjb_codepoint *syllable
);
Apply the arithmetic Hangul syllable composition of Unicode Section 3.12, the inverse of mjb_hangul_syllable_decompose. Normalization composes Hangul automatically; use this function to build a syllable from individual jamo.
Returns
MJB_STATUS_OK- The syllable was composedMJB_STATUS_INVALID_ARGUMENT-syllableis NULLMJB_STATUS_NOT_FOUND- A jamo is outside its range, so the jamo do not form a syllable
Example
mjb_codepoint syllable;
// Choseong kiyeok, jungseong a, jongseong kiyeok
if(mjb_hangul_syllable_compose(0x1100, 0x1161, 0x11A8, &syllable) != MJB_STATUS_OK) {
return 1;
}
// Syllable: U+AC01
printf("Syllable: U+%04X", syllable);
Related
API section
Utilities
Use locales, memory hooks, version details, and low-level string helpers.
mjb_property_name
Return the name of a property, NULL if the property specified is not valid.
const char *mjb_property_name(
mjb_property property
);
Example
const char *name = mjb_property_name(MJB_PR_ALPHABETIC);
// Property: Alphabetic
printf("Property: %s", name);
Specifications
mjb_locale_parse
Parse a BCP 47 language tag.
mjb_status mjb_locale_parse(
const mjb_text *id,
mjb_locale_id *locale
);
Parse a BCP 47 language tag, such as sr-Latn-RS, into its components: language, extended language, script, region, variant, extensions, private use, and grandfathered tags. Parsing is strict: malformed tags are rejected.
Returns
MJB_STATUS_OK- The tag was parsed andlocalefilledMJB_STATUS_INVALID_ARGUMENT- An argument is NULL or the tag is not a valid BCP 47 language tagMJB_STATUS_NO_MEMORY- Allocation failed
Example
mjb_text id = mjb_text_utf8("sr-Latn-RS", MJB_NUL_TERMINATED);
mjb_locale_id locale;
if(mjb_locale_parse(&id, &locale) != MJB_STATUS_OK) {
return 1;
}
// Locale: sr Latn RS
printf("Locale: %s %s %s", locale.language, locale.script, locale.region);
Related
Specifications
mjb_locale_code
Return the ISO 639-1 code of a locale, NULL if the locale is not valid.
const char *mjb_locale_code(
mjb_locale locale
);
Return the lowercase two-letter ISO 639-1 code that names locale, such as tr for MJB_LOCALE_TR. MJB_LOCALE_NOT_SET yields an empty string.
Returns
const char *- The ISO 639-1 code, or NULL iflocaleis not a validmjb_localevalue
Example
// Locale code: tr
printf("Locale code: %s", mjb_locale_code(MJB_LOCALE_TR));
Related
mjb_result_free
Free a mjb_result.
mjb_status mjb_result_free(
mjb_result *result
);
Free the memory allocated for a mjb_result. The result pointer is set to NULL.
Returns
MJB_STATUS_OK- The result was freedMJB_STATUS_INVALID_ARGUMENT-resultis NULL
Example
mjb_result result;
mjb_text input = mjb_text_utf8("A", 1);
if(mjb_convert_encoding(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_16LE, &result,
NULL) != MJB_STATUS_OK ||
mjb_result_free(&result) != MJB_STATUS_OK) {
return 1;
}
// Result released: yes
printf("Result released: %s", result.output == NULL ? "yes" : "no");
mjb_status_message
Return a UTF-8 English-language message that describes a status code.
const char *mjb_status_message(
mjb_status status
);
Return a static English-language message of at most 100 characters.
Example
const char *message = mjb_status_message(MJB_STATUS_INVALID_ARGUMENT);
// Error message: One or more arguments are invalid or inconsistent with the requested operation
printf("Error message: %s", message);
mjb_version
Output the current library version (MJB_VERSION).
const char *mjb_version(void);
Output the current library version as a string, such as "1.0.0".
Example
const char *version = mjb_version();
// Version is available: yes
printf("Version is available: %s", version[0] != '\0' ? "yes" : "no");
Related
mjb_version_number
Output the current library version number (MJB_VERSION_NUMBER).
unsigned int mjb_version_number(void);
Output the current library version number as an unsigned integer.
Example
unsigned int version = mjb_version_number();
// Version number is positive: yes
printf("Version number is positive: %s", version > 0 ? "yes" : "no");
Related
mjb_unicode_version
Output the current supported Unicode version (MJB_UNICODE_VERSION).
const char *mjb_unicode_version(void);
Output the current supported Unicode version as a string, such as "15.0.0".
Example
const char *version = mjb_unicode_version();
// Unicode version: 18.0.0
printf("Unicode version: %s", version);
Related
mjb_set_allocator
Set the process-global memory allocator.
mjb_status mjb_set_allocator(
const mjb_allocator *allocator
);
Install the allocator used for internal allocations and owned result buffers. The allocator is copied and must provide all three callbacks, which receive its context pointer. The library passes the size of every block it reallocates or frees, so arena, pool, and accounting allocators do not need to record sizes themselves: old_size and size are the size most recently requested for that block, requested sizes are never 0, realloc receives NULL with an old_size of 0 for a new block and leaves the old block valid on failure, and free is never called with NULL. The context remains caller-owned and must stay valid for the lifetime of all Mojibake allocations. Pass NULL to explicitly select the default allocator. This process-global configuration may be set only once and must happen before any other library call or concurrent library use.
Returns
MJB_STATUS_OK- The allocator was installedMJB_STATUS_INVALID_ARGUMENT- A callback is NULL, or the allocator was already configured
Example
// An allocator that counts the bytes in use. The context points to the counter.
static void *counting_alloc(void *context, size_t size) {
void *ptr = malloc(size);
if(ptr) {
*(size_t *)context += size;
}
return ptr;
}
static void *counting_realloc(void *context, void *ptr, size_t old_size, size_t new_size) {
void *new_ptr = realloc(ptr, new_size);
if(new_ptr) {
*(size_t *)context = *(size_t *)context - old_size + new_size;
}
return new_ptr;
}
static void counting_free(void *context, void *ptr, size_t size) {
*(size_t *)context -= size;
free(ptr);
}
static size_t bytes_in_use = 0;
mjb_allocator allocator = { &bytes_in_use, counting_alloc, counting_realloc, counting_free };
// Install it once, before any other library call
if(mjb_set_allocator(&allocator) != MJB_STATUS_OK) {
return 1;
}