Mojibake

Normalize a string to the requested Unicode normalization form. If the input is already normalized and no encoding conversion is needed, text->buffer is returned as-is in result->output with result->owned set to false, without allocating. Malformed subsequences follow malformed_policy, and diagnostic records the first one.

Returns

• MJB_STATUS_OK - The string was normalized (or already normal)
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_INVALID_FORM - form is not NFC, NFD, NFKC, or NFKD
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a normalized codepoint
• MJB_STATUS_OVERFLOW - The output size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed

Examples

// "Cafe" + U+0301 COMBINING ACUTE ACCENT
mjb_text input = mjb_text_utf8("Cafe\xCC\x81", MJB_NUL_TERMINATED);
mjb_result result;

if(mjb_normalize(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// NFC: Café
printf("NFC: %.*s", (int)result.output_size, result.output);

mjb_result_free(&result);

// "Cafe" + U+0301 COMBINING ACUTE ACCENT
std::string input = "Cafe\xCC\x81";
std::string output = mjb::normalize(input, mjb::NormalizationForm::NFC);

// NFC: Café
std::cout << "NFC: " << output << std::endl;

Related

• mjb_normalize_into
• mjb_normalization_quick_check
• mjb_filter
• mjb_filter_into

Specifications

• UAX #15: Unicode Normalization Forms, Unicode 18.0.0

Normalize a string using the same Unicode normalization forms and encoding rules as mjb_normalize. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. NFD and NFKD write without allocation. NFC and NFKC may allocate temporary composition storage, including during a size query.

Returns

• MJB_STATUS_OK - The required size was returned or the normalized string was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_INVALID_FORM - form is not NFC, NFD, NFKC, or NFKD
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a normalized codepoint
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_NO_MEMORY - Temporary composition allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count

Examples

const char *cafe = "Cafe\xCC\x81"; // "Cafe" + U+0301 COMBINING ACUTE ACCENT
mjb_text input = mjb_text_utf8(cafe, strlen(cafe));
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);

// Query the required size
if(mjb_normalize_into(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

char storage[5];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_normalize_into(&input, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// NFC payload (no terminator): Café
printf("NFC payload (no terminator): %.*s", (int)output.size, storage);

// "Cafe" + U+0301 COMBINING ACUTE ACCENT
std::string input = "Cafe\xCC\x81";
std::string output;

// Replaces the content of output and reuses its capacity on later calls
mjb::normalize(input, mjb::NormalizationForm::NFC, output);

// NFC: Café
std::cout << "NFC: " << output << std::endl;

Related

• mjb_normalize
• mjb_normalization_quick_check
• mjb_filter_into
• mjb_nfkc_casefold_into

Specifications

• UAX #15: Unicode Normalization Forms, Unicode 18.0.0

MJB_FILTER_LIMIT_COMBINING removes combining marks after the first MJB_FILTER_MAX_COMBINING_MARKS consecutive marks in an emitted run. This is useful for reducing Zalgo-style text while keeping ordinary accents and stacked marks. Malformed subsequences are stopped, replaced, or skipped according to malformed_policy; diagnostic records the first one encountered.

Returns

• MJB_STATUS_OK - The filtered string was returned
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a filtered codepoint
• MJB_STATUS_OVERFLOW - The output size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed

Examples

mjb_text mixed_whitespace = mjb_text_utf8("Hello\t\t\n\nworld", MJB_NUL_TERMINATED);
mjb_result result;

if(mjb_filter(&mixed_whitespace, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, MJB_ENC_UTF_8,
    &result, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Filtered: Hello world
printf("Filtered: %.*s", (int)result.output_size, result.output);

mjb_result_free(&result);

mjb_text controls = mjb_text_utf8("\x1\x2\t\n\v\f\r\x1f", MJB_NUL_TERMINATED);

if(mjb_filter(&controls, MJB_MALFORMED_STOP, MJB_FILTER_CONTROLS, MJB_ENC_UTF_8, &result,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Filtered: \t\n\v\f\r
printf("Filtered: %.*s", (int)result.output_size, result.output);

mjb_result_free(&result);

std::string mixed_whitespace = "Hello\t\t\n\nworld";

// Filtered: Hello world
std::cout << "Filtered: " << mjb::filter(mixed_whitespace, mjb::Filter::CollapseSpaces)
    << std::endl;

std::string controls = "\x1\x2\t\n\v\f\r\x1f";

// Filtered: \t\n\v\f\r
std::cout << "Filtered: " << mjb::filter(controls, mjb::Filter::Controls) << std::endl;

Related

• mjb_filter_into
• mjb_normalize

Apply the same filters as mjb_filter without allocating the final output buffer. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Filtering itself does not allocate, but MJB_FILTER_NORMALIZE may allocate temporary normalization storage. MJB_FILTER_LIMIT_COMBINING keeps the first MJB_FILTER_MAX_COMBINING_MARKS consecutive marks in each emitted run. Malformed subsequences follow malformed_policy, and diagnostic records the first one.

Returns

• MJB_STATUS_OK - The required size was returned or the filtered string was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a filtered codepoint
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_NO_MEMORY - Temporary normalization allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count

Examples

mjb_text input = mjb_text_utf8("Hello\t\t\nworld", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);

// Query the required size
if(mjb_filter_into(&input, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

char storage[11];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_filter_into(&input, MJB_MALFORMED_STOP, MJB_FILTER_COLLAPSE_SPACES, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Filtered payload (no terminator): Hello world
printf("Filtered payload (no terminator): %.*s", (int)output.size, storage);

std::string input = "Hello\t\t\nworld";
std::string output;

// Replaces the content of output and reuses its capacity on later calls
mjb::filter(input, mjb::Filter::CollapseSpaces, output);

// Filtered: Hello world
std::cout << "Filtered: " << output << std::endl;

Related

• mjb_filter
• mjb_normalize

Apply the normative NFKC_Casefold mapping and normalize the result to NFC. This transform performs compatibility folding, full default case folding, and removal of default-ignorable codepoints. It is intended for identifier comparison and is not locale-sensitive. Malformed subsequences follow malformed_policy, and diagnostic records the first one.

Returns

• MJB_STATUS_OK - The transformed string was returned
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - The output size would overflow
• MJB_STATUS_UNSUPPORTED - The transform did not stabilize, or the requested output encoding cannot represent a result codepoint
• MJB_STATUS_NO_MEMORY - Allocation failed

Examples

mjb_text input = mjb_text_utf8("Stra\xC3\x9F" "e\xC2\xAD", MJB_NUL_TERMINATED);
mjb_result result;

if(mjb_nfkc_casefold(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_8, &result,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// strasse
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

// "Straße" + U+00AD SOFT HYPHEN
std::string input = "Stra\xC3\x9F" "e\xC2\xAD";

// strasse
std::cout << mjb::nfkc_casefold(input) << std::endl;

Related

• mjb_nfkc_casefold_into
• mjb_normalize
• mjb_map_case
• mjb_is_identifier

Specifications

• The Unicode Standard, Version 18.0.0, Section 3.13: Default Case Algorithms
• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0
• UAX #44: Unicode Character Database, Unicode 18.0.0

Apply the same normative NFKC_Casefold transform as mjb_nfkc_casefold. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. The final output uses caller-provided storage, but the normalization and folding passes require temporary allocations, including during a size query.

Returns

• MJB_STATUS_OK - The required size was returned or the transformed string was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_UNSUPPORTED - The transform did not stabilize, or the requested output encoding cannot represent a result codepoint
• MJB_STATUS_NO_MEMORY - Temporary allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count

Examples

mjb_text input = mjb_text_utf8("Stra\xC3\x9F" "e\xC2\xAD", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);

// Query the required size
if(mjb_nfkc_casefold_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

char storage[7];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_nfkc_casefold_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

// NFKC casefold payload (no terminator): strasse
printf("NFKC casefold payload (no terminator): %.*s", (int)output.size, storage);

// "Straße" + U+00AD SOFT HYPHEN
std::string input = "Stra\xC3\x9F" "e\xC2\xAD";
std::string output;

// Replaces the content of output and reuses its capacity on later calls
mjb::nfkc_casefold(input, output);

// NFKC casefold: strasse
std::cout << "NFKC casefold: " << output << std::endl;

Related

• mjb_nfkc_casefold
• mjb_normalize
• mjb_map_case
• mjb_is_identifier

Specifications

• The Unicode Standard, Version 18.0.0, Section 3.13: Default Case Algorithms
• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0
• UAX #44: Unicode Character Database, Unicode 18.0.0

Apply UTS #46 nontransitional processing with STD3 ASCII, hyphen, joiner, bidi, label-length, and domain-length checks enabled. Non-ASCII labels are encoded with Punycode. A successful operational status can still set info->errors; an errored ToASCII result must not be used for DNS lookup. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - Processing completed; inspect info->errors for validation failures
• MJB_STATUS_INVALID_ARGUMENT - text, info, or result is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent the result
• MJB_STATUS_OVERFLOW - An output or Punycode size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_IDNA=0

Examples

// "bücher.example"
mjb_text domain = mjb_text_utf8("b\xC3\xBC" "cher.example", MJB_NUL_TERMINATED);
mjb_idna_info info;
mjb_result result;

if(mjb_idna_to_ascii(&domain, MJB_ENC_UTF_8, &info, &result) != MJB_STATUS_OK ||
    info.errors != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// xn--bcher-kva.example
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

// "bücher.example"
auto domain = mjb::idna_to_ascii("b\xC3\xBC" "cher.example");

if(!domain.valid()) {
    return 1;
}

// xn--bcher-kva.example
std::cout << domain.text << std::endl;

Related

• mjb_idna_to_ascii_into
• mjb_idna_to_unicode
• mjb_idna_to_unicode_into

Specifications

• UTS #46: Unicode IDNA Compatibility Processing, Unicode 18.0.0

Apply the same strict nontransitional profile as mjb_idna_to_ascii. output->encoding selects the output encoding. Set output->data to NULL to query the required byte count; otherwise output->size supplies its capacity, and on return it contains the required or written byte count. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Processing uses temporary allocations even during a size query. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The required size was returned or the ASCII domain was written
• MJB_STATUS_INVALID_ARGUMENT - text, info, or output is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent the result
• MJB_STATUS_OVERFLOW - An output or Punycode size would overflow
• MJB_STATUS_NO_MEMORY - A temporary allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_IDNA=0

Examples

// "bücher.example"
mjb_text domain = mjb_text_utf8("b\xC3\xBC" "cher.example", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
mjb_idna_info info;

// Query the required size
if(mjb_idna_to_ascii_into(&domain, &info, &output) != MJB_STATUS_OK) {
    return 1;
}

char storage[21];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_idna_to_ascii_into(&domain, &info, &output) != MJB_STATUS_OK ||
    info.errors != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// ASCII payload (no terminator): xn--bcher-kva.example
printf("ASCII payload (no terminator): %.*s", (int)output.size, storage);

// "bücher.example"
std::string domain = "b\xC3\xBC" "cher.example";
std::string output;

// Returns the mjb_idna_error flags
if(mjb::idna_to_ascii(domain, output) != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// xn--bcher-kva.example
std::cout << output << std::endl;

Related

• mjb_idna_to_ascii
• mjb_idna_to_unicode
• mjb_idna_to_unicode_into

Specifications

• UTS #46: Unicode IDNA Compatibility Processing, Unicode 18.0.0

Apply UTS #46 nontransitional processing and decode valid xn-- labels. The function returns its best-effort converted string even when info->errors records a validation problem. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - Processing completed; inspect info->errors for validation failures
• MJB_STATUS_INVALID_ARGUMENT - text, info, or result is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent the result
• MJB_STATUS_OVERFLOW - An output or Punycode size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_IDNA=0

Examples

mjb_text domain = mjb_text_utf8("xn--bcher-kva.example", MJB_NUL_TERMINATED);
mjb_idna_info info;
mjb_result result;

if(mjb_idna_to_unicode(&domain, MJB_ENC_UTF_8, &info, &result) != MJB_STATUS_OK ||
    info.errors != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// bücher.example
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

auto domain = mjb::idna_to_unicode("xn--bcher-kva.example");

if(!domain.valid()) {
    return 1;
}

// bücher.example
std::cout << domain.text << std::endl;

Related

• mjb_idna_to_unicode_into
• mjb_idna_to_ascii
• mjb_idna_to_ascii_into

Specifications

• UTS #46: Unicode IDNA Compatibility Processing, Unicode 18.0.0

Apply the same strict nontransitional profile as mjb_idna_to_unicode. output->encoding selects the output encoding. Set output->data to NULL to query the required byte count; otherwise output->size supplies its capacity, and on return it contains the required or written byte count. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Processing uses temporary allocations even during a size query. If MJB_FEATURE_IDNA=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The required size was returned or the Unicode domain was written
• MJB_STATUS_INVALID_ARGUMENT - text, info, or output is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent the result
• MJB_STATUS_OVERFLOW - An output or Punycode size would overflow
• MJB_STATUS_NO_MEMORY - A temporary allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_IDNA=0

Examples

mjb_text domain = mjb_text_utf8("xn--bcher-kva.example", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);
mjb_idna_info info;

// Query the required size
if(mjb_idna_to_unicode_into(&domain, &info, &output) != MJB_STATUS_OK) {
    return 1;
}

char storage[15];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_idna_to_unicode_into(&domain, &info, &output) != MJB_STATUS_OK ||
    info.errors != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// Unicode payload (no terminator): bücher.example
printf("Unicode payload (no terminator): %.*s", (int)output.size, storage);

std::string domain = "xn--bcher-kva.example";
std::string output;

// Returns the mjb_idna_error flags
if(mjb::idna_to_unicode(domain, output) != MJB_IDNA_ERROR_NONE) {
    return 1;
}

// bücher.example
std::cout << output << std::endl;

Related

• mjb_idna_to_unicode
• mjb_idna_to_ascii
• mjb_idna_to_ascii_into

Specifications

• UTS #46: Unicode IDNA Compatibility Processing, Unicode 18.0.0

Example

char encoded[4];
unsigned int size = mjb_codepoint_encode(0x20AC, encoded, sizeof(encoded), MJB_ENC_UTF_8);

// € sign uses 3 UTF-8 bytes
printf("%.*s sign uses %u UTF-8 bytes", (int)size, encoded, size);

Convert a string between the supported encodings (UTF-8, UTF-16LE/BE, UTF-32LE/BE). Generic UTF-16/UTF-32 input consumes a leading BOM as the encoding scheme signature and uses it to resolve byte order. Explicit-endian input preserves an initial U+FEFF as text. Generic UTF-16/UTF-32 without a BOM, and generic UTF-16/UTF-32 output, are rejected because the byte order is not specified. Malformed source subsequences are stopped, replaced, or skipped according to malformed_policy.

Returns

• MJB_STATUS_OK - The string was converted
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent an input codepoint
• MJB_STATUS_OVERFLOW - The output size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed

Example

mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_result result;

if(mjb_convert_encoding(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_16LE, &result,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// UTF-16LE bytes: 8
printf("UTF-16LE bytes: %zu", result.output_size);
mjb_result_free(&result);

Related

• mjb_convert_encoding_into
• mjb_detect_encoding
• mjb_codepoint_encode

Convert a string using the same encoding and BOM rules as mjb_convert_encoding, without allocating memory. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. The size is the encoded payload byte count: terminators are excluded, and this function does not write a terminator. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity.

Returns

• MJB_STATUS_OK - The required size was returned or the string was converted
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent an input codepoint
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count

Example

mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_16LE);

// Query the required size
if(mjb_convert_encoding_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

unsigned char storage[8];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_16LE);

if(mjb_convert_encoding_into(&input, MJB_MALFORMED_STOP, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

// UTF-16LE payload bytes (no terminator): 8
printf("UTF-16LE payload bytes (no terminator): %zu", output.size);

Related

• mjb_convert_encoding
• mjb_detect_encoding
• mjb_codepoint_encode

Convert a string to uppercase, lowercase, titlecase, or its case-folded form. Full case mappings are applied, including special casing and conditional mappings, so the output may have a different length than the input. Titlecase uses UAX #29 word boundaries: the first cased character in each word segment is titlecased, and subsequent characters in that segment are lowercased. locale selects the language-sensitive rules of SpecialCasing.txt for this call only; there is no global locale, so concurrent calls with different locales do not interfere. MJB_LOCALE_TR and MJB_LOCALE_AZ apply Turkish/Azerbaijani dotted-I casing and Turkic T case-folding mappings. MJB_LOCALE_LT applies Lithuanian dot-above casing rules, while case folding remains the default non-Turkic mapping. MJB_LOCALE_NOT_SET and every other valid locale use the default mappings. Malformed subsequences follow malformed_policy, and diagnostic records the first one.

Returns

• MJB_STATUS_OK - The case conversion succeeded
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, the malformed policy is invalid, type is not a valid case type, or locale is not a valid mjb_locale value
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a mapped codepoint
• MJB_STATUS_NO_MEMORY - Allocation failed

Example

mjb_text input = mjb_text_utf8("Stra\xC3\x9F""e", MJB_NUL_TERMINATED); // "Straße"
mjb_result result;

if(mjb_map_case(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, MJB_ENC_UTF_8,
    &result, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Upper: STRASSE
printf("Upper: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

mjb_text istanbul = mjb_text_utf8("istanbul", MJB_NUL_TERMINATED);

if(mjb_map_case(&istanbul, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_TR, MJB_ENC_UTF_8,
    &result, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Turkish upper: \xC4\xB0STANBUL
printf("Turkish upper: %.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

Related

• mjb_map_case_into
• mjb_locale_parse
• mjb_caseless_match

Specifications

• The Unicode Standard, Version 18.0.0, Section 3.13: Default Case Algorithms

Apply the same full, special, conditional, titlecase, locale-sensitive, and case folding mappings as mjb_map_case without allocating memory. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded from the byte count and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity.

Returns

• MJB_STATUS_OK - The required size was returned or the case-mapped string was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, the malformed policy is invalid, type is invalid, or locale is not a valid mjb_locale value
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_UNSUPPORTED - The requested output encoding cannot represent a mapped codepoint
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count

Example

mjb_text input = mjb_text_utf8("Stra\xC3\x9F""e", MJB_NUL_TERMINATED); // "Straße"
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);

// Query the required size
if(mjb_map_case_into(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

char storage[7];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_map_case_into(&input, MJB_MALFORMED_STOP, MJB_CASE_UPPER, MJB_LOCALE_NOT_SET, &output,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Upper payload (no terminator): STRASSE
printf("Upper payload (no terminator): %.*s", (int)output.size, storage);

Related

• mjb_map_case
• mjb_locale_parse

Specifications

• The Unicode Standard, Version 18.0.0, Section 3.13: Default Case Algorithms

Fill character with the Unicode Character Database record of a codepoint: name, category, combining class, bidirectional category, decomposition, numeric values, mirrored flag, and simple case mappings. When the library is compiled with MJB_FEATURE_CHARACTER_NAMES=0 the name field is reported as Codepoint U+XXXX.

Returns

• MJB_STATUS_OK - The character was found and filled
• MJB_STATUS_INVALID_ARGUMENT - character is NULL or the codepoint is not valid
• MJB_STATUS_NOT_FOUND - The codepoint is not assigned

Examples

mjb_character character;

// U+0665 ARABIC-INDIC DIGIT FIVE
if(mjb_codepoint_info(0x0665, &character) != MJB_STATUS_OK) {
    return 1;
}

bool is_digit = character.category == MJB_CATEGORY_ND;

// Decimal digit: yes, value: 5
printf("Decimal digit: %s, value: %d", is_digit ? "yes" : "no", character.decimal);

// U+0665 ARABIC-INDIC DIGIT FIVE
auto character = mjb::Character::from(0x0665);

if(!character) {
    return 1;
}

bool is_digit = character->category() == MJB_CATEGORY_ND;

// Decimal digit: yes, value: 5
std::cout << "Decimal digit: " << (is_digit ? "yes" : "no") << ", value: "
    << character->decimal_value().value_or(-1) << std::endl;

Related

• mjb_codepoint_block
• mjb_codepoint_script
• mjb_codepoint_property_binary
• mjb_codepoint_property_int

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Run the normalization quick-check on a string without allocating. MJB_QC_MAYBE means the string may still be normalized, and only a full normalization pass with mjb_normalize can decide.

Returns

• MJB_STATUS_OK - quick_check contains MJB_QC_YES, MJB_QC_NO, or MJB_QC_MAYBE
• MJB_STATUS_INVALID_ARGUMENT - quick_check or text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - The text encoding is invalid or a generic UTF-16/UTF-32 encoding has no byte-order information
• MJB_STATUS_INVALID_FORM - form is not NFC, NFD, NFKC, or NFKD
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence

Example

mjb_text input = mjb_text_utf8("caf\xC3\xA9", MJB_NUL_TERMINATED);
mjb_quick_check_result check;

if(mjb_normalization_quick_check(&input, MJB_NORMALIZATION_NFC, &check) != MJB_STATUS_OK) {
    return 1;
}

// NFC normalized: yes
printf("NFC normalized: %s", check == MJB_QC_YES ? "yes" : "no");

Related

• mjb_normalize

Specifications

• UAX #15: Unicode Normalization Forms, Unicode 18.0.0

mjb_detect_encoding reports BOM-derived UTF-16/UTF-32 schemes with the generic family bit plus the resolved endian bit. Passing that detected value consumes the leading BOM as a signature. Passing an explicit-endian encoding such as MJB_ENC_UTF_16BE preserves an initial U+FEFF as text. When flags overlap, as with a UTF-32LE BOM that also has the UTF-16LE BOM prefix, decoding gives UTF-32 precedence. MJB_NUL_TERMINATED is not accepted because the encoding, and therefore the NUL code-unit width, is unknown.

Example

const char utf16le[] = "\xFF\xFEH\0i\0";
mjb_encoding detected = mjb_detect_encoding(utf16le, sizeof(utf16le) - 1);
bool is_utf16le = detected == (MJB_ENC_UTF_16 | MJB_ENC_UTF_16LE);

// UTF-16LE detected: yes
printf("UTF-16LE detected: %s", is_utf16le ? "yes" : "no");

Examples

const char *input = "Plain ASCII";

// ASCII: yes
printf("ASCII: %s", mjb_is_ascii(input, strlen(input)) ? "yes" : "no");

std::string input = "Plain ASCII";

// ASCII: yes
std::cout << "ASCII: " << (mjb::is_ascii(input) ? "yes" : "no") << std::endl;

Examples

const char *input = "caf\xC3\xA9";

// Valid UTF-8: yes
printf("Valid UTF-8: %s", mjb_is_utf8(input, strlen(input)) ? "yes" : "no");

std::string input = "caf\xC3\xA9";

// Valid UTF-8: yes
std::cout << "Valid UTF-8: " << (mjb::is_utf8(input) ? "yes" : "no") << std::endl;

Examples

const char utf16be[] = "\xFE\xFF\0H\0i"; // BOM + "Hi" in UTF-16BE

// UTF-16: yes
printf("UTF-16: %s", mjb_is_utf16(utf16be, sizeof(utf16be) - 1) ? "yes" : "no");

using namespace std::string_view_literals;

// BOM + "Hi" in UTF-16BE. The sv suffix keeps the embedded NUL bytes.
auto utf16be = "\xFE\xFF\0H\0i"sv;

// UTF-16: yes
std::cout << "UTF-16: " << (mjb::is_utf16(utf16be) ? "yes" : "no") << std::endl;

Validate the complete input without producing output. Empty input is well-formed. On malformed input, diagnostic identifies the first maximal ill-formed subsequence. Generic UTF-16 and UTF-32 require a byte-order mark.

Returns

• MJB_STATUS_OK - The complete input is well-formed
• MJB_STATUS_INVALID_ARGUMENT - text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - The encoding is unsupported, or generic UTF-16/UTF-32 has no byte-order mark
• MJB_STATUS_MALFORMED_INPUT - The first malformed subsequence is described by diagnostic

Examples

const char invalid[] = "\xE2\x82";
mjb_text text = mjb_text_utf8(invalid, sizeof(invalid) - 1);
mjb_diagnostic diagnostic;

if(mjb_validate_encoding(&text, &diagnostic) != MJB_STATUS_MALFORMED_INPUT ||
    diagnostic.byte_offset != 0) {
    return 1;
}

std::string invalid = "\xE2\x82"; // A truncated three-byte sequence
mjb_diagnostic diagnostic{};

if(mjb::validate_encoding(invalid, &diagnostic) != MJB_STATUS_MALFORMED_INPUT) {
    return 1;
}

// Malformed input at byte 0
std::cout << "Malformed input at byte " << diagnostic.byte_offset << std::endl;

Related

• mjb_decode_next
• mjb_decode_previous
• mjb_is_utf8

Decode one codepoint and advance offset. With MJB_MALFORMED_STOP, malformed input returns MJB_STATUS_MALFORMED_INPUT; offset still advances over the malformed subsequence so decoding can resume. MJB_MALFORMED_REPLACE returns U+FFFD, while MJB_MALFORMED_SKIP advances until a valid codepoint or end of input. A diagnostic is reported for replacement and skipping even when the function returns success.

Returns

• MJB_STATUS_OK - A codepoint was decoded
• MJB_STATUS_END_OF_INPUT - No codepoint remains
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_INVALID_ARGUMENT - An argument or policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding cannot be decoded

Example

// "été"
mjb_text input = mjb_text_utf8("\xC3\xA9t\xC3\xA9", MJB_NUL_TERMINATED);
size_t offset = 0;
size_t count = 0;
mjb_codepoint codepoint;
mjb_status status;

while((status = mjb_decode_next(&input, MJB_MALFORMED_STOP, &offset, &codepoint, NULL)) ==
    MJB_STATUS_OK) {
    ++count;
}

if(status != MJB_STATUS_END_OF_INPUT) {
    return 1;
}

// 3 codepoints in 5 bytes
printf("%zu codepoints in %zu bytes", count, offset);

Related

• mjb_decode_previous
• mjb_validate_encoding
• mjb_codepoint_count

Decode backward from offset, using the same malformed-input policies and diagnostic contract as mjb_decode_next. On success, offset is the first byte of the decoded codepoint. Start with the input byte length to iterate from the end.

Returns

• MJB_STATUS_OK - A codepoint was decoded
• MJB_STATUS_END_OF_INPUT - No codepoint precedes offset
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_INVALID_ARGUMENT - An argument or policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding cannot be decoded

Example

// "été"
const char *word = "\xC3\xA9t\xC3\xA9";
mjb_text input = mjb_text_utf8(word, strlen(word));
size_t offset = strlen(word); // Start at the end
mjb_codepoint codepoint;

if(mjb_decode_previous(&input, MJB_MALFORMED_STOP, &offset, &codepoint,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Last codepoint: U+00E9 at byte 3
printf("Last codepoint: U+%04X at byte %zu", (unsigned)codepoint, offset);

Related

• mjb_decode_next
• mjb_validate_encoding
• mjb_codepoint_count

Count the number of decoded Unicode codepoints in a string. Malformed subsequences are stopped, replaced, or skipped according to malformed_policy. A replacement counts as one codepoint. On failure, count is set to zero.

Returns

• MJB_STATUS_OK - The count was computed
• MJB_STATUS_INVALID_ARGUMENT - count or text is NULL, text->buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - The text encoding is not a supported input encoding
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP

Example

// The "Héllö" string is five Unicode characters, but has different byte lengths in different encodings.

mjb_text utf8 = mjb_text_utf8("H\xC3\xA9ll\xC3\xB6", 7); // 7 bytes
mjb_text utf16le = mjb_text_utf16le("H\0\xE9\0l\0l\0\xF6\0", 10); // 10 bytes
size_t count;

if(mjb_codepoint_count(&utf8, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
    return 1;
}

// 5 UTF-8 characters
printf("%zu UTF-8 characters", count);

if(mjb_codepoint_count(&utf16le, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
    return 1;
}

// 5 UTF-16LE characters
printf("%zu UTF-16LE characters", count);

Related

• mjb_grapheme_count
• mjb_word_count
• mjb_sentence_count

Decode the string according to malformed_policy and call the callback for every resulting codepoint, with its position in the string and context. Use context to give the callback its own state, such as a counter or an output buffer, without global variables. The first malformed subsequence is reported in diagnostic even when it is replaced or skipped.

Returns

• MJB_STATUS_OK - Every decoded codepoint was visited
• MJB_STATUS_INVALID_ARGUMENT - The text, callback, or malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_CALLBACK_STOPPED - The callback returned false

Example

mjb_text input = mjb_text_utf8("ABC", 3);
mjb_status status = mjb_for_each_codepoint(&input, MJB_MALFORMED_STOP, NULL, NULL, NULL);

// A callback is required: yes
bool callback_required = status == MJB_STATUS_INVALID_ARGUMENT;

// A callback is required: yes
printf("A callback is required: %s", callback_required ? "yes" : "no");

Return true when the codepoint has the binary property and false when it does not. Passing an enumerated property is a type mismatch and returns MJB_STATUS_INVALID_ARGUMENT.

Example

bool is_alphabetic;

if(mjb_codepoint_property_binary('A', MJB_PR_ALPHABETIC,
    &is_alphabetic) != MJB_STATUS_OK) {
    return 1;
}

// U+0041 is alphabetic: yes
printf("U+0041 is alphabetic: %s", is_alphabetic ? "yes" : "no");

Related

• mjb_codepoint_property_int

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Passing a binary property is a type mismatch and returns MJB_STATUS_INVALID_ARGUMENT. MJB_STATUS_NOT_FOUND means that the codepoint has no stored value for the requested property.

Example

int32_t script;

if(mjb_codepoint_property_int('A', MJB_PR_SCRIPT, &script) != MJB_STATUS_OK) {
    return 1;
}

// U+0041 uses the Latin script: yes
printf("U+0041 uses the Latin script: %s", script == MJB_SC_LATN ? "yes" : "no");

Related

• mjb_codepoint_property_binary

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Return the numeric value of a codepoint, if any. If the codepoint has no numeric value, value->decimal and value->digit are set to MJB_NUMBER_NOT_VALID (-1).

Returns

• MJB_STATUS_OK - The character was found and filled
• MJB_STATUS_INVALID_ARGUMENT - value is NULL or the codepoint is not valid

Example

mjb_numeric_value num;

if(mjb_codepoint_numeric_value(0x0031, &num) != MJB_STATUS_OK) { // U+0031 = 1
    return 1;
}

// decimal=1, digit=1, numeric=1
printf("decimal=%d, digit=%d, numeric=%s", num.decimal, num.digit, num.numeric);

if(mjb_codepoint_numeric_value(0x00BD, &num) != MJB_STATUS_OK) { // U+00BD = '½'
    return 1;
}

// decimal=-1, digit=-1, numeric=1/2
printf("decimal=%d, digit=%d, numeric=%s", num.decimal, num.digit, num.numeric);

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Examples

mjb_block_info block;

if(mjb_codepoint_block('A', &block) != MJB_STATUS_OK) {
    return 1;
}

// Block: Basic Latin
printf("Block: %s", block.name);

auto block = mjb::block_info('A');

if(!block) {
    return 1;
}

// Block: Basic Latin
std::cout << "Block: " << block->name() << std::endl;

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Examples

mjb_script script = mjb_codepoint_script(0x03A9); // Greek capital omega

// Greek script: yes
printf("Greek script: %s", script == MJB_SC_GREK ? "yes" : "no");

mjb_script script = mjb::script(0x03A9); // Greek capital omega

// Greek script: yes
std::cout << "Greek script: " << (script == MJB_SC_GREK ? "yes" : "no") << std::endl;

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Return the explicit Script_Extensions set, or the ordinary Script value when the codepoint has no explicit Script_Extensions entry. Call first with scripts set to NULL to obtain the required count.

Examples

size_t count = 0;

if(mjb_codepoint_script_extensions(0x30FC, NULL, &count) != MJB_STATUS_OK) {
    return 1;
}

mjb_script scripts[3];

if(count > 3 || mjb_codepoint_script_extensions(0x30FC, scripts,
    &count) != MJB_STATUS_OK) {
    return 1;
}

// U+30FC has 2 Script_Extensions
printf("U+30FC has %zu Script_Extensions", count);

// Throws mjb::LibraryError on failure
std::vector<mjb_script> scripts = mjb::script_extensions(0x30FC);

// U+30FC has 2 Script_Extensions
std::cout << "U+30FC has " << scripts.size() << " Script_Extensions" << std::endl;

Related

• mjb_codepoint_script

Specifications

• UAX #24: Unicode Script Property, Unicode 18.0.0

Examples

// U+10FFFD valid: yes
printf("U+10FFFD valid: %s", mjb_codepoint_is_valid(0x10FFFD) ? "yes" : "no");

// U+10FFFD valid: yes
std::cout << "U+10FFFD valid: " << (mjb::is_valid(0x10FFFD) ? "yes" : "no") << std::endl;

Examples

// Letter A is graphic: yes
printf("Letter A is graphic: %s", mjb_codepoint_is_graphic('A') ? "yes" : "no");

// Letter A is graphic: yes
std::cout << "Letter A is graphic: " << (mjb::is_graphic('A') ? "yes" : "no") << std::endl;

Examples

// U+0301 is combining: yes
printf("U+0301 is combining: %s", mjb_codepoint_is_combining(0x0301) ? "yes" : "no");

// U+0301 is combining: yes
std::cout << "U+0301 is combining: " << (mjb::is_combining(0x0301) ? "yes" : "no") << std::endl;

Examples

// U+4E00 is a CJK ideograph: yes
printf("U+4E00 is a CJK ideograph: %s", mjb_codepoint_is_cjk_ideograph(0x4E00) ? "yes" : "no");

// U+4E00 is a CJK ideograph: yes
std::cout << "U+4E00 is a CJK ideograph: "
    << (mjb::is_cjk_ideograph(0x4E00) ? "yes" : "no") << std::endl;

Examples

// U+20000 is a CJK extension ideograph: yes
printf("U+20000 is a CJK extension ideograph: %s", mjb_codepoint_is_cjk_extension_ideograph(0x20000) ? "yes" : "no");

// U+20000 is a CJK extension ideograph: yes
std::cout << "U+20000 is a CJK extension ideograph: "
    << (mjb::is_cjk_extension_ideograph(0x20000) ? "yes" : "no") << std::endl;

Example

// Uppercase letters are graphic: yes
bool graphic = mjb_category_is_graphic(MJB_CATEGORY_LU);

// Uppercase letters are graphic: yes
printf("Uppercase letters are graphic: %s", graphic ? "yes" : "no");

Example

// Nonspacing marks are combining: yes
bool combining = mjb_category_is_combining(MJB_CATEGORY_MN);

// Nonspacing marks are combining: yes
printf("Nonspacing marks are combining: %s", combining ? "yes" : "no");

Examples

// Greek alpha starts an identifier: yes
bool starts = mjb_codepoint_is_id_start(0x03B1);

// Greek alpha starts an identifier: yes
printf("Greek alpha starts an identifier: %s", starts ? "yes" : "no");

// Greek alpha starts an identifier: yes
std::cout << "Greek alpha starts an identifier: "
    << (mjb::is_id_start(0x03B1) ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Digit 7 continues an identifier: yes
bool continues = mjb_codepoint_is_id_continue('7');

// Digit 7 continues an identifier: yes
printf("Digit 7 continues an identifier: %s", continues ? "yes" : "no");

// Digit 7 continues an identifier: yes
std::cout << "Digit 7 continues an identifier: "
    << (mjb::is_id_continue('7') ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Letter A is XID_Start: yes
printf("Letter A is XID_Start: %s", mjb_codepoint_is_xid_start('A') ? "yes" : "no");

// Letter A is XID_Start: yes
std::cout << "Letter A is XID_Start: " << (mjb::is_xid_start('A') ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Underscore is XID_Continue: yes
bool continues = mjb_codepoint_is_xid_continue('_');

// Underscore is XID_Continue: yes
printf("Underscore is XID_Continue: %s", continues ? "yes" : "no");

// Underscore is XID_Continue: yes
std::cout << "Underscore is XID_Continue: "
    << (mjb::is_xid_continue('_') ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Plus sign is Pattern_Syntax: yes
bool syntax = mjb_codepoint_is_pattern_syntax('+');

// Plus sign is Pattern_Syntax: yes
printf("Plus sign is Pattern_Syntax: %s", syntax ? "yes" : "no");

// Plus sign is Pattern_Syntax: yes
std::cout << "Plus sign is Pattern_Syntax: "
    << (mjb::is_pattern_syntax('+') ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Space is Pattern_White_Space: yes
bool whitespace = mjb_codepoint_is_pattern_white_space(' ');

// Space is Pattern_White_Space: yes
printf("Space is Pattern_White_Space: %s", whitespace ? "yes" : "no");

// Space is Pattern_White_Space: yes
std::cout << "Space is Pattern_White_Space: "
    << (mjb::is_pattern_white_space(' ') ? "yes" : "no") << std::endl;

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Examples

// Red heart is Extended_Pictographic: yes
bool pictographic = mjb_codepoint_is_extended_pictographic(0x2764);

// Red heart is Extended_Pictographic: yes
printf("Red heart is Extended_Pictographic: %s", pictographic ? "yes" : "no");

// Red heart is Extended_Pictographic: yes
std::cout << "Red heart is Extended_Pictographic: "
    << (mjb::is_extended_pictographic(0x2764) ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Example

mjb_plane plane = mjb_codepoint_plane(0x1F600);

// U+1F600 is in the SMP: yes
printf("U+1F600 is in the SMP: %s", plane == MJB_PLANE_SMP ? "yes" : "no");

Example

// Plane 16 is valid: yes
printf("Plane 16 is valid: %s", mjb_plane_is_valid(MJB_PLANE_PUA_B) ? "yes" : "no");

Example

// Plane: Basic Multilingual Plane
printf("Plane: %s", mjb_plane_name(MJB_PLANE_BMP, false));

Return the four-letter ISO 15924 code that names script, such as Latn for MJB_SC_LATN. MJB_SC_NOT_SET yields an empty string.

Returns

• const char * - The ISO 15924 code, or NULL if script is not a valid mjb_script value

Example

// Script code: Latn
printf("Script code: %s", mjb_script_code(MJB_SC_LATN));

Related

• mjb_codepoint_script
• mjb_codepoint_script_extensions

Compare two strings for case-insensitive equality using Unicode full default case folding. Canonical mode implements D145 and is recommended for ordinary text. Unnormalized mode implements D144 without normalization. Compatibility mode implements the iterated folding and compatibility normalization required by D146. Identifier mode implements D147 by applying NFD before mjb_nfkc_casefold; it removes default-ignorable codepoints but does not validate identifier syntax. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not examined and second_diagnostic stays clear.

Returns

• MJB_STATUS_OK - matches contains the comparison result
• MJB_STATUS_INVALID_ARGUMENT - matches, first, or second is NULL, a text buffer is NULL with a non-zero size, mode is invalid, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An input encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - An intermediate size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_UNSUPPORTED - The identifier case-folding transform did not converge

Example

mjb_text left = mjb_text_utf8("Stra\xC3\x9F" "e", MJB_NUL_TERMINATED);
mjb_text right = mjb_text_utf8("STRASSE", MJB_NUL_TERMINATED);
bool matches;

if(mjb_caseless_match(&left, &right, MJB_MALFORMED_STOP, MJB_CASELESS_CANONICAL, &matches, NULL,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Canonical caseless match: yes
printf("Canonical caseless match: %s", matches ? "yes" : "no");

Related

• mjb_map_case
• mjb_nfkc_casefold
• mjb_normalize
• mjb_collation_compare
• mjb_is_identifier

Specifications

• The Unicode Standard, Version 18.0.0, Section 3.13.5: Default Caseless Matching
• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Compare two strings using the Unicode Collation Algorithm and the default collation element table (DUCET), with strcmp-style semantics. Primary strength compares base-character weights. Secondary strength also compares accents while ignoring tertiary case differences. Tertiary strength adds case and variant differences. Quaternary strength also compares variable elements moved to level 4 by MJB_COLLATION_SHIFTED; with MJB_COLLATION_NON_IGNORABLE, it is equivalent to tertiary strength. Collation equality at a selected strength is not Unicode caseless matching; use mjb_caseless_match when case-insensitive equality is the intended operation. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not read and second_diagnostic stays clear. See [Unicode collation](#unicode-collation) for the comparison process and configuration guidance. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - order is negative, zero, or positive according to the collation order
• MJB_STATUS_INVALID_ARGUMENT - order, first, or second is NULL, a text buffer is NULL with a non-zero size, the malformed policy is invalid, or an option is invalid
• MJB_STATUS_INVALID_ENCODING - An input encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - An intermediate size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_COLLATION=0

Example

mjb_text apple = mjb_text_utf8("apple", 5);
mjb_text banana = mjb_text_utf8("banana", 6);
int order;

if(mjb_collation_compare(&apple, &banana, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
    MJB_COLLATION_TERTIARY, &order, NULL, NULL) != MJB_STATUS_OK) {
    return 1;
}

// apple sorts before banana: yes
printf("apple sorts before banana: %s", order < 0 ? "yes" : "no");

Related

• mjb_collation_key
• mjb_collation_key_into
• mjb_caseless_match

Specifications

• UTS #10: Unicode Collation Algorithm, Unicode 18.0.0

Generate a binary sort key for a string. Sort keys of different strings can be compared with memcmp and yield the same order as mjb_collation_compare when both use the same variable weighting and strength. Useful when the same strings are compared many times, such as sorting or database indexing. Empty input and non-empty input with no effective weights at the selected strength both produce a zero-length key. Malformed subsequences follow malformed_policy, and diagnostic records the first one. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The sort key was generated
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, text->buffer is NULL with a non-zero size, the malformed policy is invalid, or a collation option is invalid
• MJB_STATUS_INVALID_ENCODING - The input encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - The sort key size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_COLLATION=0

Example

mjb_text input = mjb_text_utf8("r\xC3\xA9sum\xC3\xA9", 8);
mjb_result key;

if(mjb_collation_key(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
    MJB_COLLATION_TERTIARY, &key, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Sort key is non-empty: yes
printf("Sort key is non-empty: %s", key.output_size > 0 ? "yes" : "no");
mjb_result_free(&key);

Related

• mjb_collation_key_into
• mjb_collation_compare

Specifications

• UTS #10: Unicode Collation Algorithm, Unicode 18.0.0

Generate the same binary sort key as mjb_collation_key without allocating the final key buffer. Set output->data to NULL to query the required byte count. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. A collation key is binary: output->encoding is ignored, and no terminator is included or written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the key, but nothing is written past its capacity. Collation processing still uses temporary allocations, including during a size query. Empty input and non-empty input with no effective weights at the selected strength both require zero bytes. If MJB_FEATURE_COLLATION=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The required size was returned or the binary sort key was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, text->buffer is NULL with a non-zero size, the malformed policy is invalid, or a collation option is invalid
• MJB_STATUS_INVALID_ENCODING - The input encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - The required key size would overflow
• MJB_STATUS_NO_MEMORY - Temporary allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_COLLATION=0

Example

mjb_text input = mjb_text_utf8("r\xC3\xA9sum\xC3\xA9", 8);
// A collation key is binary, so the buffer encoding is ignored
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UNKNOWN);

// Query the required size
if(mjb_collation_key_into(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
    MJB_COLLATION_TERTIARY, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

unsigned char storage[64];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UNKNOWN);

if(mjb_collation_key_into(&input, MJB_MALFORMED_STOP, MJB_COLLATION_NON_IGNORABLE,
    MJB_COLLATION_TERTIARY, &output, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Sort key is non-empty: yes
printf("Sort key is non-empty: %s", output.size > 0 ? "yes" : "no");

Related

• mjb_collation_key
• mjb_collation_compare

Specifications

• UTS #10: Unicode Collation Algorithm, Unicode 18.0.0

Use the C library formatting rules and return semantics of snprintf. If the destination buffer truncates a well-formed UTF-8 result, any incomplete trailing codepoint is removed before the terminating NULL. Truncation is at a codepoint boundary, not a grapheme-cluster boundary.

Returns

• A nonnegative value - The number of bytes the complete result requires, excluding the terminating NULL
• A negative value - The underlying vsnprintf reported an encoding error

Example

char buffer[4];
int required = mjb_utf8_snprintf(buffer, sizeof(buffer), "%s",
    "\xC3\xA9\xC3\xA9"); // éé

// 4: é
printf("%d: %s", required, buffer);

Related

• mjb_utf8_vsnprintf
• mjb_utf8_grapheme_snprintf
• mjb_is_utf8

The va_list counterpart of mjb_utf8_snprintf. It has the same UTF-8 input requirements, clipping behavior, and return semantics as mjb_utf8_snprintf.

Returns

• A nonnegative value - The number of bytes the complete result requires, excluding the terminating NULL
• A negative value - The underlying vsnprintf reported an encoding error

Example

// A printf-style function of your own that never ends with a partial codepoint
static int format_utf8(char *buffer, size_t buffer_size, const char *format, ...) {
    va_list args;
    va_start(args, format);
    int required = mjb_utf8_vsnprintf(buffer, buffer_size, format, args);
    va_end(args);

    return required;
}

char buffer[4];
int required = format_utf8(buffer, sizeof(buffer), "%s", "\xC3\xA9\xC3\xA9"); // éé

// 4: é
printf("%d: %s", required, buffer);

Related

• mjb_utf8_snprintf
• mjb_utf8_grapheme_vsnprintf
• mjb_is_utf8

Use the C library formatting rules and return semantics of snprintf. If the destination buffer truncates a well-formed UTF-8 result, the output is shortened to the largest prefix that ends at an extended grapheme-cluster boundary in the complete result. Unlike mjb_utf8_snprintf, truncation can require temporary allocation and a second evaluation of the format. Do not use %n or arguments whose values can change as a formatting side effect. On allocation failure the destination is set to an empty string, the function returns a negative value, and errno is set to ENOMEM.

Returns

• A nonnegative value - The number of bytes the complete result requires, excluding the terminating NULL
• A negative value - Formatting failed or the complete result could not be obtained

Example

char buffer[4];
int required = mjb_utf8_grapheme_snprintf(buffer, sizeof(buffer), "%s",
    "Ae\xCC\x81" "B"); // A, e + combining acute accent, B

// 5: A
printf("%d: %s", required, buffer);

Related

• mjb_utf8_grapheme_vsnprintf
• mjb_utf8_snprintf
• mjb_grapheme_prefix_length

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

The va_list counterpart of mjb_utf8_grapheme_snprintf. It has the same grapheme-safe clipping behavior, allocation requirements, and return semantics as mjb_utf8_grapheme_snprintf.

Returns

• A nonnegative value - The number of bytes the complete result requires, excluding the terminating NULL
• A negative value - Formatting failed or the complete result could not be obtained

Example

// A printf-style function of your own that never splits a grapheme cluster
static int format_graphemes(char *buffer, size_t buffer_size, const char *format, ...) {
    va_list args;
    va_start(args, format);
    int required = mjb_utf8_grapheme_vsnprintf(buffer, buffer_size, format, args);
    va_end(args);

    return required;
}

char buffer[4];
int required = format_graphemes(buffer, sizeof(buffer), "%s",
    "Ae\xCC\x81" "B"); // A, e + combining acute accent, B

// 5: A
printf("%d: %s", required, buffer);

Related

• mjb_utf8_grapheme_snprintf
• mjb_utf8_vsnprintf
• mjb_grapheme_prefix_length

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Validate a string as a Unicode identifier: the first character must be a valid identifier start and the following ones valid identifier continuations, using ID_Start/ID_Continue for the DEFAULT profile or XID_Start/XID_Continue for the NFKC profile.

Example

mjb_text identifier = mjb_text_utf8("delta_2", MJB_NUL_TERMINATED);

bool valid = mjb_is_identifier(&identifier, MJB_IDENTIFIER_NFKC);

// Valid identifier: yes
printf("Valid identifier: %s", valid ? "yes" : "no");

Related

• mjb_codepoint_is_id_start
• mjb_codepoint_is_id_continue
• mjb_codepoint_is_xid_start
• mjb_codepoint_is_xid_continue

Specifications

• UAX #31: Unicode Identifiers and Syntax, Unicode 18.0.0

Intersect the augmented Script_Extensions sets of every codepoint as specified by UTS #39. Common and Inherited resolve to ALL. Han, Hiragana, Katakana, Hangul, and Bopomofo are augmented with the Hanb, Jpan, and Kore writing-system values. Set scripts to NULL to query the required count. A mixed-script string returns EMPTY with a zero count; an empty string or a string containing only Common or Inherited characters returns ALL with a zero count. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The required count and set kind were returned, or the resolved scripts were written
• MJB_STATUS_INVALID_ARGUMENT - count, kind, or text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - The input encoding is invalid or lacks required byte-order information
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_OUTPUT_TOO_SMALL - The script buffer capacity is smaller than the required count
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_SECURITY=0

Example

mjb_text input = mjb_text_utf8("\xE3\x81\xAD\xE3\x82\xAC", MJB_NUL_TERMINATED); // Hiragana + Katakana
mjb_script scripts[1];
size_t count = 1;
mjb_script_set_kind kind;

if(mjb_resolved_script_set(&input, scripts, &count, &kind) != MJB_STATUS_OK) {
    return 1;
}

bool japanese = kind == MJB_SCRIPT_SET_RESOLVED && count == 1 && scripts[0] == MJB_SC_JPAN;

// Japanese writing system: yes
printf("Japanese writing system: %s", japanese ? "yes" : "no");

Related

• mjb_codepoint_script_extensions
• mjb_is_identifier
• mjb_confusable_skeleton
• mjb_confusable_match

Specifications

• UTS #39: Unicode Security Mechanisms, Unicode 18.0.0

Compute the UTS #39 bidiSkeleton(LTR, input): apply the Unicode Bidirectional Algorithm through L4, then NFD, remove default-ignorables, substitute prototypes from confusables.txt, and reapply NFD. Skeletons can be stored or indexed so future confusable checks can compare them directly. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The confusable skeleton was returned
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_UNSUPPORTED - The output encoding cannot represent a skeleton codepoint
• MJB_STATUS_OVERFLOW - The output size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_SECURITY=0

Example

// Cyrillic U+0435 in place of e
mjb_text input = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED);
mjb_result result;

if(mjb_confusable_skeleton(&input, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) {
    return 1;
}

// hello
printf("%.*s", (int)result.output_size, result.output);
mjb_result_free(&result);

Related

• mjb_confusable_skeleton_into
• mjb_confusable_match
• mjb_is_identifier

Specifications

• UTS #39: Unicode Security Mechanisms, Unicode 18.0.0

Compute the same UTS #39 bidiSkeleton(LTR, input) as mjb_confusable_skeleton without allocating the final output buffer. output->encoding selects the output encoding. Set output->data to NULL to query the required size. If output->data is non-NULL, output->size supplies its capacity; on return it contains the required size when the buffer is too small, or the written size on success. Terminators are excluded and are not written. If capacity is insufficient, or another error occurs, the output buffer may hold a partial prefix of the result, but nothing is written past its capacity. Bidirectional resolution, normalization, and skeleton mapping still require temporary allocations, including during a size query. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED.

Returns

• MJB_STATUS_OK - The required size was returned or the skeleton was written
• MJB_STATUS_INVALID_ARGUMENT - output or text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - An encoding is invalid
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_UNSUPPORTED - The output encoding cannot represent a skeleton codepoint
• MJB_STATUS_OVERFLOW - The required output size would overflow
• MJB_STATUS_NO_MEMORY - Temporary allocation failed
• MJB_STATUS_OUTPUT_TOO_SMALL - The output capacity is smaller than the required byte count
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_SECURITY=0

Example

// Cyrillic U+0435 in place of e
mjb_text input = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED);
mjb_buffer output = mjb_buffer_make(NULL, 0, MJB_ENC_UTF_8);

// Query the required size
if(mjb_confusable_skeleton_into(&input, &output) != MJB_STATUS_OK) {
    return 1;
}

char storage[5];

if(output.size > sizeof(storage)) {
    return 1;
}

output = mjb_buffer_make(storage, sizeof(storage), MJB_ENC_UTF_8);

if(mjb_confusable_skeleton_into(&input, &output) != MJB_STATUS_OK) {
    return 1;
}

// Skeleton payload (no terminator): hello
printf("Skeleton payload (no terminator): %.*s", (int)output.size, storage);

Related

• mjb_confusable_skeleton
• mjb_confusable_match
• mjb_is_identifier

Specifications

• UTS #39: Unicode Security Mechanisms, Unicode 18.0.0

Compute the confusable skeleton of both strings and store true when the skeletons are equal, meaning the two strings are visually confusable, such as "good" and "gооd" with Cyrillic о. If MJB_FEATURE_SECURITY=0 the function always returns MJB_STATUS_FEATURE_NOT_ENABLED. Malformed subsequences follow malformed_policy. first_diagnostic and second_diagnostic each record the first malformed subsequence of their own text, with offsets into that text; when first stops on malformed input, second is not examined and second_diagnostic stays clear.

Returns

• MJB_STATUS_OK - confusable contains the comparison result
• MJB_STATUS_INVALID_ARGUMENT - confusable, first, or second is NULL, a text buffer is NULL with a non-zero size, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - An input encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP
• MJB_STATUS_OVERFLOW - An intermediate size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed
• MJB_STATUS_FEATURE_NOT_ENABLED - The library was compiled with MJB_FEATURE_SECURITY=0

Example

mjb_text latin = mjb_text_utf8("hello", MJB_NUL_TERMINATED);
mjb_text mixed = mjb_text_utf8("h\xD0\xB5llo", MJB_NUL_TERMINATED); // Cyrillic е
bool confusable;

if(mjb_confusable_match(&latin, &mixed, MJB_MALFORMED_STOP, &confusable, NULL,
    NULL) != MJB_STATUS_OK) {
    return 1;
}

// Visually confusable: yes
printf("Visually confusable: %s", confusable ? "yes" : "no");

Related

• mjb_confusable_skeleton
• mjb_confusable_skeleton_into
• mjb_is_identifier

Specifications

• UTS #39: Unicode Security Mechanisms, Unicode 18.0.0

Iterate the line break opportunities of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it. Stateful break functions require an explicit text->byte_length and do not accept MJB_NUL_TERMINATED; determine the length once before iteration.

Example

mjb_text input = mjb_text_utf8("Hello world", 11);
mjb_line_break_state state;
state.index = 0;
mjb_break_type type = mjb_line_break_next(&input, &state);

// First line-break result is set: yes
printf("First line-break result is set: %s", type != MJB_BT_NOT_SET ? "yes" : "no");

Related

• mjb_grapheme_break_next
• mjb_word_break_next
• mjb_sentence_break_next

Specifications

• UAX #14: Unicode Line Breaking Algorithm, Unicode 18.0.0

Iterate the word boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it. Stateful break functions require an explicit text->byte_length and do not accept MJB_NUL_TERMINATED; determine the length once before iteration.

Example

mjb_text input = mjb_text_utf8("Hello world", 11);
mjb_word_break_state state;
state.index = 0;
size_t boundaries = 0;

while(mjb_word_break_next(&input, &state) != MJB_BT_NOT_SET) {
    ++boundaries;
}

// Word-break positions: 11
printf("Word-break positions: %zu", boundaries);

Related

• mjb_grapheme_break_next
• mjb_sentence_break_next
• mjb_word_prefix_length

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Iterate the sentence boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it. Stateful break functions require an explicit text->byte_length and do not accept MJB_NUL_TERMINATED; determine the length once before iteration.

Example

mjb_sentence_break_state state;
state.index = 0;
size_t boundaries = 0;
mjb_text input = mjb_text_utf8("Hello. Goodbye.", 15);

while(mjb_sentence_break_next(&input, &state) != MJB_BT_NOT_SET) {
    ++boundaries;
}

// Sentence-break positions: 15
printf("Sentence-break positions: %zu", boundaries);

Related

• mjb_grapheme_break_next
• mjb_word_break_next

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Count the sentence segments produced by the default Unicode sentence-boundary rules. The default rules carry no abbreviation list, so text such as Dr. Smith counts as two sentences. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.

Returns

• MJB_STATUS_OK - The count was computed
• MJB_STATUS_INVALID_ARGUMENT - count is NULL, the text is invalid, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding is not a supported input encoding
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP

Example

mjb_text input = mjb_text_utf8("Hello. How are you? Fine!", MJB_NUL_TERMINATED);
size_t count;

if(mjb_sentence_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Sentences: 3
printf("Sentences: %zu", count);

Related

• mjb_sentence_break_next
• mjb_grapheme_count
• mjb_word_count

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Iterate the grapheme cluster (user-perceived character) boundaries of a string. Start with state.index set to 0 and call repeatedly with the same state until it returns MJB_BT_NOT_SET. Each call returns the break type of one position, from the boundary after the first codepoint to the end of the text, and stores its byte offset in state.position, so each segment runs from one break position to the next. A malformed subsequence counts as one position, like U+FFFD, but an incomplete subsequence at the end of the text ends the iteration at the position before it. Stateful break functions require an explicit text->byte_length and do not accept MJB_NUL_TERMINATED; determine the length once before iteration.

Example

mjb_text input = mjb_text_utf8("e\xCC\x81", 3); // e + combining acute accent
mjb_grapheme_break_state state;
state.index = 0;
size_t codepoints = 0;

while(mjb_grapheme_break_next(&input, &state) != MJB_BT_NOT_SET) {
    ++codepoints;
}

// Codepoints examined: 2
printf("Codepoints examined: %zu", codepoints);

Related

• mjb_word_break_next
• mjb_sentence_break_next
• mjb_line_break_next
• mjb_grapheme_prefix_length

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Example

mjb_text input = mjb_text_utf8("A\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9Z", MJB_NUL_TERMINATED); // A🇮🇹Z
size_t bytes = mjb_grapheme_prefix_length(&input, 2);

// First two graphemes use 9 bytes
printf("First two graphemes use %zu bytes", bytes);

Count user-perceived characters: the number of extended grapheme cluster segments in the string. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.

Returns

• MJB_STATUS_OK - The count was computed
• MJB_STATUS_INVALID_ARGUMENT - count is NULL, the text is invalid, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding is not a supported input encoding
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP

Example

mjb_text input = mjb_text_utf8("A\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // A🇮🇹
size_t count;

if(mjb_grapheme_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Grapheme clusters: 2
printf("Grapheme clusters: %zu", count);

Related

• mjb_grapheme_prefix_length
• mjb_grapheme_break_next
• mjb_terminal_width

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Example

mjb_text input = mjb_text_utf8("Hello world", MJB_NUL_TERMINATED);
size_t bytes = mjb_word_prefix_length(&input, 1);

// First word segment uses 5 bytes
printf("First word segment uses %zu bytes", bytes);

Count the words in a string: the word-break segments that contain at least one alphabetic or numeric character. Punctuation, whitespace, and symbol segments are not counted, so Hello, world! counts as two words. Hyphenated compounds count each part, matching the default Unicode word-boundary rules. Unlike mjb_word_prefix_length, whose max_segments counts every raw segment, this function skips non-word segments. For scripts segmented by dictionary lookup in other implementations, such as Chinese, Japanese, Thai, Lao, Khmer, and Burmese, the count approximates one word per character: Mojibake does not use frequency dictionaries to keep the size of the library small. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, count is set to zero.

Returns

• MJB_STATUS_OK - The count was computed
• MJB_STATUS_INVALID_ARGUMENT - count is NULL, the text is invalid, or the malformed policy is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding is not a supported input encoding
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP

Example

mjb_text input = mjb_text_utf8("Hello, world! It works.", MJB_NUL_TERMINATED);
size_t count;

if(mjb_word_count(&input, MJB_MALFORMED_STOP, &count, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Words: 4
printf("Words: %zu", count);

Related

• mjb_word_break_next
• mjb_word_prefix_length
• mjb_grapheme_count
• mjb_sentence_count

Specifications

• UAX #29: Unicode Text Segmentation, Unicode 18.0.0

Resolve the embedding levels of a paragraph following the Unicode Bidirectional Algorithm. The resolved paragraph can then be split into lines and reordered visually with mjb_bidi_reorder_line and mjb_bidi_line_runs.

Returns

• MJB_STATUS_OK - The paragraph was resolved
• MJB_STATUS_INVALID_ARGUMENT - result or text is NULL, or text->buffer is NULL with a non-zero size
• MJB_STATUS_INVALID_ENCODING - The input encoding is invalid or lacks byte-order information
• MJB_STATUS_MALFORMED_INPUT - The input contains an ill-formed code-unit sequence
• MJB_STATUS_OVERFLOW - The paragraph size would overflow
• MJB_STATUS_NO_MEMORY - Allocation failed

Example

mjb_text input = mjb_text_utf8("abc \xD7\x90\xD7\x91\xD7\x92", MJB_NUL_TERMINATED); // abc אבג
mjb_bidi_paragraph paragraph;

if(mjb_bidi_resolve(&input, MJB_DIRECTION_AUTO, &paragraph) != MJB_STATUS_OK) {
    return 1;
}

// Paragraph codepoints: 7
printf("Paragraph codepoints: %zu", paragraph.count);
mjb_bidi_paragraph_free(&paragraph);

Related

• mjb_bidi_paragraph_free
• mjb_bidi_reorder_line
• mjb_bidi_line_runs

Specifications

• UAX #9: Unicode Bidirectional Algorithm, Unicode 18.0.0

Example

mjb_text input = mjb_text_utf8("\xD7\x90\xD7\x91\xD7\x92", MJB_NUL_TERMINATED); // אבג
mjb_bidi_paragraph paragraph;
size_t visual_order[3];

if(mjb_bidi_resolve(&input, MJB_DIRECTION_AUTO, &paragraph) != MJB_STATUS_OK ||
    mjb_bidi_reorder_line(&paragraph, 0, paragraph.count,
        visual_order) != MJB_STATUS_OK) {
    return 1;
}

// First visual index: 2
printf("First visual index: %zu", visual_order[0]);
mjb_bidi_paragraph_free(&paragraph);

Related

• mjb_bidi_resolve
• mjb_bidi_line_runs

Specifications

• UAX #9: Unicode Bidirectional Algorithm, Unicode 18.0.0

Example

mjb_text input = mjb_text_utf8("abc", 3);
mjb_bidi_paragraph paragraph;
size_t visual_order[3];
size_t run_count = 0;

if(mjb_bidi_resolve(&input, MJB_DIRECTION_LTR, &paragraph) != MJB_STATUS_OK ||
    mjb_bidi_reorder_line(&paragraph, 0, 3, visual_order) != MJB_STATUS_OK ||
    mjb_bidi_line_runs(&paragraph, visual_order, 3, NULL,
        &run_count) != MJB_STATUS_OK) {
    return 1;
}

// Visual runs: 1
printf("Visual runs: %zu", run_count);
mjb_bidi_paragraph_free(&paragraph);

Related

• mjb_bidi_resolve
• mjb_bidi_reorder_line

Specifications

• UAX #9: Unicode Bidirectional Algorithm, Unicode 18.0.0

Example

mjb_text input = mjb_text_utf8("abc", 3);
mjb_bidi_paragraph paragraph;

if(mjb_bidi_resolve(&input, MJB_DIRECTION_LTR, &paragraph) != MJB_STATUS_OK) {
    return 1;
}

mjb_bidi_paragraph_free(&paragraph);

// Paragraph released: yes
printf("Paragraph released: %s", paragraph.chars == NULL ? "yes" : "no");

Related

• mjb_bidi_resolve

Example

mjb_emoji_properties emoji;

if(mjb_codepoint_emoji_properties(0x1F600, &emoji) != MJB_STATUS_OK) {
    return 1;
}

// U+1F600 has Emoji_Presentation: yes
printf("U+1F600 has Emoji_Presentation: %s", emoji.presentation ? "yes" : "no");

Related

• mjb_emoji_sequence_info
• mjb_codepoint_is_emoji

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// Number sign has the Emoji property: yes
bool emoji = mjb_codepoint_is_emoji('#');

// Number sign has the Emoji property: yes
printf("Number sign has the Emoji property: %s", emoji ? "yes" : "no");

// Number sign has the Emoji property: yes
std::cout << "Number sign has the Emoji property: "
    << (mjb::is_emoji('#') ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// Grinning face defaults to emoji presentation: yes
bool presentation = mjb_codepoint_is_emoji_presentation(0x1F600);

// Grinning face defaults to emoji presentation: yes
printf("Grinning face defaults to emoji presentation: %s", presentation ? "yes" : "no");

// Grinning face defaults to emoji presentation: yes
std::cout << "Grinning face defaults to emoji presentation: "
    << (mjb::is_emoji_presentation(0x1F600) ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// Medium skin tone is an emoji modifier: yes
bool modifier = mjb_codepoint_is_emoji_modifier(0x1F3FD);

// Medium skin tone is an emoji modifier: yes
printf("Medium skin tone is an emoji modifier: %s", modifier ? "yes" : "no");

// Medium skin tone is an emoji modifier: yes
std::cout << "Medium skin tone is an emoji modifier: "
    << (mjb::is_emoji_modifier(0x1F3FD) ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// Waving hand accepts an emoji modifier: yes
bool modifier_base = mjb_codepoint_is_emoji_modifier_base(0x1F44B);

// Waving hand accepts an emoji modifier: yes
printf("Waving hand accepts an emoji modifier: %s", modifier_base ? "yes" : "no");

// Waving hand accepts an emoji modifier: yes
std::cout << "Waving hand accepts an emoji modifier: "
    << (mjb::is_emoji_modifier_base(0x1F44B) ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// Zero-width joiner is an emoji component: yes
bool component = mjb_codepoint_is_emoji_component(0x200D);

// Zero-width joiner is an emoji component: yes
printf("Zero-width joiner is an emoji component: %s", component ? "yes" : "no");

// Zero-width joiner is an emoji component: yes
std::cout << "Zero-width joiner is an emoji component: "
    << (mjb::is_emoji_component(0x200D) ? "yes" : "no") << std::endl;

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Example

mjb_text flag = mjb_text_utf8("\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // 🇮🇹
mjb_emoji_sequence emoji;

if(mjb_emoji_sequence_info(&flag, &emoji) != MJB_STATUS_OK) {
    return 1;
}

// Sequence codepoints: 2
printf("Sequence codepoints: %zu", emoji.codepoint_count);

Related

• mjb_is_emoji_sequence
• mjb_is_rgi_emoji

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Example

mjb_text keycap = mjb_text_utf8("1\xEF\xB8\x8F\xE2\x83\xA3", MJB_NUL_TERMINATED); // 1️⃣

bool listed = mjb_is_emoji_sequence(&keycap);

// Listed emoji sequence: yes
printf("Listed emoji sequence: %s", listed ? "yes" : "no");

Related

• mjb_is_rgi_emoji
• mjb_emoji_sequence_info

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Example

mjb_text flag = mjb_text_utf8("\xF0\x9F\x87\xAE\xF0\x9F\x87\xB9", MJB_NUL_TERMINATED); // 🇮🇹

bool rgi = mjb_is_rgi_emoji(&flag);

// RGI emoji: yes
printf("RGI emoji: %s", rgi ? "yes" : "no");

Related

• mjb_is_emoji_sequence
• mjb_emoji_sequence_info

Specifications

• UTS #51: Unicode Emoji, Unicode 18.0.0

Truncate only at extended grapheme-cluster boundaries, using the same terminal-cell policy as mjb_terminal_width. The returned prefix stops before a cluster that would exceed the budget or cannot be measured as printable, single-line terminal text.

Example

mjb_text input = mjb_text_utf8("A\xE7\x95\x8C", MJB_NUL_TERMINATED); // A界
size_t bytes = mjb_grapheme_prefix_length_for_width(&input, MJB_TERMINAL_WIDTH_NARROW, 2);

// Two columns include 1 byte
printf("Two columns include %zu byte", bytes);

Truncate only at word-break boundaries, using the same terminal-cell policy as mjb_terminal_width. The returned prefix stops before a segment that would exceed the budget or cannot be measured as printable, single-line terminal text.

Example

mjb_text input = mjb_text_utf8("Hello world", MJB_NUL_TERMINATED);
size_t bytes = mjb_word_prefix_length_for_width(&input, MJB_TERMINAL_WIDTH_NARROW, 6);

// Six columns include 6 bytes
printf("Six columns include %zu bytes", bytes);

Example

mjb_east_asian_width width;

if(mjb_codepoint_east_asian_width(0x754C, &width) != MJB_STATUS_OK) { // 界
    return 1;
}

// U+754C is wide: yes
printf("U+754C is wide: %s", width == MJB_EAW_WIDE ? "yes" : "no");

Related

• mjb_terminal_width

Specifications

• UAX #11: East Asian Width, Unicode 18.0.0

Estimate the number of fixed terminal cells occupied by printable, single-line text. The input is normalized to NFC so canonically equivalent text has the same width. Combining and format characters occupy no additional cells, listed emoji-presentation sequences occupy two cells, and text-presentation sequences retain their East Asian Width. This is a deterministic terminal policy, not a measurement of proportional glyph advances. Use grapheme boundaries for cursor movement, selection, deletion, and user-perceived character counts. Controls, line separators, and paragraph separators are rejected because their effect depends on terminal state. Malformed subsequences follow malformed_policy, and diagnostic records the first one. On failure, width is set to zero.

Returns

• MJB_STATUS_OK - The width was computed
• MJB_STATUS_INVALID_ARGUMENT - width is NULL, the profile or malformed policy is invalid, or the text is invalid
• MJB_STATUS_INVALID_ENCODING - The encoding is invalid or lacks required byte-order information
• MJB_STATUS_UNSUPPORTED - The input contains a control, line separator, or paragraph separator
• MJB_STATUS_NO_MEMORY - NFC normalization could not allocate memory
• MJB_STATUS_OVERFLOW - The width would overflow
• MJB_STATUS_MALFORMED_INPUT - Malformed input was encountered with MJB_MALFORMED_STOP

Example

mjb_text input = mjb_text_utf8("A\xE7\x95\x8C", MJB_NUL_TERMINATED); // A界
size_t width;

if(mjb_terminal_width(&input, MJB_MALFORMED_STOP,
    MJB_TERMINAL_WIDTH_NARROW, &width, NULL) != MJB_STATUS_OK) {
    return 1;
}

// Terminal cells: 3
printf("Terminal cells: %zu", width);

Related

• mjb_codepoint_east_asian_width
• mjb_grapheme_prefix_length_for_width
• mjb_grapheme_break_next
• mjb_is_rgi_emoji

Specifications

• UAX #11: East Asian Width, Unicode 18.0.0
• UAX #29: Unicode Text Segmentation, Unicode 18.0.0
• UTS #51: Unicode Emoji, Unicode 18.0.0

Examples

// U+1100 is a leading Jamo: yes
printf("U+1100 is a leading Jamo: %s", mjb_codepoint_is_hangul_leading_jamo(0x1100) ? "yes" : "no");

// U+1100 is a leading Jamo: yes
std::cout << "U+1100 is a leading Jamo: "
    << (mjb::is_hangul_leading_jamo(0x1100) ? "yes" : "no") << std::endl;

Examples

// U+1161 is a vowel Jamo: yes
printf("U+1161 is a vowel Jamo: %s", mjb_codepoint_is_hangul_vowel_jamo(0x1161) ? "yes" : "no");

// U+1161 is a vowel Jamo: yes
std::cout << "U+1161 is a vowel Jamo: "
    << (mjb::is_hangul_vowel_jamo(0x1161) ? "yes" : "no") << std::endl;

Examples

// U+11A8 is a trailing Jamo: yes
printf("U+11A8 is a trailing Jamo: %s", mjb_codepoint_is_hangul_trailing_jamo(0x11A8) ? "yes" : "no");

// U+11A8 is a trailing Jamo: yes
std::cout << "U+11A8 is a trailing Jamo: "
    << (mjb::is_hangul_trailing_jamo(0x11A8) ? "yes" : "no") << std::endl;

Examples

// U+1100 is Hangul Jamo: yes
printf("U+1100 is Hangul Jamo: %s", mjb_codepoint_is_hangul_jamo(0x1100) ? "yes" : "no");

// U+1100 is Hangul Jamo: yes
std::cout << "U+1100 is Hangul Jamo: " << (mjb::is_hangul_jamo(0x1100) ? "yes" : "no") << std::endl;

Examples

// U+AC00 is a Hangul syllable: yes
printf("U+AC00 is a Hangul syllable: %s", mjb_codepoint_is_hangul_syllable(0xAC00) ? "yes" : "no");

// U+AC00 is a Hangul syllable: yes
std::cout << "U+AC00 is a Hangul syllable: "
    << (mjb::is_hangul_syllable(0xAC00) ? "yes" : "no") << std::endl;

Example

char name[32];

if(mjb_hangul_syllable_name(0xAC01, name, sizeof(name)) != MJB_STATUS_OK) {
    return 1;
}

// Name: HANGUL SYLLABLE GAG
printf("Name: %s", name);

Apply the arithmetic Hangul syllable decomposition of Unicode Section 3.12, the inverse of mjb_hangul_syllable_compose.

Returns

• MJB_STATUS_OK - The syllable was decomposed
• MJB_STATUS_INVALID_ARGUMENT - codepoints is NULL
• MJB_STATUS_NOT_FOUND - codepoint is not a Hangul syllable

Example

mjb_codepoint decomposition[3];

if(mjb_hangul_syllable_decompose(0xAC01, decomposition) != MJB_STATUS_OK) {
    return 1;
}

// Decomposition starts with: U+1100
printf("Decomposition starts with: U+%04X", decomposition[0]);

Related

• mjb_hangul_syllable_compose
• mjb_hangul_syllable_name

Apply the arithmetic Hangul syllable composition of Unicode Section 3.12, the inverse of mjb_hangul_syllable_decompose. Normalization composes Hangul automatically; use this function to build a syllable from individual jamo.

Returns

• MJB_STATUS_OK - The syllable was composed
• MJB_STATUS_INVALID_ARGUMENT - syllable is NULL
• MJB_STATUS_NOT_FOUND - A jamo is outside its range, so the jamo do not form a syllable

Example

mjb_codepoint syllable;

// Choseong kiyeok, jungseong a, jongseong kiyeok
if(mjb_hangul_syllable_compose(0x1100, 0x1161, 0x11A8, &syllable) != MJB_STATUS_OK) {
    return 1;
}

// Syllable: U+AC01
printf("Syllable: U+%04X", syllable);

Related

• mjb_hangul_syllable_decompose
• mjb_normalize

Example

const char *name = mjb_property_name(MJB_PR_ALPHABETIC);

// Property: Alphabetic
printf("Property: %s", name);

Specifications

• UAX #44: Unicode Character Database, Unicode 18.0.0

Parse a BCP 47 language tag, such as sr-Latn-RS, into its components: language, extended language, script, region, variant, extensions, private use, and grandfathered tags. Parsing is strict: malformed tags are rejected.

Returns

• MJB_STATUS_OK - The tag was parsed and locale filled
• MJB_STATUS_INVALID_ARGUMENT - An argument is NULL or the tag is not a valid BCP 47 language tag
• MJB_STATUS_NO_MEMORY - Allocation failed

Example

mjb_text id = mjb_text_utf8("sr-Latn-RS", MJB_NUL_TERMINATED);
mjb_locale_id locale;

if(mjb_locale_parse(&id, &locale) != MJB_STATUS_OK) {
    return 1;
}

// Locale: sr Latn RS
printf("Locale: %s %s %s", locale.language, locale.script, locale.region);

Related

• mjb_locale_code
• mjb_map_case

Specifications

• BCP 47: Tags for Identifying Languages

Return the lowercase two-letter ISO 639-1 code that names locale, such as tr for MJB_LOCALE_TR. MJB_LOCALE_NOT_SET yields an empty string.

Returns

• const char * - The ISO 639-1 code, or NULL if locale is not a valid mjb_locale value

Example

// Locale code: tr
printf("Locale code: %s", mjb_locale_code(MJB_LOCALE_TR));

Related

• mjb_locale_parse
• mjb_map_case

Free the memory allocated for a mjb_result. The result pointer is set to NULL.

Returns

• MJB_STATUS_OK - The result was freed
• MJB_STATUS_INVALID_ARGUMENT - result is NULL

Example

mjb_result result;

mjb_text input = mjb_text_utf8("A", 1);

if(mjb_convert_encoding(&input, MJB_MALFORMED_STOP, MJB_ENC_UTF_16LE, &result,
    NULL) != MJB_STATUS_OK ||
    mjb_result_free(&result) != MJB_STATUS_OK) {
    return 1;
}

// Result released: yes
printf("Result released: %s", result.output == NULL ? "yes" : "no");

Return a static English-language message of at most 100 characters.

Example

const char *message = mjb_status_message(MJB_STATUS_INVALID_ARGUMENT);

// Error message: One or more arguments are invalid or inconsistent with the requested operation
printf("Error message: %s", message);

Output the current library version as a string, such as "1.0.0".

Example

const char *version = mjb_version();

// Version is available: yes
printf("Version is available: %s", version[0] != '\0' ? "yes" : "no");

Related

• mjb_version_number
• mjb_unicode_version

Output the current library version number as an unsigned integer.

Example

unsigned int version = mjb_version_number();

// Version number is positive: yes
printf("Version number is positive: %s", version > 0 ? "yes" : "no");

Related

• mjb_version
• mjb_unicode_version

Output the current supported Unicode version as a string, such as "15.0.0".

Example

const char *version = mjb_unicode_version();

// Unicode version: 18.0.0
printf("Unicode version: %s", version);

Related

• mjb_version
• mjb_version_number

Install the allocator used for internal allocations and owned result buffers. The allocator is copied and must provide all three callbacks, which receive its context pointer. The library passes the size of every block it reallocates or frees, so arena, pool, and accounting allocators do not need to record sizes themselves: old_size and size are the size most recently requested for that block, requested sizes are never 0, realloc receives NULL with an old_size of 0 for a new block and leaves the old block valid on failure, and free is never called with NULL. The context remains caller-owned and must stay valid for the lifetime of all Mojibake allocations. Pass NULL to explicitly select the default allocator. This process-global configuration may be set only once and must happen before any other library call or concurrent library use.

Returns

• MJB_STATUS_OK - The allocator was installed
• MJB_STATUS_INVALID_ARGUMENT - A callback is NULL, or the allocator was already configured

Example

// An allocator that counts the bytes in use. The context points to the counter.
static void *counting_alloc(void *context, size_t size) {
    void *ptr = malloc(size);

    if(ptr) {
        *(size_t *)context += size;
    }

    return ptr;
}

static void *counting_realloc(void *context, void *ptr, size_t old_size, size_t new_size) {
    void *new_ptr = realloc(ptr, new_size);

    if(new_ptr) {
        *(size_t *)context = *(size_t *)context - old_size + new_size;
    }

    return new_ptr;
}

static void counting_free(void *context, void *ptr, size_t size) {
    *(size_t *)context -= size;
    free(ptr);
}

static size_t bytes_in_use = 0;
mjb_allocator allocator = { &bytes_in_use, counting_alloc, counting_realloc, counting_free };

// Install it once, before any other library call
if(mjb_set_allocator(&allocator) != MJB_STATUS_OK) {
    return 1;
}

Related

• mjb_result_free
• mjb_bidi_paragraph_free