unicode/utf8

UTF-8 rune counts versus graphemes, validation, and DecodeRune on invalid sequences, checked on Go 1.24.0.

模块页正文目前为英文。

The unicode/utf8 package translates between runes and UTF-8 byte sequences. Import it as "unicode/utf8". Examples were checked on Go 1.24.0.

A rune is a Unicode code point. RuneCount and RuneCountInString count runes, not grapheme clusters (user-perceived characters) and not bytes. This page does not repeat the strings package discussion of byte indexes in "Go语言"; it links there for Index and slicing. Combining marks and ZWJ emoji sequences are several runes even when they look like one glyph.

Examples were checked against Go 1.24.0. See the official package documentation for the complete API.

Go reference overview →

API index

APISummary
utf8.RuneCountInString, utf8.RuneCountCount Unicode code points in a string or byte slice, which is not the same as counting glyphs.
utf8.Valid, utf8.DecodeRune, utf8.FullRune, utf8.ValidRune, utf8.RuneLen, utf8.EncodeRuneDetect invalid UTF-8, decode a bad or incomplete sequence as RuneError, and encode a valid rune.

RuneCount is not a grapheme count

The precomposed character é (U+00E9) is one rune and two UTF-8 bytes. The decomposed sequence e plus combining acute accent (U+0301) is two runes and three bytes, and it is not equal to the precomposed string. Both may display as a single glyph, depending on the font. The thumbs-up emoji is one rune and four bytes.

Official documentation states that RuneCount treats erroneous and short encodings as single runes of width 1 byte. There is no grapheme-cluster iterator in this package; a third-party segmenter would be out of scope here.

utf8.RuneCountInString, utf8.RuneCount

func RuneCountInString(s string) (n int)
func RuneCount(p []byte) int

Count Unicode code points in a string or byte slice, which is not the same as counting glyphs.

ParameterDescription
s / pA string or a slice of UTF-8 bytes. Invalid sequences still contribute one rune per bad byte.

Returns: The number of runes, including U+FFFD-width counts for invalid bytes as documented for RuneCount.

  • len(s) remains a byte count, as already shown on the strings page.
go
package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	composed := "é"
	decomposed := "é"
	fmt.Println(utf8.RuneCountInString(composed), len(composed), utf8.RuneCountInString(decomposed), len(decomposed))
	fmt.Println(composed == decomposed, utf8.RuneCountInString("👍"), len("👍"))
}

Expected output

1 2 2 3
false 1 4

Related: strings

Official documentation →

Validation and invalid sequences

Valid and ValidString report whether every byte is well-formed UTF-8. DecodeRune unpacks the first encoding. Official documentation says that an empty slice returns (RuneError, 0), and that an invalid encoding returns (RuneError, 1). RuneError is U+FFFD. An encoding is invalid if it is incorrect UTF-8, encodes a rune that is out of range, or is not the shortest possible encoding.

FullRune is false for an incomplete prefix such as the single lead byte 0xc3. ValidRune is false for a surrogate such as U+D800, and RuneLen then returns -1. EncodeRune writes the UTF-8 bytes for a valid rune into a buffer.

utf8.Valid, utf8.DecodeRune, utf8.FullRune, utf8.ValidRune, utf8.RuneLen, utf8.EncodeRune

func Valid(p []byte) bool
func DecodeRune(p []byte) (r rune, size int)
func FullRune(p []byte) bool
func ValidRune(r rune) bool
func RuneLen(r rune) int
func EncodeRune(p []byte, r rune) int
const RuneError = '\uFFFD'

Detect invalid UTF-8, decode a bad or incomplete sequence as RuneError, and encode a valid rune.

ParameterDescription
pA UTF-8 byte slice for Valid, DecodeRune, FullRune, or the destination for EncodeRune.
rA rune. Use rune(0xD800) for a surrogate; a � character literal is rejected by the compiler.

Returns: Valid returns a bool. DecodeRune returns the rune and its width in bytes. RuneLen returns the encoded width, or -1 if r cannot be encoded. EncodeRune returns the number of bytes written.

  • The compiler rejects a � rune literal as an invalid Unicode code point, so the example uses rune(0xD800).
go
package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	fmt.Println(utf8.Valid([]byte{0xff}))
	r, size := utf8.DecodeRune([]byte{0xff})
	fmt.Println(r == utf8.RuneError, size)
	r, size = utf8.DecodeRune(nil)
	fmt.Println(r == utf8.RuneError, size)
	fmt.Println(utf8.FullRune([]byte{0xc3}), utf8.RuneCount([]byte{0xff, 0xfe}))
	fmt.Println(utf8.ValidRune(rune(0xD800)), utf8.RuneLen(rune(0xD800)))
	buf := make([]byte, 4)
	n := utf8.EncodeRune(buf, 'é')
	fmt.Printf("%q %d\n", buf[:n], n)
}

Expected output

false
true 1
true 0
false 2
false -1
"é" 2

Official documentation →