bufio

bufio.Scanner default token limits and ErrTooLong, plus Reader and Scanner buffer aliasing, checked on Go 1.24.0.

The bufio package wraps an io.Reader or io.Writer with a buffer. Import it as "bufio". Scanner is a convenience for splitting input into tokens, by default lines. Reader and Writer are the buffered I/O types. Examples were checked on Go 1.24.0.

This page does not repeat the io package discussion of short reads, ReadFull, Copy, or EOF. Official documentation already warns that Scanner.Bytes, Reader.Peek, and Reader.ReadSlice return slices that stop being valid at the next read. Use Text, ReadBytes, or an explicit copy if you need to keep the data.

Examples were checked against Go 1.24.0. See the official package documentation for the complete API.

Go reference overview →

API index

APISummary
bufio.NewScanner, Scanner.Scan, Scanner.Text, Scanner.Err, Scanner.Buffer, bufio.ErrTooLongScan lines, check Err at EOF, and detect a token that exceeds the buffer limit.
Scanner.Bytes, Reader.ReadSlice, Writer.FlushCopy Scanner or Reader buffer slices before the next read, and flush a Writer so the destination sees the bytes.

Scanner, Err, and the token size limit

NewScanner(r) scans r with ScanLines by default, stripping line endings. Scan returns true while a token is available. After Scan returns false, Err reports the first non-EOF error, or nil at a normal end of input. Text returns the current token as a newly allocated string.

MaxScanTokenSize is 64 * 1024. Official documentation states that this is the maximum size used to buffer a token unless you call Buffer, and that the actual maximum may be smaller because the buffer may also need to hold a newline. A token that does not fit causes Scan to stop with ErrTooLong. The example uses Buffer with a 4-byte maximum so the failure does not require a 64 KiB line. Always check Err after the loop; do not look only at Scan.

bufio.NewScanner, Scanner.Scan, Scanner.Text, Scanner.Err, Scanner.Buffer, bufio.ErrTooLong

func NewScanner(r io.Reader) *Scanner
func (s *Scanner) Scan() bool
func (s *Scanner) Text() string
func (s *Scanner) Err() error
func (s *Scanner) Buffer(buf []byte, max int)
const MaxScanTokenSize = 64 * 1024
var ErrTooLong = errors.New("bufio.Scanner: token too long")

Scan lines, check Err at EOF, and detect a token that exceeds the buffer limit.

ParameterDescription
rThe underlying io.Reader. NewScanner does not close it.
buf / maxBuffer sets the initial backing array and the maximum token size. Official documentation says the maximum token size must be less than the larger of max and cap(buf). Buffer panics if it is called after scanning has started.

Returns: Scan returns whether a token was read. Text returns that token without the trailing newline. Err returns nil at a clean EOF, or ErrTooLong when a token is too large.

Errors

  • ErrTooLong — The token is larger than the Scanner buffer allows. Compare with errors.Is. The default limit is MaxScanTokenSize, 64 KiB.
  • The examples print errors.Is and sc.Err() == nil rather than the ErrTooLong string.
go
package main

import (
	"bufio"
	"errors"
	"fmt"
	"strings"
)

func main() {
	sc := bufio.NewScanner(strings.NewReader("one\ntwo\nthree"))
	var lines []string
	for sc.Scan() {
		lines = append(lines, sc.Text())
	}
	fmt.Printf("%q %v\n", lines, sc.Err())
	sc2 := bufio.NewScanner(strings.NewReader("abcdefghij\n"))
	sc2.Buffer(make([]byte, 0, 4), 4)
	fmt.Println(sc2.Scan(), errors.Is(sc2.Err(), bufio.ErrTooLong))
}

Expected output

["one" "two" "three"] <nil>
false true

Related: io

Official documentation →

Buffer aliasing and Flush

Scanner.Bytes returns the current token without allocating. Official documentation states that the underlying array may be overwritten by a later Scan. The example keeps both a string copy and the raw slice, then continues scanning with an 8-byte Buffer. On Go 1.24.0 the copy stays AAAA, and the retained slice becomes HHHH, the last token. That final value is an implementation result for this input, not a promise of what the stale slice will contain; the documented rule is that the slice is no longer valid.

Reader.ReadSlice returns a slice into the internal buffer and, officially, those bytes stop being valid at the next read. Most callers should use ReadBytes or ReadString, which copy. With NewReaderSize(..., 8) and the input below, the first ReadSlice result was overwritten after later reads on Go 1.24.0. Peek has the same validity rule. Writer.Write may stay in the buffer until Flush; the destination stays empty until then.

Scanner.Bytes, Reader.ReadSlice, Writer.Flush

func (s *Scanner) Bytes() []byte
func NewReaderSize(rd io.Reader, size int) *Reader
func (b *Reader) ReadSlice(delim byte) (line []byte, err error)
func (b *Reader) Peek(n int) ([]byte, error)
func NewWriter(w io.Writer) *Writer
func (b *Writer) Flush() error

Copy Scanner or Reader buffer slices before the next read, and flush a Writer so the destination sees the bytes.

ParameterDescription
delim / nReadSlice reads until delim, including it. Peek returns the next n bytes without advancing, and those bytes also stop being valid at the next read.

Returns: Bytes and ReadSlice return slices that may alias internal buffers. Flush writes buffered data to the underlying Writer and returns any write error.

  • string(held) or append([]byte(nil), held...) makes a copy you can keep. Do not treat a later mutation of the old slice as a portable value; the output below was measured on Go 1.24.0.
go
package main

import (
	"bufio"
	"bytes"
	"fmt"
	"strings"
)

func main() {
	sc := bufio.NewScanner(strings.NewReader("AAAA\nBBBB\nCCCC\nDDDD\nEEEE\nFFFF\nGGGG\nHHHH\n"))
	sc.Buffer(make([]byte, 8), 8)
	sc.Scan()
	held := sc.Bytes()
	copied := string(held)
	for sc.Scan() {
	}
	fmt.Println(copied, string(held), sc.Err() == nil)
	rdr := bufio.NewReaderSize(strings.NewReader("aaa\nbbb\nccc\nddd\neee\nfff\n"), 8)
	first, err := rdr.ReadSlice('\n')
	fmt.Printf("%q %v\n", first, err)
	for i := 0; i < 4; i++ {
		_, err = rdr.ReadSlice('\n')
	}
	fmt.Printf("%q %v\n", first, err)
	var dst bytes.Buffer
	w := bufio.NewWriter(&dst)
	w.WriteString("hi")
	fmt.Printf("%q %d\n", dst.String(), w.Buffered())
	w.Flush()
	fmt.Printf("%q\n", dst.String())
}

Expected output

AAAA HHHH true
"aaa\n" <nil>
"eee\n" <nil>
"" 2
"hi"

Official documentation →