Skip to main content

internal/ingest/ingest.go

internal/ingest · 276 lines · 11 declarations · source

This file carries the package documentation, rendered on the package page.

Declarations​

const DefaultSegmentBytes, MinSegmentBytes, MaxSegmentBytes and 1 more​

const (
	// DefaultSegmentBytes is the measured ceiling explained above.
	DefaultSegmentBytes = 8 << 10
	// MinSegmentBytes keeps a segment large enough to hold a sentence of any language with its
	// context; below it the cut falls inside sentences constantly and every quote is unlocatable.
	MinSegmentBytes = 256
	// MaxSegmentBytes is the write contract's own message ceiling.
	MaxSegmentBytes = domain.MaxMessageBytes
	// DefaultMaxFileBytes bounds one file before it is read. A document larger than this is not a
	// document an agent's memory should hold as turns; it is a corpus, and it is split by its owner.
	DefaultMaxFileBytes = 16 << 20
)

source

var ErrUnsupportedFormat, ErrTooLarge, ErrEmpty and 4 more​

var (
	ErrUnsupportedFormat = errors.New("unsupported document format: .txt, .md and .json are read")
	ErrTooLarge          = errors.New("document exceeds the file size ceiling")
	ErrEmpty             = errors.New("document carries no text")
	ErrInvalidText       = errors.New("document is not valid UTF-8 text without NUL bytes")
	ErrMalformed         = errors.New("document is not the JSON shape this contract defines")
	ErrSegmentCeiling    = errors.New("segment ceiling must be between 256 bytes and the message ceiling")
	ErrUnsegmentable     = errors.New("document has a run of whitespace longer than a segment")
)

source

type Document​

type Document struct {
	// Name is how the operator named the file; it goes into the manifest, never onto the wire.
	Name string
	// Text is the document as the segments see it: for a text file, the file trimmed of trailing
	// whitespace; for a JSON document, the title, a blank line and the text. Offsets are into this.
	Text string
	// Digest is SHA-256 of Text, so two files with the same words are one document and an edited
	// file is a different one.
	Digest [32]byte
	// OccurredAt is when the document says it happened, or zero when it does not say, in which case
	// the API stamps ingestion time as it does for a conversation that gives none.
	OccurredAt time.Time
}

Document is what was read: the text every segment is cut from, and the identity it will carry.

source

type jsonDocument​

type jsonDocument struct {
	Title      string     `json:"title"`
	Text       string     `json:"text"`
	OccurredAt *time.Time `json:"occurred_at"`
}

jsonDocument is the one JSON shape this contract defines. Unknown fields are refused rather than ignored, because a caller who wrote `occurredAt` and was silently ignored has lost a date without being told.

source

func Read​

func Read(path string, maxBytes int) (Document, error)

Read loads one document, refusing before anything is sent. maxBytes bounds the file and is checked before the file is read, so a file that would not fit is never in memory.

source

type Segment​

type Segment struct {
	Ordinal   int
	ByteStart int
	ByteEnd   int
	Text      string
}

Segment is one piece of a document: the bytes [ByteStart, ByteEnd) of Document.Text, whole.

source

func Split​

func Split(text string, ceiling int) ([]Segment, error)

Split cuts text into segments of at most ceiling bytes whose concatenation is the text. Every cut lies on a code-point boundary and prefers, in order, the end of a paragraph, the end of a sentence, whitespace, and finally the last code point that fits.

source

func cut​

func cut(window string) int

cut returns where to end the first segment of a window that is exactly one ceiling long and is known not to be the end of the text. The choice is the latest boundary of the best kind that leaves at least one word in the segment.

source

func isTerminator​

func isTerminator(r rune) bool

source

var keyNamespace​

var keyNamespace = uuid.MustParse("6f1d8a1e-3b7c-4d2a-9f0e-5c4b3a2d1e0f")

keyNamespace is the fixed UUID namespace under which a segment names itself. It is a constant so that the same document sent from two machines produces the same keys, which is what makes the second send a replay rather than a duplicate.

source

func Key​

func Key(digest [32]byte, ordinal int, role, dataSubjectID string) uuid.UUID

Key derives the idempotency key of one segment. It covers the document's digest, the segment's ordinal, the role and the data subject, so the same words observed as a different speaker or under a different subject are a different observation, as the API would store them.

source