Documentation-driven AI systems need training questions that are tightly coupled to the content they index. Rather than generating questions synthetically (which risks drift from the actual documentation), a well-designed documentation pipeline can extract questions that documentation authors have already written—embedded in MDX frontmatter, custom <Questions> JSX components, and OpenAPI specification extensions. These author-curated questions capture the intent behind each page and serve as high-quality training signal for retrieval and question-answering systems.
The extraction pipeline scans three distinct source formats. MDX files may declare questions in YAML frontmatter under a questions key. They may also embed questions in <Questions> or <FAQ> JSX components rendered in the page body. OpenAPI specifications (YAML or JSON) use a vendor extension x-docs-questions at the operation or path level. All three formats are parsed into a normalized Question struct that includes the question text, an optional answer, metadata tags, and a context ID linking the question to its source document or API endpoint.
This approach turns documentation maintenance into training data maintenance—when an author updates a page and its questions, the training pipeline automatically picks up the changes on the next ingestion run, keeping the question bank aligned with the current state of the docs.
MDX files begin with a YAML frontmatter block delimited by ---. The question extractor parses this block and looks for a questions key containing a list of question objects. Each object has a q (question text) field and optionally an a (answer) field and tags array. This is the simplest extraction path—standard YAML parsing with schema validation.
The frontmatter question schema is intentionally minimal: q (string, required), a (string, optional), tags (string array, optional). Keeping the schema simple encourages documentation authors to actually write questions—a complex schema with many required fields becomes friction that reduces adoption. The context ID is derived automatically from the file path, not specified by the author.
Some documentation frameworks render questions as interactive components in the page body rather than metadata in the frontmatter. The extractor uses a lightweight MDX parser to find <Questions>, <FAQ>, and <QA> components in the document tree, then extracts their props and children as question/answer pairs. This requires parsing JSX well enough to extract attribute values and text children without needing a full React runtime.
Rather than using regex to find question components (which breaks on nested JSX, multiline props, and expression attributes), the extractor parses the MDX into an AST using a lightweight parser. It then walks the AST looking for JSX elements with known component names. This approach handles edge cases like <Questions items={data} /> (expression props) and <FAQ>\n<Q>...</Q>\n<A>...</A>\n</FAQ> (nested components) correctly.
OpenAPI specifications describe API endpoints. The x-docs-questions vendor extension adds questions at the operation level (e.g., under paths./users.get) or at the path level. The extractor parses the OpenAPI spec, walks the path/operation tree, and collects questions from these extensions. The context ID is derived from the HTTP method and path (e.g., GET /users), linking each question to its API endpoint.
Regardless of source format (frontmatter YAML, JSX component, or OpenAPI extension), all extracted questions are emitted as the same Question struct: Text, Answer, Tags, ContextID, Source (which format it came from), and SourcePath (file path or spec URL). This normalization means downstream consumers—training pipelines, evaluation harnesses, search indexes—don’t need to know or care where the question originated.
Each question is linked to its source document through a context ID. For MDX files, this is typically the URL path derived from the file’s location in the documentation tree (e.g., docs/api/authentication maps to /api/authentication). For OpenAPI questions, it’s the operation identifier (GET /users/{id}). Context IDs enable the QA system to scope answers to specific documentation pages and to update questions when their source documents change.
questions key if present<Questions>/<FAQ> componentsx-docs-questionsQuestion structspackage questions
import (
"bufio"
"io"
"strings"
"gopkg.in/yaml.v3"
)
// FrontmatterQuestion is the YAML schema for a question in frontmatter.
type FrontmatterQuestion struct {
Q string `yaml:"q"`
A string `yaml:"a,omitempty"`
Tags []string `yaml:"tags,omitempty"`
}
// Frontmatter holds the parsed frontmatter block.
type Frontmatter struct {
Questions []FrontmatterQuestion `yaml:"questions,omitempty"`
}
// ExtractFrontmatterQuestions parses MDX frontmatter and returns questions.
func ExtractFrontmatterQuestions(r io.Reader, contextID string) ([]Question, error) {
scanner := bufio.NewScanner(r)
// Find opening ---.
if !scanner.Scan() || strings.TrimSpace(scanner.Text()) != "---" {
return nil, nil // No frontmatter.
}
// Read until closing ---.
var fmLines []string
for scanner.Scan() {
line := scanner.Text()
if strings.TrimSpace(line) == "---" {
break
}
fmLines = append(fmLines, line)
}
var fm Frontmatter
if err := yaml.Unmarshal([]byte(strings.Join(fmLines, "\n")), &fm); err != nil {
return nil, fmt.Errorf("parsing frontmatter: %w", err)
}
var questions []Question
for _, fq := range fm.Questions {
if fq.Q == "" {
continue // Skip empty questions.
}
questions = append(questions, Question{
Text: fq.Q,
Answer: fq.A,
Tags: fq.Tags,
ContextID: contextID,
Source: SourceFrontmatter,
SourcePath: contextID,
})
}
return questions, nil
}package questions
import (
"encoding/json"
"fmt"
"strings"
)
// OpenAPISpec is a minimal representation of an OpenAPI document.
type OpenAPISpec struct {
Paths map[string]PathItem `json:"paths" yaml:"paths"`
}
// PathItem contains operations keyed by HTTP method.
type PathItem map[string]json.RawMessage
// ExtractOpenAPIQuestions walks an OpenAPI spec for x-docs-questions.
func ExtractOpenAPIQuestions(spec *OpenAPISpec, specPath string) ([]Question, error) {
var questions []Question
for path, methods := range spec.Paths {
// Check path-level extension.
if raw, ok := methods["x-docs-questions"]; ok {
qs, err := parseExtensionQuestions(raw, path, specPath)
if err != nil {
return nil, fmt.Errorf("path %s: %w", path, err)
}
questions = append(questions, qs...)
}
// Check operation-level extensions.
for method, raw := range methods {
if strings.HasPrefix(method, "x-") {
continue
}
var op struct {
Questions json.RawMessage `json:"x-docs-questions"`
}
if err := json.Unmarshal(raw, &op); err != nil {
continue
}
if op.Questions == nil {
continue
}
contextID := fmt.Sprintf("%s %s", strings.ToUpper(method), path)
qs, err := parseExtensionQuestions(op.Questions, contextID, specPath)
if err != nil {
return nil, fmt.Errorf("%s %s: %w", method, path, err)
}
questions = append(questions, qs...)
}
}
return questions, nil
}
func parseExtensionQuestions(raw json.RawMessage, contextID, specPath string) ([]Question, error) {
var items []struct {
Q string `json:"q"`
A string `json:"a,omitempty"`
Tags []string `json:"tags,omitempty"`
}
if err := json.Unmarshal(raw, &items); err != nil {
return nil, err
}
var questions []Question
for _, item := range items {
if item.Q == "" {
continue
}
questions = append(questions, Question{
Text: item.Q,
Answer: item.A,
Tags: item.Tags,
ContextID: contextID,
Source: SourceOpenAPI,
SourcePath: specPath,
})
}
return questions, nil
}Extracted questions serve as training pairs for RAG systems. Each question maps to a known context document, creating a ground-truth dataset for training and evaluating retrieval models. When the retriever is given a question, it should rank the source document highly—if it doesn’t, the retrieval model needs tuning.
Questions can be used as automated smoke tests for documentation quality. If a page has no questions, it may lack clear purpose. If questions mention concepts not covered in the page content, the documentation has gaps. Running the question extractor as part of a CI pipeline surfaces these issues before they reach production.
Extracted questions are natural candidates for search suggestions. When a user starts typing a query that partially matches an extracted question, the search UI can suggest the full question text. Because these questions were written by documentation authors, they use the terminology and framing that users expect.
The extraction pipeline uses a registry pattern to dispatch to the correct extractor based on file type. MDX files route to the frontmatter+JSX extractor, OpenAPI specs route to the extension extractor, and new formats can be added by registering a new extractor function. This is the same pattern used by the format registry in the document processor.
// ExtractorFunc extracts questions from a file.
type ExtractorFunc func(r io.Reader, path string) ([]Question, error)
// Registry maps file patterns to extractors.
type Registry struct {
extractors []registryEntry
}
type registryEntry struct {
pattern glob.Glob
extractor ExtractorFunc
}
func NewRegistry() *Registry {
r := &Registry{}
r.Register("**/*.mdx", ExtractFromMDX)
r.Register("**/*.md", ExtractFromMDX) // MDX-compatible Markdown.
r.Register("**/openapi.yaml", ExtractFromOpenAPI)
r.Register("**/openapi.json", ExtractFromOpenAPI)
r.Register("**/swagger.yaml", ExtractFromOpenAPI)
return r
}
func (r *Registry) Register(pattern string, fn ExtractorFunc) {
r.extractors = append(r.extractors, registryEntry{
pattern: glob.MustCompile(pattern),
extractor: fn,
})
}
func (r *Registry) Extract(path string, reader io.Reader) ([]Question, error) {
for _, entry := range r.extractors {
if entry.pattern.Match(path) {
return entry.extractor(reader, path)
}
}
return nil, nil // No extractor matches; not an error.
}The same question may appear in both the frontmatter and a JSX component on the same page, or in multiple OpenAPI operations. The pipeline deduplicates by computing a hash of the normalized question text and context ID, keeping only the first occurrence. This prevents duplicate training examples without requiring authors to coordinate across formats.
On re-runs, the pipeline only processes files that have changed since the last extraction (detected via file modification timestamps or Git diff). Previously extracted questions for unchanged files are loaded from a cache. This makes re-extraction fast even for large documentation sites with thousands of MDX files.
?, some don’t), or different Unicode normalization forms. Normalize question text before hashing for deduplication and before storing for training.<Questions> to <FAQSection>, the extractor breaks. Make the list of recognized component names configurable rather than hardcoded, and document the expected component interface.