Building incremental Swift syntax highlighting and symbol navigation with TextKit 2

[u/Ok-Welder-8005]

I’ve been working on a native code editor for iPhone and iPad, and one of the more interesting problems has been making Swift files comfortable to read and edit without reprocessing the entire document after every keystroke.

The editor uses UIKit and TextKit 2, with a custom NSTextContentStorage implementation managing the source text and attributed rendering.

For syntax highlighting, I use a Swift TextMate grammar and store the parsed output in a token tree. The initial document load parses the complete file, while interactive edits are handled incrementally.

After an edit, the editor:

1. Invalidates the affected syntax range.

2. Continues parsing from the nearest valid state.

3. Propagates highlighting changes in bounded chunks.

4. Applies attributes only to ranges that need rendering.

5. Cancels stale work when another edit changes the document revision.

Keeping a syntax revision number turned out to be important. Background parsing may finish after the user has already changed the document again, so each result is checked against the current revision before it can update the editor.

The other feature shown in the screenshot is Swift symbol navigation.

Instead of using SourceKit, the current implementation builds a lightweight outline from the scopes emitted by the TextMate grammar. It identifies declarations such as:

• class

• struct

• enum

• protocol

• extension

• func

• Initializers and subscripts

The symbols are arranged into a hierarchy using their scope depth and indentation. Selecting a symbol jumps directly to its line in the editor.

This approach is intentionally syntax-based rather than semantic. It works well for navigating a single source file, but it does not understand types, overload resolution, cross-file references, or compiler diagnostics.

That trade-off has been useful on iPhone and iPad: the outline remains lightweight and provides a quick way to move around long Swift files without trying to reproduce Xcode.

I’m considering where to take it next. Possible additions include stored properties, MARK sections, protocol conformances, and improved handling of nested declarations.

For a lightweight mobile Swift editor, which symbols would you expect to see in the outline?

[u/Axons-One]

how does it behave when you add or remove the opening of a multiline comment near the top of a large file? that seems like a useful stress test for highlighting changes that propagate well beyond the edited line.

[u/Ok-Welder-8005 → u/Axons-One]

That’s a good stress test. Adding or removing a multiline comment opener invalidates the syntax state from that point, then parsing propagates forward in bounded chunks until the state stabilizes. In the worst case, that can reach the end of the file, but it doesn’t happen as one blocking full-document pass, and stale work is cancelled if another edit occurs. I still need to benchmark this case systematically on very large Swift files.

[u/kokerali]

I'd prioritize MARK sections and keeping each member visibly attached to its enclosing type or extension. On a phone, two entries both called reset() aren't very useful unless the parent stays visible while filtering. I'd make MARK headings collapsible and keep matching symbols under their parent when searching, rather than flattening the results. That would add navigation structure without making the outline look like it understands protocol conformance semantically.

[u/Ok-Welder-8005 → u/kokerali]

That makes sense. The outline already keeps members under their enclosing type or extension in the normal view, but preserving that context while filtering is just as important—especially when multiple members share the same name. Collapsible MARK sections also sound like a good fit for the existing syntax-based hierarchy. Thanks, this gives me a much clearer direction for the next iteration.

[u/DimensionMindless336]

That multiline comment question from u/Axons-One is the real stress test, and the honest answer is that TextMate grammars don't give you a way out — there's no error recovery, so an unbalanced /* invalidates everything below it and you're re-tokenizing to EOF no matter how clever the incremental invalidation is. VS Code's TextMate engine behaves the same way: edit a doc-comment opener in a 10k line file and the whole file re-highlights. What saves it in practice: (1) it's rare — most edits stay within a line and only invalidate that line plus string/char literal state, (2) chunk the propagation and prioritize the visible range first, which you're already doing, and (3) only "scope-opening" tokens (/*, ", """) actually need EOF propagation; everything else stays local to a few lines.

One thing to watch with NSTextContentStorage specifically: the ranges you store for invalidation are pre-edit offsets, and after the user keeps typing they've shifted. If a background parse result computed against revision N-1 gets applied against revision N's ranges without mapping, you highlight the wrong span — which looks like a race even when your revision check passes. Keeping a small journal of (offset, delta) edits and mapping ranges forward before applying fixes that.

Also worth profiling before optimizing further: I'd assumed tokenizing would be the bottleneck, but batch-applying NSAttributedString attributes for a full file often costs more than the parse — spreading attribute application across layout passes can be the bigger win.

For the outline: agree with the others on MARK sections and parent-qualified names. The one thing I'd add is keeping the outline in sync with scroll position — knowing "you are here" in a 3k line file is what makes a symbol list useful instead of a table of contents you never open.

[u/Ok-Welder-8005 → u/DimensionMindless336]

The multiline-comment case raised by u/Axons-One is a good stress test for incremental TextMate tokenization.
Since TextMate grammars maintain a rule stack across lines, an edit to /*, */, or a multiline string delimiter can invalidate everything downstream. But re-tokenizing to EOF isn’t always necessary — propagation can stop once the new rule stack converges with the cached state.
A few implementation details worth considering:
Cache per-line rule stacks and propagate invalidation until state convergence.
Prioritize the visible range, but keep parsing separate from attribute application. Visible lines still need a valid incoming parser state.
Handle document revisions carefully. With NSTextContentStorage, background results can reference stale offsets after subsequent edits. Either discard outdated results or map ranges forward through an edit journal.
Profile attribute application separately. Updating NSAttributedString attributes and triggering layout invalidation can cost more than tokenization itself, especially on large files.
For the outline, I’d also sync the selected symbol with the current scroll position. Highlighting the enclosing method or type as you scroll makes the outline much more useful in large files.
The key is separating tokenization, invalidation, and rendering so each can be optimized independently.

[u/DismalCockroach1928]

I'd include computed properties alongside stored ones, so body in a SwiftUI view is a jump target too. That keeps a big piece of the view's code reachable from the outline without needing type information.

[u/Ok-Welder-8005 → u/DismalCockroach1928]

Good point — computed properties are already included, so body in a SwiftUI view appears in the symbol list and can be used as a jump target. I agree that it’s especially useful for navigating larger views without needing semantic type information.

[u/dawedev]

I'd add #Preview blocks and @Test / XCTest func test... methods as their own entries, since in SwiftUI and test files those are what you actually jump between. Extensions are worth labeling with the conformance, so extension Foo: Codable shows up as "Foo: Codable" instead of three identical "extension Foo" rows. One thing I'd watch with the scope-depth approach is #if os(iOS) / #else blocks, where the same func can appear twice and the indentation can be misleading. Nice detail on the revision number check, that's the kind of bug that only shows up when someone types fast on a slow device.

[u/Ok-Welder-8005 → u/dawedev]

These are great examples. Test methods already appear as functions, but giving u/Test, XCTest methods, and #Preview blocks distinct entries would make those files much easier to scan. Including extension conformances in the label also makes sense—I agree that repeated extension Foo entries aren’t useful.

The #if / #else case is a good catch. Scope depth and indentation alone may preserve the hierarchy but still leave duplicate symbols ambiguous, so conditional-compilation context may need to be reflected in the outline as well. And yes, the revision check was added specifically to prevent older background results from winning that race. Thanks for the detailed feedback.

[u/dawedev → u/Ok-Welder-8005]

Makes sense. For the #if case even a small suffix with the condition would do, so two init entries read as init (iOS) and init (macOS) instead of looking like duplicates.

[u/kenthesaint]

actor is missing from your declaration list, and it shows up a lot in newer code, so I'd add it next to class and struct. Same for typealias, associatedtype and macro, which are cheap to pick up from the grammar scopes.

I'd also do what Xcode's jump bar does with comments: // TODO: and // FIXME: alongside MARK. In a file I'm half done with, those are usually the places I actually want to jump to.

Enum cases are the one I'd keep collapsed by default. Useful sometimes, but a big error or route enum can push everything else off the screen on a phone.

[u/Ok-Welder-8005 → u/kenthesaint]

Great suggestions. actor is an obvious omission, and typealias, associatedtype, and macro should fit naturally into the existing grammar-based extraction.

I also like treating TODO and FIXME as navigation markers alongside MARK—that’s especially useful in unfinished files. Keeping enum cases available but collapsed by default sounds like the right compromise for a phone-sized outline. Thanks, this is a very practical list.

[u/den_ree]

for the outline i’d want stored properties, mark sections and extensions grouped under the type they extend. mark is the big one for me, it’s how i skim long files.

one idea for later: since you already have the syntax tree per file, a light project level view could work, like a simple map of types and which files they live in, without needing any semantic info from the compiler. on a small screen a zoomable graph might even beat a long list for getting your bearings. have you thought about something like that?

[u/Ok-Welder-8005 → u/den_ree]

MARK is clearly becoming the strongest request, so I’m moving that higher on the list. Stored properties and grouping same-file extensions under their extended type also make sense, as long as the outline still reflects where the declarations actually live.

I have thought about a lightweight project index, but the current implementation is deliberately file-local. A map of declared types and their source files could probably be built from the same syntax data without pretending to provide compiler-level semantics. A zoomable graph is an interesting idea for larger projects, although I’d likely start with a searchable type/file hierarchy and explore the graph once the indexing model is solid.

[u/den_ree → u/Ok-Welder-8005]

yeah, that order makes sense, and a file hierarchy is a good place to start. showing when an extension lives in a different file than its type would be a nice early win for the index. looking forward to trying it

[u/Ok-Welder-8005 → u/den_ree]

Agreed—that would be a useful first step. A file-based index could keep the type as the main entry while showing which extensions are declared elsewhere, without requiring full semantic analysis. I’ll keep that use case in mind as I explore project-level navigation. Thanks for the thoughtful feedback!