ProjectsCode Comment Remover
A command-line program that reads a source file and prints it again with the comments removed and everything else exactly as it was, for five languages with three styles of comment, while keeping the comments that tools rely on. Building it teaches how a scanner keeps track of where it is in the text, why searching for comment markers breaks as soon as one sits inside a string, and how the same stream of tokens sits under compilers, minifiers and formatters.
From Coding Challenges #87 ↗ by John Crickett
No sign-up. You tell the mentor your language: Java, Python or Node.js.
What you'll learn · 6 concepts
In the order the steps use them. Each one links to the best places to learn it, and you explain it in your own words once its steps are built.
Every step removes comments without touching strings that contain the same characters, and from step 2 a block comment can run across many lines.
Covers: the states: code, string, character literal, line comment, block comment · escaped quotes inside strings · state carried from one line to the next · why a line-by-line pattern breaks · nesting needs a counter or a stack
Crafting Interpreters: Scanning (Robert Nystrom) ↗ · Lexical analysis (Wikipedia) ↗ · Finite-state machine (Wikipedia) ↗
Steps 1 to 3 cover three comment styles across four languages, and each has strings, character literals or slashes that rules borrowed from another language get wrong.
Covers: line comments and block comments · block comments that do or don't nest · quote styles, multi-line strings and raw strings · strings with embedded code · character literals and digit separators · a slash that starts a regular expression · a backslash at the end of a line
Python Language Reference: Lexical analysis (comments, string literals) ↗ · MDN: JavaScript lexical grammar (comments, regular expression and template literals) ↗ · cppreference: Comments in C (no nesting, line splicing, the C99 change) ↗
The page's expected outputs drop the comment-only lines and keep every line of code as it was. The language of step 2 can end a statement at a line break, and the languages of step 3 define a comment as a space.
Covers: everything outside comments copied byte for byte · a deleted comment can glue two words together · a block comment with a line break counts as a line break · comment-only lines and leftover spaces · line numbers that still match
cppreference: Phases of translation (phase 3, each comment becomes one space) ↗ · The Go Programming Language Specification: Comments ↗
Step 4 keeps doc comments and step 5 keeps linter and build directives, so the remover has to tell the comments that are part of the program from the ones that are only notes.
Covers: doc comments attached to the declaration below · directives read by linters and build tools · the exact form a tool recognises · directives tied to a line or a place in the file · comments the system reads, such as the interpreter line
Go Doc Comments (doc comments and directives) ↗ · flake8: Selecting and Ignoring Violations (noqa comments) ↗ · Shebang (Unix) (Wikipedia) ↗
Step 4 can only tell a doc comment from an ordinary one by looking at the token after it, and the page's going-further ideas, a minifier and an obfuscator, are built on the same stream.
Covers: tokens with a kind, exact text and position · comments kept as trivia attached to tokens · context from the token before · lookahead to the token after · minifiers, formatters, highlighters and obfuscators
Microsoft Learn: the .NET Compiler Platform syntax model (see Syntax trivia) ↗ · Minification (programming) (Wikipedia) ↗ · Prettier: Rationale (see the section on comments) ↗
The page gives no test files. Step 1 asks you to test on real open-source code and your own examples, and steps 2 and 3 give one expected output each.
Covers: properties that hold for any input · the same tokens before and after, minus the comments · running it twice changes nothing · the compiler or interpreter as an oracle · real code plus hand-written nasty cases
Metamorphic testing (Wikipedia) ↗ · Black documentation: The basics (see --fast and --safe) ↗
What you'll build · 6 steps
Steps of 30 to 60 minutes, in your own editor and language. Each one has a check, so you know when it works.
Create the project and settle the interface, which the page leaves open: the program takes the path of one source file, picks the comment rules from the file's extension, and prints the result to standard output. Collect a few real source files in each of the page's languages to test on.
Works whenBefore removing anything, the program prints a file back byte for byte, so comparing its output with the original shows no difference.
For a scripting language whose comments start with a hash sign and run to the end of the line, remove every comment, whether it fills a whole line or follows code. A hash sign inside a string is not a comment, and a string written as documentation is code, so both stay.
Works whenOn real open-source files and on examples of your own, no comments are left, every string containing a hash sign is unchanged, and each stripped file still runs the same as the original.
Concepts: A scanner is a state machine · testing a rewrite that must keep the meaning
For the main scripting language of web pages, remove both of its comment styles: line comments that start with a double slash, and block comments that open with a slash and a star and close with a star and a slash, even many lines later. Strings, template strings and regular expression literals that contain those characters stay as they are.
Works whenThe page's example function comes out exactly like its expected output: lines that held only a comment are gone, blank lines stay, and every line of code is unchanged.
Concepts: A scanner is a state machine · each language has its own lexical rules · removing only the comments
Apply the same two comment styles to two compiled systems languages, with the rules that are their own: character literals in single quotes, a backslash at the end of a line comment that pulls the next line into it, and in one of them, raw strings and digit separators.
Works whenThe page's example comes out exactly like its expected output, a block comment squeezed between two words leaves them as two words, and a stripped file of your own still compiles and behaves the same.
Concepts: each language has its own lexical rules · removing only the comments · testing a rewrite that must keep the meaning
For a compiled language whose documentation tool reads comments, remove every comment except doc comments: the ones directly above a top-level declaration of the package, a constant, a function, a type or a variable, with no blank line in between.
Works whenIn a file of your own, a comment directly above a top-level declaration survives whole, every line of it, while a comment with a blank line before its declaration, or one inside a function body, is removed.
Concepts: comments that carry meaning · the token stream
Also keep the comments that configure tools: the linter suppression on a line of the step 1 language, and the code-generation and build directives of the step 4 language. Every other comment is still removed.
Works whenThe page's three directive examples come through unchanged and in place, the linter still skips the warning the suppression names, and a file marked to be left out of the build is still left out.
Concepts: comments that carry meaning