Measuring the Amount of Text and Code in My Blog Posts

In 2017, out of pure curiosity, I decided to check what portion of my blog posts consisted of written text versus source code. Back then, I wrote a concise 21-line Perl program that gave me a quick answer.

In 2021—while experimenting with Perl 6 (now Raku)—I rewrote that script and, to my surprise, received completely different numbers. Due to lack of time, I set the topic aside, but recently returned to it to conduct a comprehensive investigation.

It turned out that a seemingly trivial problem—“count text and code characters on an HTML page”—conceals a fascinating world of engineering nuances. From bugs in open-source CPAN libraries and subtle DOM tree nesting issues to Unicode specifications (graphemes vs. code points) and network-layer HTTP performance optimizations.

In this article, I will take you on a journey through successive attempts to reconcile results across 4 programming languages (Perl 5, Raku, Python, and Rust), presenting complete statistics for 68 blog posts, code ratio evolution charts, and unexpected benchmark results.

Act I: 2017 and the 21-Line Perl 5 Script

My initial 2017 approach relied on a simple premise: fetch the blog home page, extract post links from <h2> headers, download each post individually, and count characters in text elements (<h1>-<h4>, <p>, <li>) versus code blocks (<pre>).

Here is the original script utilizing the HTML::TagParser library:

#!/usr/bin/env perl
use warnings;
use strict;
use HTML::TagParser;

my $url = 'https://gustawdaniel.com';
my @tags = ("h1 h2 h3 h4 li p", "pre");

print "|     text |     code | title \n";

my @list = HTML::TagParser->new( $url )->getElementsByTagName( "h2" );
foreach my $elem ( @list ) {
    my $post = HTML::TagParser->new( $url . $elem->firstChild()->getAttribute( "href" ) );
    my @str = ("", "");
    foreach my $i ( (0, 1) ) {
        my @elements = map { $post->getElementsByTagName($_) } split / /, $tags[$i];
        $str[$i] = join("", map { $_->innerText } @elements);
    }
    printf("| %8d | %8d | %-60s \n", (map { $str[$_] =~ y===c } (0, 1)), $elem->innerText);
}

It worked flawlessly… until I re-ran it years later in a modern environment and hit two major issues.

Problem 1: Missing HTTPS Support

The first error upon re-running in a clean environment was:

URI::Fetch failed: Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)

It turned out that HTML::TagParser internally uses LWP::UserAgent. By default in Perl, LWP only supports http://. Installing liblwp-protocol-https-perl (or via CPAN: cpanm LWP::Protocol::https) enabled encrypted connections.

Problem 2: Anatomy of a CPAN Bug (HTML::TagParser)

After fixing HTTPS, the script ran, but for certain posts (e.g., about Zipf’s Law and Benford’s Law) it printed exactly 0 code characters, even though those articles contained numerous <pre> blocks!

Debugging revealed a bug in HTML::TagParser itself (HTML/TagParser.pm, line 257). When parsing double-quoted attribute values ("), the module used a regular expression that truncated attribute strings at the first occurrence of a… single quote (')!

The blog post URLs were structured as follows:

• /posts/en/zipf's-law-in-nodejs/ $\rightarrow$ truncated to /posts/en/zipf
• /posts/en/benford's-law/ $\rightarrow$ truncated to /posts/en/benford

Fetching those malformed URLs returned 404 error pages (which contained no <pre> tags).

The hacky Perl fix was to bypass the buggy parser method and access Perl’s internal node array structure ($node->[0]->[$node->[1]]->[2]), which holds raw, unparsed HTML attribute text:

sub get_href {
    my ($node) = @_;
    my $raw = $node->[0]->[$node->[1]]->[2] // "";
    if ($raw =~ /href=(?:"([^"]+)"|'([^']+)'|([^\s>]+))/i) {
        return $1 // $2 // $3;
    }
    return $node->getAttribute("href");
}

Library bug resolved, but this was only the beginning of our discoveries.

Act II: The Raku Rewrite and the DOM Nesting Challenge

In 2021, I tried rewriting the algorithm in Raku. In Raku, instead of HTML::TagParser, I used the modern DOM::Tiny module with CSS selectors:

#!/usr/bin/env raku
use WWW;
use DOM::Tiny;

my $base_url = 'https://gustawdaniel.com';
my $dom = DOM::Tiny.parse(get($base_url));

for $dom.find('h3 a') -> $a {
    my $url  = $a.attr('href');
    my $post = DOM::Tiny.parse(get($url));

    my $text = $post.find('h1, h2, h3, h4, p, li').map(*.all-text).join;
    my $code = $post.find('pre').map(*.all-text).join;

    printf("| %8d | %8d | %-60s \n", $text.chars, $code.chars, $a.all-text.trim);
}

The code shrank to just 18 lines and became remarkably clean. However, during an audit across all 68 blog posts, I noticed discrepancies.

In an old post from 2017 (“Application with FOSUserBundle and Google Maps API”), Raku returned 42,469 code characters, while other parsers reported 42,529 characters.

Why Were 60 Characters Missing?

This post contained 70 <pre> code blocks featuring HTML and Twig template snippets. Inside some of those <pre> blocks were nested HTML elements, such as:

<pre class="astro-code"><code><div class="container eternity-form">...</div></code></pre>

Raku’s linear tree walker, without tracking parent node context, encountered the nested <div> tag, “forgot” it was inside a <pre> block, and counted that inner HTML snippet as article text instead of code!

The fix was recursively propagating an $is_pre context flag down the DOM tree:

sub collect-text($tree, $is_pre = False) {
    my $t = "";
    my $c = "";
    for $tree.children -> $node {
        if $node ~~ DOM::Tiny::HTML::Text {
            if $is_pre { $c ~= $node.text; } else { $t ~= $node.text; }
        } elsif $node ~~ DOM::Tiny::HTML::Tag {
            my $in_code = $is_pre || $node.tag eq "pre";
            my ($sub_t, $sub_c) = collect-text($node, $in_code);
            $t ~= $sub_t;
            $c ~= $sub_c;
        }
    }
    return ($t, $c);
}

Act III: HTML Parsing Nuances – Spaces and Entities

Comparing different HTML parsing libraries in Python (selectolax), Rust (scraper), Perl 5 (Mojo::DOM), and Raku (DOM::Tiny) uncovered further subtle differences:

1. HTML Entities (&, <, >, "): Some parsers return raw text with unescaped entities (where & counts as 5 characters), while others decode them to &, <, >, " (where & is 1 character). For consistency across languages, all parsers must operate on decoded text.

2. Syntax Highlighter <span> Tags: Modern static site engines (like Astro with Shiki or Prism) break code blocks into dozens of nested <span class="line"><span class="token keyword">const</span>...</span> elements. Some HTML parsers insert extra whitespace when joining adjacent <span> text nodes during all_text extraction. If a parser injects a space after every highlighted token, the code character count inflates unnaturally!

Act IV: The Unicode Mystery – Graphemes vs. Code Points vs. Bytes

The most interesting discovery occurred while auditing post #13 (tRPC – super fast development cycle…).

The text character count results for this post were:

• Rust: 10,424 characters
• Python: 10,424 characters
• Perl 5: 10,424 characters
• Raku: 10,423 characters (exactly 1 character less!)

Why did Raku report 1 fewer character? I scanned the post’s text character by character for non-ASCII symbols. The culprit was in the sentence:

“I learned tRPC today and fall in love ❤️ instantly…”

Consider the red heart emoji ❤️. In Unicode, this symbol consists of two Code Points:

1. U+2764 (HEAVY BLACK HEART ❤)
2. U+FE0F (Invisible VARIATION SELECTOR-16, instructing the renderer to display a colored emoji).

How Do Languages Count Characters?

• Python (len("❤️")): Counts Unicode Code Points $\rightarrow$ 2 characters.
• Rust (str.chars().count()): Counts Unicode Code Points $\rightarrow$ 2 characters.
• Perl 5 (length("❤️")): Counts Unicode Code Points $\rightarrow$ 2 characters.
• Raku ("❤️".chars): Larry Wall designed Raku to operate natively on NFG (Normalized Form Grapheme). Raku counts what a human user sees on screen as one glyph $\rightarrow$ 1 grapheme!

To align Raku with Python, Rust, and Perl 5 (counting Code Points), we simply substitute .chars with .codes:

# .codes counts Unicode Code Points instead of NFG graphemes
printf("| %8d | %8d | %-60s \n", $text.codes, $code.codes, $title);

With this change, Raku achieved 100% byte-for-byte agreement with the other languages.

Act V: Results Table for All 68 Posts

With unified counting logic and proper DOM nesting handling, I generated complete statistics for all 68 posts published on the blog.

Summary Statistics:

• Total Posts Analyzed: 68
• Total Text Characters: 504,233
• Total Source Code Characters: 430,084
• Total Blog Volume: 934,317 characters
• Overall Code Ratio: 46.03%

Measurement Table:

Text and Code Evolution Charts

The charts below illustrate article volume changes and code ratio percentages over time:

Act VI: Showdown Across 4 Programming Languages

I built matching implementations of the character counting algorithm across 4 languages:

1. Python (httpx + selectolax)

#!/usr/bin/env python3
import httpx
from selectolax.parser import HTMLParser

BASE_URL = "https://gustawdaniel.com"
res = httpx.get(BASE_URL)
parser = HTMLParser(res.text)

for a in parser.css("h3 > a"):
    path = a.attributes.get("href", "")
    url = path if path.startswith("http") else f"{BASE_URL}{path}"
    post_parser = HTMLParser(httpx.get(url).text)

    code_str = "".join(pre.text(deep=True) for pre in post_parser.css("section pre"))
    for pre in post_parser.css("section pre"):
        pre.decompose()
    text_str = "".join(sec.text(deep=True) for sec in post_parser.css("section"))

    print(f"| {len(text_str):8d} | {len(code_str):8d} | {a.text().strip():<60} |")

2. Rust (ureq + scraper)

use ureq;
use scraper::{Html, Selector};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let body = ureq::get("https://gustawdaniel.com").call()?.into_string()?;
    let fragment = Html::parse_fragment(&body);

    for element in fragment.select(&Selector::parse("h3 > a")?) {
        let path = element.value().attr("href").unwrap_or("");
        let post_body = ureq::get(path).call()?.into_string()?;
        let post_fragment = Html::parse_fragment(&post_body);

        let mut code = String::new();
        let mut text = String::new();

        for section in post_fragment.select(&Selector::parse("section")?) {
            for node in section.descendants() {
                if let Some(t) = node.value().as_text() {
                    if node.parents().any(|p| p.value().as_element().map_or(false, |e| e.name() == "pre")) {
                        code.push_str(t);
                    } else {
                        text.push_str(t);
                    }
                }
            }
        }
        println!("| {:8} | {:8} | {:<60} |", text.chars().count(), code.chars().count(), element.text().collect::<String>());
    }
    Ok(())
}

3. Perl 5 (Mojo::UserAgent + Mojo::DOM)

#!/usr/bin/env perl
use warnings;
use strict;
use Mojo::DOM;
use Mojo::UserAgent;

my $ua   = Mojo::UserAgent->new;
my $main = $ua->get('https://gustawdaniel.com')->res->dom;

sub collect_text {
    my ($node) = @_;
    my ($t, $c) = ("", "");
    for my $child ($node->child_nodes->each) {
        if ($child->type eq "text") {
            $t .= $child->content;
        } elsif ($child->type eq "tag") {
            if ($child->tag eq "pre") {
                $c .= $child->all_text;
            } else {
                my ($st, $sc) = collect_text($child);
                $t .= $st; $c .= $sc;
            }
        }
    }
    return ($t, $c);
}

for my $elem ($main->find('h3 a')->each) {
    my $post = $ua->get($elem->attr('href'))->res->dom;
    my ($text, $code) = ("", "");
    for my $sec ($post->find('section')->each) {
        my ($t, $c) = collect_text($sec);
        $text .= $t; $code .= $c;
    }
    printf("| %8d | %8d | %-60s \n", length($text), length($code), $elem->all_text);
}

Verification Audit across All 68 Posts:

All scripts yielded identical counts down to the single character.

Act VII: Performance Benchmark and HTTP Connection Pooling

With 4 working and 100% consistent scripts, I measured execution performance across all programs (Wall-clock real time, CPU time, and peak RAM consumption Max RSS).

Initial Unexpected Result

In the initial benchmark run, Perl 5 with Mojo::UserAgent took 11.39 s, while standard Rust took 5.21 s, and Python took 5.74 s.

However, it was puzzling why Python and standard Rust were only ~2x faster than Perl, given that Rust’s CPU spent only 0.37 s doing parsing work!

The bottleneck lay in the network layer and HTTP Keep-Alive connection persistence:

1. Mojo::UserAgent in Perl reuses persistent TCP/TLS connections.
2. Standard Rust (rust/src/main.rs) invoked static ureq::get(&url) in a loop. Without a shared agent holding a connection pool, every one of the 68 requests initiated a fresh TCP connection and TLS handshake.
3. Python (count_text_and_code.py) invoked top-level httpx.get(url) in a loop, creating a transient client per call. Using with httpx.Client() as client: would enable Connection Pooling (Keep-Alive) in Python as well.

Rust Optimization: rust/src/bin/connection_pool.rs

We instantiate an explicit ureq::Agent::new_with_defaults() that maintains a pool of open sockets and reuses them across requests:

// rust/src/bin/connection_pool.rs - explicit agent with connection pooling
let agent = ureq::Agent::new_with_defaults();

let body: String = agent.get(BASE_URL).call()?.into_string()?;

for element in fragment.select(&selector) {
    // Reusing existing TLS connections via the same agent instance:
    let post_body: String = agent.get(&url).call()?.into_string()?;
}

Final Benchmark Metrics

The table below presents the precise timing and RAM consumption measurements recorded across all implementations:

Performance Comparative Charts

Wall-Clock Real Time Execution

CPU Processor Time

Peak RAM Memory Usage

Summary

What started as a simple metric calculation turned into a rich software engineering study:

1. Library Quirks: Even popular CPAN packages can contain regex edge-case bugs truncating attribute values at single quotes.
2. DOM Hierarchy: Recursively track node context (such as <pre>) to avoid misclassifying nested code tags as body text.
3. Unicode Specs: Before comparing string lengths across languages, verify if you are measuring graphemes (NFG), Code Points, or UTF-8 bytes.
4. Performance & RAM: Rust with Connection Pooling delivers a 3.34 s execution time while requiring only 15 MB RAM (5.7x less than Python and 26x less than Raku).
5. Network I/O Optimization: Reusing HTTP Keep-Alive sessions has a far greater impact on overall execution time for network-bound tasks than choice of programming language or raw HTML parser speed.