MarkosWeb
My own crawl-and-data infrastructure: a domain registry with agentic-readiness scoring and a radar tracking 20,000+ accelerator companies, with a weekly digest that publishes itself.
Overview
MarkosWeb is my own crawl-and-data property, built and operated solo. It began in 2007 as a domain-statistics site and is being rebuilt as the AI Agentic Readiness Directory: for any domain it measures how well the site can be read, trusted and transacted with by AI agents, then publishes the result as a human report, a machine-readable JSON record and an embeddable score badge.
The legacy site was a PHP and MySQL application that rendered domain pages from the database into a static HTML cache served by nginx behind Cloudflare. Its data set had stopped updating years earlier, so instead of migrating a live database I moved a curated snapshot of the most requested pages to Cloudflare Pages and kept the full archive as an offline backup.
The rebuilt directory runs on Cloudflare with Astro and TypeScript. Reports fill themselves: a domain nobody has checked yet is audited live on first request and joins the public directory. A second product line, Startup Ideas, tracks more than 20,000 companies from accelerator portfolios, classifies each into a fixed taxonomy, follows what happens to them and publishes a weekly digest.
What was built
- Agentic-readiness audits: 30 deterministic checks per site across agent interaction, machine extraction, structured data, AI discovery and trust, scored as a fraction of passed checks.
- Self-filling reports, where looking up a domain that has not been audited runs the audit on the spot and adds it to the directory.
- Machine access for agents and developers: a JSON record per domain, an OpenAPI spec, an MCP endpoint over streamable HTTP and embeddable SVG score badges.
- Free ownership verification by DNS TXT record, homepage meta tag or well-known file, re-checked periodically, with a Verified mark and owner-provided facts.
- Directory hubs: categories ranked by measured readiness, an AI-ready leaderboard, an llms.txt adoption directory, statistics and guides, with localized entry points.
- The Startup Ideas radar: accelerator companies classified into a fixed taxonomy, with trend, gap, overlap and idea-validation views, plus a weekly digest published on the site.
- A static migration of the legacy site: a top-pages snapshot for Cloudflare Pages, scripts that rewrote absolute links, stubbed retired forms and tools, and removed dead widgets.
Stack
Frontend
- Astro
- Page templates of the rebuilt directory, built as static output and deployed to Cloudflare.
Backend
- TypeScript
- Main language of the rebuilt directory codebase, alongside Astro templates and SQL.
- Cloudflare Workers
- Dynamic layer on Cloudflare's edge behind live domain lookups and API responses.
- PHP + MySQL (legacy)
- Original application that rendered domain pages from a large MySQL database into a static cache.
Data
- SQL
- Schema and queries behind the directory and startup data sets in the rebuilt project.
- Python
- Migration tooling that selected, rewrote and cleaned legacy pages for static hosting.
AI
- MCP (streamable HTTP)
- Lets AI assistants such as Claude query readiness data and the startup radar directly.
Infrastructure & DevOps
- Cloudflare Pages
- Hosting for the static snapshot of the legacy site and for the rebuilt directory.
- nginx (legacy)
- Served the pre-rendered HTML cache directly, without touching PHP, before the Cloudflare move.
Integrations & Payments
- OpenAPI
- Published specification for the per-domain JSON API used by developers and agents.
Engineering highlights
Chose a static snapshot over a database migration: log analysis showed most requests came from bots and the legacy data had been frozen for years, so only the most requested pages moved, sized to fit Cloudflare Pages' free-tier file limit.
Scoring follows a Lighthouse-style method: deterministic checks, a headline that is a fraction of passed checks rather than an invented 0 to 100, and an explicit not-assessed state instead of guesses.
Linking is tied to verification: unverified profiles mention a domain as plain text with no outbound link, verified profiles carry exactly one link to the owner's homepage, and links are never reciprocal.