Results at a glance
- Software pricing fields13/13
- App Store listings and absences8/8
- G2 review IDs and ratings13/13
- Store prices and stock5/8
- Text PDF cells9/12
- Scanned Census PDF cells (OCR)1/4
- Changes held through the next poll2/2
Each row has its own unit: fields, storefronts, reviews, table cells or change events. Text PDFs show first-run results. The monitor row counts changes that stayed visible through a scheduled check.
Public webpages First attempt
We score each field separately, such as a price, room size or stock status. For each App Store page, we check whether the app is listed or absent in that country. A page counts as correct only if every answer matches the source.
| Task | Pages with every answer correct | Field checks passed |
|---|---|---|
| Software pricing | 4/4 | |
| Hotel room details | 3/4 | |
| Store prices and stock | 1/3 | |
| App Store visibility | 8/8 |
Software pricing passed on all 4 pages, with 13 of 13 pricing fields matched. Allbirds matched 3 of 4 fields, so that page did not pass. The API left size 8 stock blank.
See all 19 webpages, expected answers and first responses
We used JSON extraction for pricing, hotels and stores. For App Store URLs we used Markdown retrieval. We checked the storefront against Apple’s page and its country-specific app lookup. Both had to confirm an absent listing for an HTTP 404 to count as correct.
Software pricing
| Input source | What we checked and expected | What came back |
|---|---|---|
| Momentic | Standard step $0.0125. Browser per minute $0.005 Standard, $0.0067 Large. | All 3 values matched. |
| Linear | Basic $10 and Business $16 per user per month. Business billed annually. Enterprise custom pricing, so no number. | All 4 values matched, including an explicit null for Enterprise. |
| Resend | Pro $20/month, 50,000 included emails/month and $0.90 per 1,000 extra emails. | All 3 values matched. |
| PostHog | 1 Free project, 6 pay-as-you-go projects and 2 years of pay-as-you-go retention. | All 3 values matched. |
Hotel room details
| Input source | What we checked and expected | What came back |
|---|---|---|
| Park Hyatt Kuala Lumpur | Minimum areas of 54 m² for Tower King and 104 m² for Park Suite, plus the bedroom photo URL for each. | Page access error. No requested values returned (0/4). |
| The Singapore EDITION | Garden Premier King photo and Premier Suite living room photo, matched by gallery caption and image asset. | Both image assets matched (2/2). |
| The Hoxton, Shoreditch | Shoebox 12 m², Cosy Up 20 m², and Cosy Up maximum occupancy of 2 adults. | All 3 values matched. |
| The Ludlow | Loft King 325 ft² and Ludlow Loft 405 ft². | Both areas matched (2/2). |
Store prices and stock
| Input source | What we checked and expected | What came back |
|---|---|---|
| Allbirds Mist | Sizes 8 and 9 both $100. Size 8 out of stock, size 9 in stock. | Both prices and size 9 stock matched. Size 8 stock returned null (3/4). |
| Ugmonk Walnut base kit | $69 and in stock. | Both values matched (2/2). |
| Ugmonk Walnut kit with refills | $89 and in stock. | Page access error. No requested values returned (0/2). |
App Store visibility
| Input source | What we checked and expected | What came back |
|---|---|---|
| Spotify · US | Public listing visible. | Listing returned for the US storefront. |
| Spotify · UK | Public listing visible. | Listing returned for the UK storefront. |
| Venmo · US | Public listing visible. | Listing returned for the US storefront. |
| Venmo · UK | No public listing. | Source returned HTTP 404. |
| BBC iPlayer · UK | Public listing visible. | Listing returned for the UK storefront. |
| BBC iPlayer · US | No public listing. | Source returned HTTP 404. |
| PayPay · Japan | Public listing visible. | Listing returned for the Japan storefront. |
| PayPay · US | No public listing. | Source returned HTTP 404. |
Same request, two reruns
We sent the same requests through AgentRouter for the first run and both reruns. Three of the five samples with initial misses returned every field correctly on both reruns. Each bar shows how many fields matched on that attempt.
| Test sample | First run | Rerun 1 | Rerun 2 |
|---|---|---|---|
| Park Hyatt Kuala LumpurBoth room sizes matched. The API returned no photo links. | 0/4 | 2/4 | 2/4 |
| Ugmonk kit with refillsPage access error first. Both reruns found $89 and in stock. | 0/2 | 2/2 | 2/2 |
| Allbirds MistSize 8 stock was missing initially. Both reruns found it out of stock. | 3/4 | 4/4 | 4/4 |
| Federal Reserve H.4.1No Wednesday Reserve Bank credit value on any of the three attempts. | 3/4 | 3/4 | 3/4 |
| BEA GDPBoth reruns found the missing Q2 GDP (3.8%) and investment (−13.8%) figures. | 2/4 | 4/4 | 4/4 |
Both reruns scored the same on all five samples. Independent checks found the same merchant reference values and identical Fed and BEA PDF files. The scored calls show how the full route behaved. They cannot pinpoint whether a difference came from AgentRouter, Context.dev’s API or access to the source.
How to isolate the cause of a changed result
Send the same URL, schema and options directly to Context.dev and through AgentRouter close together, several times, using a fixed source. For each gateway call, compare the raw upstream response with what the gateway returned. A field present upstream and missing from that same call's tool output points to the gateway. If direct API responses vary on the fixed source, the upstream path can vary on its own.
We logged gateway queue rejections separately. Those requests were rejected before execution and are outside the scores shown here.
See the Hyatt response and the three passing controls
The Hyatt request asked for the floor area and bedroom photo URL of the Tower King and Park Suite. The first request returned a page access error. Both reruns returned 54 m² and 104 m² and an explicit null for each photo URL, so the answer stayed at 2/4. The reference photos were listed on the source.
We also resent three requests that had already passed. The Singapore EDITION photos matched 2/2, Ugmonk’s base kit price and stock matched 2/2, and the BLS CPI table matched 4/4 on every attempt. Their original inputs and expected answers are listed under Public webpages and PDF tables and OCR.
G2 reviews IDs, ratings and dates
We gave the G2 parser two product review-list URLs. We matched each returned review to the public listings by ID, then compared its star rating and publication date. Both products had fewer than 25 public reviews.
| G2 product | ID and rating matched | Date matched (Pacific) |
|---|---|---|
| Plausible Analytics | 7/7 | 6/7 |
| Umami | 6/6 | 4/5 |
All 13 IDs and ratings matched. 10 of 12 publication dates matched the public display after converting the API timestamps to Pacific time. One Umami review showed only its last edit date. It counted for ID and rating only.
See the 13 review records and four named lookups
Eight of 12 display dates matched in UTC. Pacific time resolved two one-day differences. A Plausible review still differed by 14 days and an Umami review by four days. The stars below are the same on G2 and in the API response.
Plausible Analytics
| Review ID | Stars | Date on G2 | API date (Pacific) |
|---|---|---|---|
| 14059296 | 5 | 2026-11-11 | 2026-11-11 |
| 13871456 | 5 | 2026-10-27 | 2026-10-27 |
| 13850818 | 5 | 2026-10-23 | 2026-10-23 |
| 13628585 | 5 | 2026-09-28 | 2026-09-28 |
| 9765355 | 4.5 | 2024-06-22 | 2024-06-08 |
| 8435208 | 5 | 2023-08-09 | 2023-08-09 |
| 6833777 | 5 | 2022-07-10 | 2022-07-10 |
Umami
| Review ID | Stars | Date on G2 | API date (Pacific) |
|---|---|---|---|
| 12371028 | 4.5 | 2026-02-17 | 2026-02-17 |
| 10032446 | 4.5 | 2024-08-21 | 2024-08-21 |
| 9928232 | 4.5 | 2024-07-26 | 2024-07-26 |
| 9926970 | 5 | 2024-07-26 | 2024-07-26 |
| 7184555 | 3.5 | Updated 2023-03-28 | 2022-10-10 |
| 7597495 | 5 | 2023-01-19 | 2023-01-15 |
Earlier test of four named reviews
Each request named a review and asked for its rating and date. Neither route got a complete answer (0/4 each). The G2 parser returns up to 25 recent reviews per product and has no pagination. G2 parser documentation
| Input source | What we checked and expected | What came back |
|---|---|---|
| Slack | “Intuitive Team Collaboration, Though Search and Alerts Need Work.” 4.5 stars, published 2026-11-04, updated 2026-11-08. | Generic scraper: page access error. G2 parser: target absent from 25 results. |
| monday CRM | “monday CRM with Unleashed made our sales process smooth.” 4.5 stars, published 2026-11-09. | Generic scraper: page access error. G2 parser: target absent from 25 results. |
| Linear | “Linear hits the right balance of performance and features.” 5 stars, published 2026-08-10. | Generic scraper: page access error. G2 parser: target absent from 25 results. |
| Plausible Analytics | “easy to use and secure tool for site tracking.” 4.5 stars, published 2024-06-22. | Generic scraper: both fields null. G2 parser: found the target among 7 results, returned 4.5 stars and 2024-06-08. |
PDF tables and OCR
We asked for four specific table cells per PDF. To count as correct, each value had to come from the requested row and column, with the right period and unit.
Text PDFs
9/12Correct cells across three text PDFs on the first run. The Fed PDF was still missing the same cell after both reruns.
Uploaded Census scan with OCR
1/4We used optical character recognition to read four table cells from one scanned page. Only California’s 1940 population was correct. OCR read California’s rank as 55 instead of 5. It read both Illinois numbers without keeping them tied to the Illinois row.
See the four PDFs, exact cells and responses
We sent the public PDF URLs to the scraper and asked for structured JSON. The first Census scan request had no OCR and found no cells. Enabling OCR by URL returned an access error. We then uploaded just PDF page 24 with OCR enabled.
| Input source | What we checked and expected | What came back |
|---|---|---|
| BLS CPI · page 2, Table A | October all-items monthly change unpublished. December monthly change 0.3%, annual change 2.7%, gasoline annual change −3.4%. | Returned explicit null for October and all three December values (4/4). |
| Fed H.4.1 · pages 1–2, Table 1 | Reserve Bank credit weekly change 21,729 and Wednesday level 6,533,633. Reserve balances weekly change −3,463 and Wednesday level 2,980,514. Units are USD millions. | Returned three values correctly. Wednesday Reserve Bank credit was null in JSON on all three runs (3/4 each). A separate Markdown request contained it. |
| BEA GDP · page 7, Table 1 | Q2 and Q3 annualized changes: real GDP 3.8% and 4.4%, private investment −13.8% and 0%. | Returned both Q3 values initially (2/4). Both reruns returned all four and still flagged truncation. |
| Census scan · PDF page 24, Table 15 | California 1940 rank 5 and population 6,907,387. Illinois 1960 rank 4 and population 10,081,158. | No cells by URL. Uploaded with OCR, California population matched. California rank read 55. Both Illinois numbers appeared without a reliable state label (1/4 for upload). |
Page monitoring 10-minute polling
We set a 10-minute check schedule for two test pages. We changed the stock count on one page and held the other at 40 units. The chart measures the time from publishing each change to Context.dev recording a change event.
- Stock 17 → 29 units5 min 58 sec
- Stock 29 → 8 units4 min 31 sec
Both lasting changes were detected on the next scheduled check (2/2). Most of each delay was the wait for that check. Once a check began, the event appeared about six seconds later in both cases. The unchanged page produced zero events in two checks.
This schedule can catch changes that stay visible until the next check. A brief change can come and go between checks, as the 61-second update did.
See the test page inputs, exact times and events
We recorded the baseline text on two static pages we hosted for the test: Stock DEMO-A | revision 0 | 17 units available and Stock DEMO-B | revision 0 | 40 units available. We verified each update on the public URL. A detection counted if the event showed the right old and new text, including the revision and stock count. All times are Pacific. Monitor schedules
| Input | Published or checked | What came back |
|---|---|---|
| Stock 17 → 29 | Published 11:11:15 a.m. Check began 11:17:07 a.m. | Event at 11:17:13 a.m. with the correct old and new text. |
| Stock 29 → 5 → 29 | Published 11:18:06 a.m., restored 11:19:07 a.m. | No event. The update came and went between two checks. |
| Stock 29 → 8 | Published 11:23:08 a.m. Check began 11:27:33 a.m. | Event at 11:27:39 a.m. with the correct old and new text. |
| Control held at 40 | Automatic checks started at 11:21:27 and 11:31:13 a.m. | No change and no event ID on either check. |
We also tested Run now on the public Los Angeles clock. The clock element changed from 11:01:50 to 11:02:28 a.m. and produced a change event. The date element stayed on November 16 and produced no event (2/2 expected outcomes).
What to use it for
- Pricing and storefrontsStart with known URLs and explicit fields. Recheck blank stock values.
- Recent G2 reviewsMatch on review IDs and verify dates that drive a decision.
- PDF tables and alertsCheck required cells against the source. Polling can miss changes shorter than the interval.
How we tested
We set the expected answers before scoring. An explicit empty value counted as correct only when the source also had no value for that field. The results cover selected samples from one day.
Requests, timing and cost
AgentRouter handled 27 initial requests, three diagnostics and 16 reruns. The initial requests and diagnostics cost $0.216 and the reruns cost $0.240. We ran the original requests and scoring rules again in two windows about 15 minutes apart. Native G2, OCR and monitoring ran directly on Context.dev.
Merchant reference data matched before and after testing. The Fed and BEA PDFs had identical hashes across rerun checks. Direct Hyatt and BLS reference fetches returned HTTP 403 during reruns, so we checked archived values against indexed copies of their official pages.
Sources
- Webpages
- Linear pricing · Hyatt rooms · Allbirds Mist · Ugmonk with refills
- PDF tables
- BLS CPI, page 2, Table A · Fed H.4.1, pages 1–2, Table 1 · BEA GDP, page 7, Table 1 · Census scan, page 24, Table 15
- G2 and monitoring
- Plausible review with a date discrepancy · Public clock used for on-demand checks
SOTA2, Context.dev API Feature Review, November 16, 2026.