Audience
Technical builder
The system is tuned for AI and data engineering research, not general news consumption.
Case study · Information Digest Agentic
Auditable AI signal pipeline.
Information Digest Agentic is a personal research pipeline for AI and data engineering signals. It turns RSS, YouTube and GitHub inputs into typed intermediate records, a weekly synthesis, specialist lens reports and downstream delivery artifacts.
Raw source capture, typed Silver analysis, Gold synthesis, source audit, specialist lenses and downstream delivery are separated by deterministic boundaries.
Problem
A summarizer is not enough when the input stream keeps growing.
The useful question is not only what changed this week. It is which sources produced the signal, which items were filtered out, which conclusions survived synthesis and what evidence can be inspected later.
Failure mode
Untraceable synthesis
A polished summary is not very useful if it cannot show which source material shaped the result.
Constraint
Keep AI bounded
AI work happens inside typed tasks and gates; deterministic Python owns routing, validation, delivery and health reporting.
Design choice
Separate data movement from AI judgment.
The pipeline uses a medallion-style structure: raw Markdown source capture, Silver JSON analysis, Gold weekly synthesis and downstream delivery outputs.
The important boundary is the Silver layer. Source items become typed records before they are allowed into synthesis. That makes scoring, filtering, source references and audit behavior inspectable.
Agentic behavior is used where it adds judgment: analysis, synthesis, specialist lenses and PKM planning. Fetching, validation, routing, file movement and health reports remain deterministic code.
Pipeline model
How a weekly run moves through the system.
Step 1
Fetch. RSS, YouTube and GitHub sources are captured into raw Markdown files with source-specific fetch reports.
Step 2
Silver. Each raw item is analyzed into validated JSON with score, source reference and traceability fields.
Step 3
Gold. High-scoring Silver records are reduced into a weekly digest and then audited against source references.
Step 4
Lenses. Specialist reports inspect agent skills, evals, orchestration, portability, reference cases and portfolio ideas.
Step 5
Delivery. Gold and lens outputs are copied into a PKM inbox, and the digest can be curated into an infographic artifact.
Latest recorded run
- Week 2026-29 processed 83 source items
- 83/83 Silver analyses succeeded
- 60 items passed the minimum score threshold
- 6/6 specialist lens reports completed
- PKM delivery and infographic output completed
Implementation
What the current branch includes.
Runtime control
- Prefect 3 flow coordinates fetch, Silver, Gold, lenses, PKM delivery and infographic tasks.
- Agent Runtime validates task policy before launching CLI-backed AI calls.
- Provider and model concurrency slots are acquired before the CLI process starts.
Traceability
- Silver records carry source references and origin-file fields.
- Gold synthesis is followed by a source audit gate.
- Run outputs include fetch reports, pipeline health, source health and agent audit JSONL.
Delivery surfaces
- Weekly Gold digests are written as Markdown.
- Specialist lens reports are delivered beside the digest.
- Infographic tasks curate, render and validate a visual artifact from the digest.
Evidence
Verified locally before this case page was added.
The portfolio page uses only evidence that is implemented or verified in the current local project state. External CI and full live CLI runs are not claimed here.
Checks and artifacts
- 133 offline tests passed; 33 CLI/E2E tests deselected
- Week 2026-29 pipeline health artifact recorded
- 36 sources tracked in source-health report
- 26 sources ok, 0 degraded, 0 dead in latest source-health summary
- Agent audit log separates queue wait and execution time
Boundaries
What this project does not claim yet.
This is a personal research pipeline, not a packaged multi-tenant product. The current public link points to the `agentic-workflow` branch of the original Information Digest repository.
It does not claim production document intelligence: there is no OCR, layout-aware PDF parsing, hybrid retrieval, reranking or page-level citation QA in this case.
There is no hosted CI, Docker delivery or public sample dataset claimed on this page yet. Those are the right next hardening steps before treating the repository as fully showcase-ready.