A model will quote your website with a straight face and still be wrong. The passage it retrieved was a mega-menu, a cookie banner, and a pricing table from a page you stopped maintaining. Retrieval augmented generation does not invent trust. It inherits whatever you bothered to ingest.
That is the miss hiding inside a lot of AI for business pilots. The assistant sounds fluent. The corpus is a junk drawer. A customer, a sales rep, and an auditor can all walk away with the same confident answer about an offer you killed last quarter.
Ingest less, on purpose
AWS’s walkthrough on preparing public web content for Amazon Bedrock Knowledge Bases puts the constraint in one line:
When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest.
Amazon Bedrock Knowledge Bases is the retrieval layer here. You point it at documents. It parses, chunks, embeds, indexes, and hands passages back to an application, often with a generated answer and citations. On a fully managed knowledge base, Bedrock runs the vector store. On a customer-managed knowledge base, you run the store. The differences show up in metadata, filters, and which API writes the answer. Hybrid search and reranking are separate configuration choices. They depend on the knowledge base you are using. Do not assume one arrives free with the other.
Neither option repairs a bad crawl. If the stored text is navigation chrome, the citation looks official and still wastes the call. If the stored text is last quarter’s price, your sales assistant keeps selling it.
For public sites you are allowed to crawl, start with the managed Amazon Bedrock Web Crawler. Give it seed URLs or a sitemap. Keep it inside a scope and a rate limit. It can wait for JavaScript-rendered pages. It honors robots.txt. After the first full pass, it syncs what was added, updated, or deleted.
Reach for a custom preparation step only when that crawler cannot give you the page you actually want: the main column without the boilerplate, a bounded section of the site, or metadata your filters can trust. That is the narrow job of Crawl4AI, the open-source Python crawler credited to UncleCode, in AWS’s sample sample-crawl4ai-bedrock-knowledge-base. The sample authors are explicit that its architecture and defaults are a starting point, not a requirement of Knowledge Bases.
What you should refuse to crawl
This pattern is a timed ingest of sites you already know. A schedule starts a container task. The task crawls the seeds, writes Markdown and a small metadata file, uploads both to the S3 bucket behind the knowledge base, and starts an ingestion job. Bedrock, not the crawler, then chunks, embeds, and indexes. Your application asks for passages, or for a grounded answer, after that work is done.
This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.
“AI agents” is doing a lot of unpaid work in board slides. Agentic retrieval, the managed path that can both search and draft an answer, is narrower than the slogan and more useful. AWS currently documents it as a planner: a foundation model breaks the question into sub-queries, retrieves, decides whether it has enough, may pull an entire document, and then writes. Citations can mark character spans in that answer and point back at the result list. Generating the answer is on by default. One request can name up to five managed knowledge base retrievers. The API is limited to fully managed knowledge bases. The same walkthrough uses RetrieveAndGenerate when the knowledge base is customer-managed. None of that is a bot browsing the live web while a buyer waits.
The sample processes public pages only and does not implement authenticated sessions. The managed Web Crawler can use basic, form-based, or SAML authentication. Extending the sample would mean session handling, credentials in Secrets Manager, and a security review. Do not point it at private or internal endpoints.
Replace every placeholder before a deploy: knowledge base ID, data source ID, bucket, and seed URL. Then confirm the template cannot bind to a bucket you do not control. The building-permit page used as an example is an illustration, not a dataset.
The failure that looks like a green check
Each scheduled event in the sample launches one standalone task. Nothing in the stack stops a second run from overlapping the first. A failed run is not announced. The sample adds no retry policy, dead-letter queue, or alarm.
The quieter failure is the one operators miss. An ingestion job can report COMPLETE when some documents failed to ingest, and the sample task then exits with code 0 after logging the reasons. A dashboard that watches status codes will call that a healthy night. The corpus is missing pages, and the assistant will not say so.
Uploads are additive. Dropping a URL from the seed list, or watching a page disappear from the site, does not delete the Markdown already in S3 or its metadata sidecar. Last quarter’s price stays searchable until someone deletes both objects and starts ingestion again. If document deletion protection is on, a sync that would delete more than its threshold share of documents skips the delete phase. The walkthrough does not state that percentage. Check it before you assume a cleanup ran.
Tearing down the automation does not delete the knowledge base, the bucket, or the pages already uploaded. Stale content survives the schedule that created it.
You pay for the task while it runs, for logs, for an idle network path if you enabled one, and separately for the knowledge base. If the account blocks public access from the VPC, a task in a public subnet cannot reach the sites at all. A NAT gateway is one way out, and it bills by the hour even when nothing is crawling. That charge is networking. It does not buy you a retrieval feature. The bill is still the easy part. The hard part is naming who owns a citation after the offer is gone.
Chunking changes what the citation quotes
Because this pipeline writes Markdown, a citation can quote far more than the sentence that matched. AWS documents a specific behavior: when content is processed as Markdown and a page contains a table or figure, Knowledge Bases can attach the full page or the full table as broader context, then replace the matching chunk with that context at retrieval. Chunks that share the context come back as one result. You may get fewer hits than you asked for.
That applies to default, fixed-size, semantic, and hierarchical chunking. It does not apply if you chose no chunking. The attachment happens only when the full page or table is 10, 000 characters or fewer. Past that limit, the original matching chunk is what comes back.
The effect cuts both ways, and it lands on the same stale price sheet. A current rate table can make the answer better, because the model sees the whole table. An outdated one comes back looking thorough. Review real pages from your own sites before customers see them. Crawl4AI’s pruning filter is a heuristic meant to drop navigation and keep headings, lists, and tables. Heuristics miss.
Chunking is the choice you cannot casually revise. The walkthrough warns that the strategy is selected when the data source is connected and cannot be changed afterward. Switching means a new data source. Evaluate on your pages before you freeze a production source.
AWS’s chunking documentation describes the default as chunks of approximately 300 tokens, with sentence boundaries preserved. That is a different setting from fixed-size chunking, where you set a maximum token count and an overlap percentage. No chunking stores each file as one chunk. It also drops page-number citations and filtering by page number. Semantic chunking adds foundation-model cost that grows with volume. If you need chunk-level hits on tables, AWS’s guidance is to skip its chunker and split the content yourself. Raising a token limit will not merge text across page or section boundaries the parser already respected.
Metadata size belongs in the same decision. With an S3 vector bucket, attached page or table context counts toward each vector’s metadata-size limit. AWS does not recommend hierarchical chunking there, and combined chunking above 8, 000 tokens can hit those limits. The Crawl4AI walkthrough separately notes that Amazon S3 Vectors can attach up to 1 KB of custom metadata per vector. Confirm the current ceiling before you design around it.
The sample attaches its own fields through a sidecar. For a file named page.md, the metadata file must be named page.md.metadata.json and must sit in the same S3 location. The walkthrough treats Bedrock’s cap on that sidecar as 10 KB. In the sample, a sidecar that would exceed the cap stops the run with an error before either file is written. On the managed S3 data source, leave the metadata-files prefix empty so those sidecars are found beside the documents.
A category such as web, or a crawl timestamp, can keep a web corpus from mixing blindly with other uploads. That is precision. It is not permission.
Guardrails are not an access policy
Metadata filtering can improve retrieval precision, but it is not an authorization boundary.
The managed S3 connector can enforce document-level access with ACL files. The sample does not create them. If an AI for sales tool and a public help bot share a bucket, a retrieval filter will not keep the wrong audience out.
Public pages become model context. Text on a compromised marketing page, or a page someone edited to manipulate the bot, can attempt prompt injection through retrieval. The authors of the walkthrough are direct: do not rely on guardrails alone to catch it.
Confirm the guardrail on the exact API you call. Do not assume a policy configured somewhere else is attached or applied. Current AWS documentation for agentic retrieval says a guardrail is supported only when the foundation model type is set to custom. The managed default, or omitting the type, fails validation. Policies that anonymize personally identifiable information, or that use that anonymize action on regular expressions, are unsupported and fail validation rather than quietly applying. Those are current API rules, not a permanent law of the product. Recheck them against the call you actually make.
Contextual checks that score whether an answer is grounded in the retrieved passages, and relevant to the question, are worth configuring where the API allows them. They do not replace a short list of trusted seeds, citations a person can open, and a human on higher-risk answers. The sample also pins the crawler’s robots.txt check on, with no switch in its deployment path to turn it off. That is a decent baseline. It does not decide which sites you are willing to defend.
Questions worth answering before the first schedule
- Should we build a custom crawler before we try the managed one?
No. If the managed Web Crawler can reach and index the pages you are allowed to use, start there. Crawl4AI earns a place when you need cleaner Markdown, a bounded extract, or metadata the managed crawl will not attach. It prepares documents. It does not replace the knowledge base.
- Is this the same as an AI agent browsing the web while it answers?
No. This is a timed ingest of known public properties into S3, then into a knowledge base. Live web access at inference time is a different feature, with a different risk and audit profile. Agentic retrieval plans queries against what you already ingested.
- Will a metadata filter keep the wrong audience out of a document?
No. Filters can sharpen a search. They are not an authorization boundary. Document-level ACLs are a separate control, and this sample does not create them.
- What happens to a page we remove from the website?
In this sample, nothing, until someone deletes the Markdown file and its matching sidecar and runs ingestion again. The pipeline adds and updates the objects it writes. It does not mirror deletions from the site. Check deletion protection before you assume a sync removed the old pages.
- Does a COMPLETE ingestion job mean every page landed?
Not in this sample. Some documents can fail while the job still reports COMPLETE, and the task can exit successfully after only logging why. There is no alarm. Watch the failure reasons, not just the status.
- Why can a citation look larger than the sentence that matched?
AWS documents this for Markdown that contains a table or figure. The knowledge base may replace the matching chunk with up to 10, 000 characters of page or table context and collapse related chunks into one result. Above that character limit, the original chunk is what comes back. No chunking does not do this replacement.
- Who is accountable when a deleted offer is still cited?
Whoever owns the corpus, not the model. If the stale page is still in the bucket, retrieval will keep handing it over. Name an owner for the seed list, for deletion, and for reviewing citations before a sales or customer workflow treats them as current.
Most companies do not need a cleverer crawler. They need a shorter list of sites they are allowed to trust, a chunking choice they made on purpose, a way to notice a silent failed ingest, and a deletion path that matches how fast their offers change. Freshness without that discipline is just a faster way to cite the wrong page.