Email: The boring stuff that prevents AI disasters

Subject:
The boring stuff that prevents AI disasters
Received:
11/20/2025, 3:59:04 PM
From:
"Matt Pocock (AI Hero)" <matt@aihero.dev>
To:
porch.hat.81@rss.andrealin.it
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;style&gt;.ck-link { text-decoration: underline; }&lt;/style&gt;
&lt;style&gt;.ck-link { text-decoration: underline; }&lt;/style&gt;
&lt;style&gt;@media only screen { .email * { word-break: break-word; } }
@media screen and (max-width: 384px) { .mail-message-content { width: 414px !important; } }
@media only screen { .email * { word-break: break-word; } }
@media screen and (max-width: 384px) { .mail-message-content { width: 414px !important; } }
@media only screen and (max-width:600px) { .ck-mobile-font-size { font-size:50px !important; }  }
@media only screen and (max-width:600px) { .ck-mobile-font-size { font-size:50px !important; }  }
@media only screen and (max-width:600px) { .ck-mobile-font-size { font-size:50px !important; }  }&lt;/style&gt;
&lt;meta http-equiv="Content-Type" content="text/html; charset=UTF-8"&gt;
&lt;/head&gt;
&lt;body style="margin:0;padding:0;-webkit-font-smoothing:antialiased;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%"&gt;
&lt;div class="email" style="margin-left:18px;margin-right:18px"&gt;
&lt;!--[if mso]&gt;&lt;style&gt;.button-table { padding: 12px 0 }&lt;/style&gt;&lt;![endif]--&gt;&lt;table cellpadding="0" cellspacing="0" style="border-collapse:separate;mso-table-lspace:0pt;mso-table-rspace:0pt;width:100%;margin:0 auto"&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td style="vertical-align:top"&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Last time we talked about 5 of the key ways AI systems can go wrong that are distinct to building apps integrated with an LLM vs traditional deterministic software engineering.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Today, let’s talk about the strategies and tools you can use to detect and stop AI disasters &lt;em&gt;before&lt;/em&gt; they strike.&lt;/p&gt;
&lt;h2 class="" style="font-weight:bold;font-style:normal;font-size:1em;margin:0;font-size:1.5em;margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#000000;line-height:1.5"&gt;1. Observability is your key to early warnings AND fixing errors&lt;/h2&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Of course, &lt;strong&gt;all&lt;/strong&gt; good software engineering needs to follow good security practices.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;But when it comes to working with AI, the foundation to spotting problems is &lt;em&gt;Observability.&lt;/em&gt;&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Observability with LLM software gives you powers far beyond basic logging and error handling:&lt;/p&gt;
&lt;ul class="unordered_list" style="margin:1em 0;margin-left:1em;padding:0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5;list-style-position:outside !important"&gt;
&lt;li class="list_item" style="padding:0 0 0 1em;margin:0;margin-left:1em"&gt;&lt;span&gt;it gives you an early warning system for wasted tokens, so you can improve your context engineering. &lt;/span&gt;&lt;/li&gt;
&lt;li class="list_item" style="padding:0 0 0 1em;margin:0;margin-left:1em"&gt;&lt;span&gt;it helps you spot low-quality responses before users do &lt;/span&gt;&lt;/li&gt;
&lt;li class="list_item" style="padding:0 0 0 1em;margin:0;margin-left:1em"&gt;&lt;span&gt;and it’s how you avoid guessing about what’s actually happening between your code and the model, whether it’s in development or production.&lt;/span&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;I’m so bullish on Observability as a foundational skill that I included Langsmith in my &lt;a href="https://click.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5/e0hph0u7qxor7ob8/aHR0cHM6Ly93d3cuYWloZXJvLmRldi93b3Jrc2hvcHMvYWktc2RrLXY1LWNyYXNoLWNvdXJzZQ==" target="_blank" class="ck-link" rel="noopener noreferrer" style="color:#0875c1" url-id="1807416808"&gt;AI SDK Crash Course&lt;/a&gt;, which is included for free in &lt;a href="https://click.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5/owhkhwuwgzdqwmhv/aHR0cHM6Ly93d3cuYWloZXJvLmRldi9jb2hvcnRzL2J1aWxkLXlvdXItb3duLWFpLXBlcnNvbmFsLWFzc2lzdGFudC1pbi10eXBlc2NyaXB0" target="_blank" class="ck-link" rel="noopener noreferrer" style="color:#0875c1" url-id="1807416810"&gt;my “AI Assistant” cohort&lt;/a&gt;. It’s a killer platform for tying all this stuff together!&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;The sooner you have Observability added to your app, the safer you are from problems &lt;em&gt;before&lt;/em&gt; they strike.&lt;/p&gt;
&lt;h2 class="" style="font-weight:bold;font-style:normal;font-size:1em;margin:0;font-size:1.5em;margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#000000;line-height:1.5"&gt;2. Know your tools deeply, front and back&lt;/h2&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Bad UX usually comes from compromise. Backend teams and frontend teams can’t agree. “We can’t do it that way because the backend needs this format” or “our framework only lets us do X.” And so the UI you create ends up built by committee - and doesn’t serve anyone!&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;This is why full-stack frameworks matter. They give you full control of the whole picture, so you can build the UI of your user’s dreams. The AI SDK gives you insanely powerful tools for communicating with users while the LLM is streaming.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;In the cohort, we’ll explore custom data parts, basically a custom RPC that transmits any information from backend to frontend during streaming. You can stream custom UI, display dynamic loading/streaming states, and trigger actions on the frontend. It’s wild what becomes possible.&lt;/p&gt;
&lt;h2 class="" style="font-weight:bold;font-style:normal;font-size:1em;margin:0;font-size:1.5em;margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#000000;line-height:1.5"&gt;3. Manually inspect your data&lt;/h2&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;This strategy can feel tedious, but it’s crucial to the point that you can’t avoid it:&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;&lt;strong&gt;&lt;em&gt;You’ve got to look at your actual data, both what goes into the LLM and what comes out.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;The good news is that you can find or even build handy little tools to help you do this more easily! While building your assistant, we’ll also build a custom chunk viewer so you can manually observe your chunks and see how different retrieval algorithms perform.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;On the data prep side, good chunks = less token waste.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;On the user experience side, good retrieval = higher quality responses and fewer mistakes.&lt;/p&gt;
&lt;h2 class="" style="font-weight:bold;font-style:normal;font-size:1em;margin:0;font-size:1.5em;margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#000000;line-height:1.5"&gt;4. Master systems design&lt;/h2&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;AI engineering stretches my architectural brain like no other programming I’ve done. You’re designing systems, like you do in backend code, but with a capricious, probabilistic AI at the centre.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;The result is that I find myself asking a different kind of question when making design decisions, and maybe more importantly, spending more time on the system design than the implementation.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Sometimes it’s as simple as spending more time thinking about how I break complex tasks into smaller ones that the LLM can actually handle.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;Other times, it’s the mental effort of deciding what parts of a workflow belong to the LLM while others get handled in code.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;In all cases, these kinds of decisions require more systems level thinking. So the better you get at system design, the faster you can validate ideas. The faster you learn what works and what doesn’t!&lt;/p&gt;
&lt;h2 class="" style="font-weight:bold;font-style:normal;font-size:1em;margin:0;font-size:1.5em;margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#000000;line-height:1.5"&gt;Practice these techniques with me in my next cohort&lt;/h2&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;I’ve designed my next cohort course &lt;a href="https://click.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5/owhkhwuwgzdqwmhv/aHR0cHM6Ly93d3cuYWloZXJvLmRldi9jb2hvcnRzL2J1aWxkLXlvdXItb3duLWFpLXBlcnNvbmFsLWFzc2lzdGFudC1pbi10eXBlc2NyaXB0" target="_blank" class="ck-link" rel="noopener noreferrer" style="color:#0875c1" url-id="1807416810"&gt;&lt;strong&gt;Build Your Own AI Personal Assistant in Typescript&lt;/strong&gt;&lt;/a&gt; around learning and practicing all these techniques from the ground up, so your next (or first) AI application follows can be as safe as possible.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;You’ll learn all these strategies in the cohort by building a complicated project from scratch: layering in retrieval, memory, human-in-the-loop workflows, and evals.&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;By the end, you’ll know how to protect your AI systems from disaster. And you’ll have built something real to prove it!&lt;/p&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;If you’re thinking about building &lt;em&gt;anything&lt;/em&gt; with an LLM integration in 2026, and haven’t checked out the course page yet, make sure you do before &lt;strong&gt;Monday Nov 24 at 11:59 PM Pacific when the price goes up by 40%!&lt;/strong&gt;&lt;/p&gt;
&lt;!--[if !mso]&gt;--&gt;&lt;table width="100%" style="border-collapse:separate;mso-table-lspace:0pt;mso-table-rspace:0pt"&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td align="center" style="vertical-align:top"&gt;&lt;a class="email-button" href="https://click.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5/owhkhwuwgzdqwmhv/aHR0cHM6Ly93d3cuYWloZXJvLmRldi9jb2hvcnRzL2J1aWxkLXlvdXItb3duLWFpLXBlcnNvbmFsLWFzc2lzdGFudC1pbi10eXBlc2NyaXB0" target="_blank" rel="noopener noreferrer" style="color:#0875c1;border:0 none;margin-bottom:0.5em;font-size:16px;line-height:1.5;background-color:#0875c1;color:#ffffff;border-radius:4px;border-color:#0875c1;background-color:#0875c1;box-sizing:border-box;border-style:solid;color:#ffffff;display:inline-block;width:auto;text-align:center;text-decoration:none;padding:18px 32px;margin-top:48px;margin-bottom:48px;font-size:24px;border-radius:4px 4px 4px 4px" url-id="1807416810"&gt;&lt;strong&gt;ENROLL TODAY&lt;/strong&gt;&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;!--&lt;![endif]--&gt;&lt;!--[if mso]&gt;&lt;table class="button-table" width="100%" border="0" cellSpacing="0" cellPadding="0" style="margin-top:48px;margin-bottom:48px"&gt;&lt;tr&gt;&lt;td align="center"&gt;&lt;table border="0" cellSpacing="0" cellPadding="0" width="auto"&gt;&lt;tr&gt;&lt;td align="center" bgcolor="#0875c1" style="background-color:#0875c1;overflow:hidden;padding:18px 32px"&gt;&lt;a class="email-button" href="https://www.aihero.dev/cohorts/build-your-own-ai-personal-assistant-in-typescript" target="_blank" rel="noopener noreferrer" style="background-color:inherit;display:inline-block;text-decoration:none;margin:0;color:#ffffff;font-size:24px" data-ck-element="button"&gt;&lt;strong&gt;ENROLL TODAY&lt;/strong&gt;&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;![endif]--&gt;&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;-Matt&lt;/p&gt;
&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;div class="email-footer"&gt;&lt;table cellpadding="0" cellspacing="0" style="border-collapse:separate;mso-table-lspace:0pt;mso-table-rspace:0pt;width:100%;margin:0 auto"&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td style="vertical-align:top"&gt;
&lt;div&gt;&lt;div class="email-footer" style="color:#4b4b4b"&gt;
  &lt;p style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5;color:#4b4b4b; margin:1em 0;font-family:Arial,-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;;font-size:12px;line-height:1.5;display:block;margin:13px 0;"&gt;&lt;a href="https://click.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5/vqhk9p90bmuoe5rwoztg/aHR0cHM6Ly93d3cudG90YWx0eXBlc2NyaXB0LmNvbS91bnN1YnNjcmliZWQ_c2VxdWVuY2U9YWlfaGVybw==" trigger-id="5153768" style="color:#0875c1" url-id="1807416811"&gt;Unsubscribe from AI Hero Emails&lt;/a&gt;. You can also &lt;a href="https://unsubscribe.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5" style="color:#0875c1"&gt;&lt;span style="font-size:12px"&gt;unsubscribe&lt;/span&gt;&lt;/a&gt;&lt;span style="font-size:12px"&gt; from all emails at any time.&lt;/span&gt;&lt;/p&gt;
  &lt;p style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5;color:#4b4b4b; margin:1em 0;font-family:Arial,-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;;line-height:1.5;display:block;margin:13px 0;"&gt;&lt;span style="font-size:10px"&gt;12333 Sowden Rd, Ste. B, PMB #97429, Houston, TX 77080&lt;/span&gt;&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p class="" style="margin:1em 0;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',sans-serif;color:#2d2d2f;font-size:16px;line-height:1.5"&gt;&lt;/p&gt;
&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;!-- --&gt;
&lt;img src="https://open.convertkit-mail2.com/zlu5lqnvkkunhkwn9ezhphwogmg00a6hkknn5" alt="" style="border:0 none;display:block;height:auto;line-height:100%;outline:none;-webkit-text-decoration:none;text-decoration:none;max-width:100%"&gt;
&lt;/body&gt;
&lt;/html&gt;
Email Content
