Merge pull request #79 from cmpadden/post/evolution-of-data-engineering

post: The evolution of data engineering

Author: cmpadden

content/articles/ai-the-context-engineer.md

@@ -0,0 +1,40 @@
+---
+title: 'The evolution of data engineering'
+draft: true
+date: "2025-07-18"
+tags: ["ai", "data"]
+categories: ["programming"]
+---
+
+With the advancements of LLM-based tooling, the responsibilities (and maybe even the title!) of data engineers is changing.
+
+## Origin
+
+Before we talk about the changes, let's reflect on the inception of data engineering.
+
+The practice of data engineering, and the need for dedicated roles, was born out of complexity, and the need for someone to manage it.
+With an explosion in data volumes, and an increase in tools to manage it, a domain-specific engineer was born with a specialized set of skills.
+Some of these engineers came from the world of database administration, others from software engineering, and some from analytics.
+As the domain that they worked in evolved, so did their responsibilities and skill-sets.
+
+## Questions
+
+But, while the tools and challenges that these engineers solved changed, the underlying need for their work remained the same: to support the answering of questions.
+
+- "Why do I have no customers?"
+- "What is the significance of these protein structures?"
+- "What is the airspeed velocity of an unladen swallow?"
+
+The wrangling of data, storage, processing, and everything in-between was done out of necessity so that you, or some stakeholder, could make sense of it.
+
+## Context
+
+We're reaching another inflection point where the underlying tools are evolving.
+This time, we're seeing an emergence of systems that can make sense of large volumes of data, however the new complexity is that these systems rely on carefully managed metadata, or _context_.
+This context can store how data is loaded, how it's transformed, and how it's queried.
+You no longer have to write the tool-specific code to interface with various APIs, nor the queries to make sense of it.
+You just have to ensure that the information is present and curated.
+
+The new responsibility of this kind of engineer is to manage that context.
+
+Some folks are speculating that we will see the rise of the "context engineer", but while it's uncertain what the title will be, it's clear that we are seeing a change in how data engineers operate, and an evolution in the responsibilities that they hold to support the answering of questions.

pages/articles/[...slug].vue

@@ -4,18 +4,10 @@ const route = useRoute();
 const { data: page } = await useAsyncData(route.path, () => {
   return queryCollection("content").path(route.path).first();
 });
-
-//// Redirect missing articles to `/`, or defined redirects to their intended location
-//if (page.value === null) {
-//  await navigateTo({
-//    path: "/",
-//  });
-//}
 </script>
 
 <template>
-  <div class="container mx-auto text-gray-700 dark:text-white" v-if="page">
-    <article class="mb-10 space-y-6 shadow-sm dark:shadow-lg">
+    <div class="container mx-auto max-w-[1024px] text-gray-700 dark:text-white mb-10 space-y-6 shadow-lg" v-if="page">
       <!-- title -->
       <div class="flex">
         <!-- https://content.nuxt.com/components/content-slot -->
@@ -69,12 +61,11 @@ const { data: page } = await useAsyncData(route.path, () => {
         - Use prose-pre:bg-white to work with @nuxt/content syntax highlighting, otherwise background-color defaults to `.prose:where(pre)`
       -->
       <article
-        class="prose max-w-none text-gray-700 dark:text-gray-300 prose-h2:mt-8 prose-a:font-bold prose-a:text-orange-400 prose-a:no-underline hover:prose-a:text-orange-500 prose-blockquote:text-gray-600 dark:prose-blockquote:text-gray-400 prose-code:text-gray-900 dark:prose-code:text-white prose-pre:bg-gray-100 dark:prose-pre:bg-black"
+        class="prose max-w-[1024px] text-gray-700 dark:text-gray-300 prose-h2:mt-8 prose-a:font-bold prose-a:text-orange-400 prose-a:no-underline hover:prose-a:text-orange-500 prose-blockquote:text-gray-400 prose-code:text-white prose-pre:bg-black prose-p:text-justify"
       >
         <ContentRenderer v-if="page" :value="page" />
       </article>
-    </article>
-  </div>
+    </div>
 </template>
 
 <style>