jaykim88/new-tech-evaluation

New Tech and AI Integration Evaluation

Purpose

Decide whether to adopt a new library / framework version / AI capability with evidence — bundle impact, type safety, maintenance health, security, migration cost — not hype. Default to skepticism: proven "boring" tech is the baseline, and each new dependency spends an innovation token — the burden is on the candidate to beat what you'd otherwise write or already use.

Universal — evaluation rubric (size / types / maintenance / migration / license / a11y) applies to any frontend stack; size-check tools differ.

Procedure

New library evaluation

1. Measure bundle impact first

   • Primary: Bundlephobia for the candidate
   • Fallback when Bundlephobia data is stale or missing (a known issue since ~2024): pkg-size.dev or npx vite-bundle-visualizer against a real install
   • Look at min+gzip (what users actually download), NOT raw size
   • Check tree-shakability — a 200KB library that tree-shakes to 8KB beats a 30KB monolith
   • Gate library adoption on size delta in PR review

2. TypeScript support quality

   • First-class TypeScript (types shipped with library) >> @types/* package >> @ts-ignore needed
   • Check: do the types actually represent runtime behavior, or are they any underneath?

3. Maintenance health

   • Last release date (< 6 months)
   • Open issue count + median response time
   • Number of contributors (bus factor > 1)
   • Major-version stability (breaking changes every 6 months = high migration cost)
   • Adoption momentum: downloads/week trend (not absolute), is it the de-facto choice or a niche bet? A well-maintained library the community has moved away from is still a risk

4. Migration cost (build vs borrow) — and exit cost

   • Hours to integrate
   • Hours to migrate existing code (if replacing something)
   • Hours of ongoing maintenance (updates, breaking-change handling)
   • Exit cost, not just entry cost: how hard is it to remove later? A thin wrapper is cheap to swap; something that metastasizes through the codebase (ORM, state lib, styling system) is high lock-in — weight that as a one-way-door decision (see decision-records)
   • Compatibility: supported React/Node versions, peer-dep conflicts, and Server Component support — a client-only library forces 'use client' and drags its subtree into the client bundle (see bundle-optimization)
   • Compare to "just write it ourselves" — sometimes 200 lines is cheaper than a dep

4b. License compatibility

• Safe for commercial use: MIT, Apache 2.0, BSD-2-Clause, BSD-3-Clause, ISC
• Caution required: LGPL (linking restrictions), MPL (file-level copyleft)
• Avoid for closed-source products: GPL, AGPL — viral copyleft
• Check transitive deps too (a dual-licensed top-level dep can still pull GPL deps)

4c. Accessibility (for UI libraries)

• Keyboard navigation works out of the box?
• ARIA attributes correct?
• prefers-reduced-motion honored?
• Non-negotiable for UI libraries — a "great DX" component that fails a11y becomes tech debt fast

4d. Security & supply-chain

• Known vulnerabilities: npm audit / Snyk / OSV against the candidate and its transitive tree
• Maintainer trust: recent ownership transfer, a typosquatted name, or a lone unverified maintainer = supply-chain risk
• Each dependency is attack surface and install-time code (postinstall scripts) — fewer, well-vetted deps beat many convenient ones (see security-audit)

5. POC code

   • Use the candidate in an actual project pattern, not a toy example
   • Benchmark against current solution if replacing one
   • Document what worked and what didn't

6. Document decision in ADR (see decision-records skill for template — MADR recommended for 3+ alternatives)

   • Even rejections deserve an ADR — saves the team from re-evaluating the same library in 6 months

React / Next.js major upgrade

1. Read the official migration guide thoroughly
2. Run the codemod (Next.js ships codemods for major version bumps)
3. Audit deprecated APIs in build output
4. Update one feature area at a time, ship incrementally

AI integration evaluation

1. Streaming UI patterns

   • Server Action returns ReadableStream → render progressively via useChat or custom reader
   • Loading state shows partial output as it arrives (don't block UI on complete response)

2. 3-state handling for AI responses

   • Streaming: progressive render + visible "AI is thinking" indicator
   • Complete: final state with regenerate button
   • Failed: error message + retry + fallback path

3. Human-in-the-loop gates

   • For high-stakes AI output (financial, legal, medical, code-deploy)
   • Always show the AI output for review before applying
   • Audit trail: who approved, when, what input produced it

4. Graceful degradation

   • AI API down? App should keep working via non-AI flow
   • Never make the AI a single point of failure
   • Cache previous AI responses where it makes sense

5. Cost monitoring

   • Track tokens per session, per user
   • Alert on cost spikes (often signals a prompt-injection or loop bug)

6. Trust boundary: protect what goes in, distrust what comes out

   • Prompt injection: untrusted content in the prompt can hijack instructions — don't interpolate user/third-party text into a system prompt unguarded
   • Treat AI output as untrusted input: never dangerouslySetInnerHTML it or run it as code/SQL without validation — it's an XSS/RCE vector like any user input
   • Data governance: user data sent to a third-party API leaves your boundary — scrub PII, check data-retention / training opt-out and region/compliance
   • (see security-audit)

7. Pin the model + add an eval harness

   • Pin the model / API version — outputs drift across versions, so an "upgrade" can silently regress your feature
   • You can't assert exact strings: build an eval set (golden cases, LLM-as-judge) to catch quality regressions (see test-strategy)

Completion Criteria

• [ ] POC code exists for the candidate, not just docs reading
• [ ] Bundle impact measured (Bundlephobia + actual analyzer)
• [ ] Security/supply-chain checked (npm audit/OSV clean, maintainer trust); Server Component compatibility verified
• [ ] Exit cost / lock-in assessed, not just integration cost
• [ ] Decision documented in ADR (adopt or reject — both deserve documentation)
• [ ] For AI integration: 3-state UI implemented, HITL gate for high-stakes output, graceful degradation verified
• [ ] For AI: output treated as untrusted (not rendered/executed unsanitized); model version pinned + eval harness in place

Output

• POC code: branch poc/<library-or-feature-name> with realistic usage (not toy example), benchmark script in scripts/poc-benchmark-<name>.ts
• Evaluation report: docs/evaluations/<library-or-feature>-YYYY-MM-DD.md with sections:
  • ## Bundle impact (Bundlephobia + actual analyzer numbers)
  • ## TypeScript support (first-class / @types / @ts-ignore needed)
  • ## Maintenance health (last release, contributors, issue response)
  • ## Migration cost (estimated hours)
  • ## License compatibility (MIT / Apache / GPL / etc.)
  • ## A11y (if UI library)
  • ## Verdict (adopt / reject / re-evaluate in N months)
• Decision ADR: docs/adr/ADR-NNN-adopt-<library>.md (or reject-) — even rejections deserve documentation
• AI integration only: streaming UI implementation, HITL gate code, graceful degradation fallback

Implementation

React + Next.js (default)

• Size check: Bundlephobia / pkg-size.dev / npx vite-bundle-visualizer
• Security/supply-chain: npm audit / OSV-Scanner / Snyk; check the candidate's transitive tree
• Server Component compat: does the lib work without 'use client'? Client-only libs inflate the client bundle (cross-check bundle-optimization)
• React/Next.js major upgrade: official codemods (npx @next/codemod)
• AI streaming: Server Action returning ReadableStream + useChat (Vercel AI SDK) or custom reader
• AI evals: pin the model id; golden-set or LLM-as-judge eval suite for regression (cross-check test-strategy)
• Build-time check: next build warns on deprecated APIs

Other stacks

• Vue / Nuxt: same Bundlephobia; Nuxt has its own codemods for major upgrades
• SvelteKit: same Bundlephobia; SvelteKit migration guides per release
• Angular: same Bundlephobia; Angular's ng update handles major version upgrades automatically (best-in-class)
• AI streaming (any framework): ReadableStream is a web standard — adapter pattern works across stacks; differences are in how the framework's streaming primitives consume it

Related skills

• decision-records — every adoption decision results in an ADR; weigh adoption as a one-way vs two-way door
• bundle-optimization — size delta + Server Component compatibility are key adoption gates
• security-audit — dependency supply-chain risk and treating AI output as untrusted
• test-strategy — eval harness for non-deterministic AI features

Reference

• Key insight encoded: Always compare on min+gzip + tree-shakability — a 200KB library that tree-shakes to 8KB beats a 30KB monolith. Gate library adoption on Bundlephobia size delta in PR review, and check Server Component compatibility (client-only libs balloon the bundle). Evaluate exit cost and supply-chain risk, not just entry cost — and default to boring/proven tech. For AI integration, the HITL gate is mandatory for high-stakes output (review + audit trail), AI is never a single point of failure, and crucially: treat AI output as untrusted input (don't render/execute it unsanitized) and pin the model behind an eval harness so version drift doesn't silently regress quality.