Skip to content

Roadmap Update Discussion & Recommendations

Analysis of proposed changes based on real-world integration feedback
Decision framework for prioritizing user needs vs product focus

🎯 Executive Summary

The integration feedback reveals a critical gap between our current roadmap and real-world production needs. While our planned features are valuable, we're missing fundamental usability requirements that block adoption.

Key Decision: Should we prioritize user-requested configuration flexibility over our planned innovation features?

Recommendation: Yes - User adoption requires addressing configuration gaps immediately, but we can do this while maintaining our innovative edge.

🚨 Critical Issues That Demand Immediate Action

1. DataFrame Orientation Bug (URGENT)

  • Status: Core documented functionality doesn't work
  • Impact: Breaks user trust, blocks current adoption
  • Decision: MUST fix in v0.2.5 - non-negotiable
  • Timeline: <48 hours from identification

2. Output Type Inflexibility (HIGH PRIORITY)

  • Status: Fundamental architectural limitation
  • Impact: Forces users to write wrapper functions, blocks production use
  • Decision: MUST address in v0.3.0 - adoption blocker
  • Scope: Requires API design changes but maintains backwards compatibility

🤔 Debating the Proposed Changes

FOR the Proposed Changes

✅ Real User Validation

  • Feedback comes from actual production integration attempt
  • User tested 16 different functions - comprehensive evaluation
  • Specific pain points with concrete examples
  • Financial ML use case represents high-value enterprise segment

✅ Maintains Core Principles

  • Zero new dependencies for all proposed features
  • Performance-first approach with memory optimizations
  • Configuration-driven approach aligns with existing architecture
  • Still focused on ML/data workflows (our unique strength)

✅ Accelerates Adoption

  • Addresses immediate blockers preventing current use
  • Provides migration path from existing utilities
  • Enterprise-ready features (redaction, audit trails)
  • Domain-specific configurations reduce onboarding friction

AGAINST the Proposed Changes

⚠️ Scope Creep Risk

  • Adding v0.2.5 delays planned v0.3.0 features
  • Configuration complexity could make API harder to use
  • Multiple output types increase testing surface area
  • Auto-detection features add "magic" that may not be reliable

⚠️ Dilutes Innovation Focus

  • Time spent on configuration could delay pickle bridge
  • Risk of becoming "yet another JSON library" instead of "ML-first"
  • Advanced ML types (our competitive advantage) gets delayed
  • Snapshot testing (unique value prop) moves further out

⚠️ Feature Request Trap

  • One user's requests may not represent broader community
  • Adding features reactively instead of strategically
  • Risk of building for the loudest user instead of largest market
  • Configuration options can lead to decision paralysis

Tier 1: Must Have (Non-Negotiable)

These block adoption and must be addressed:

  1. Fix DataFrame orientation bug (v0.2.5)
  2. Rationale: Documented functionality must work
  3. Risk: Zero - this is a bug fix

  4. Impact: Restores user trust

  5. Basic output type flexibility (v0.3.0)

  6. Rationale: Users need datetime objects and list outputs
  7. Risk: Low - well-defined scope
  8. Impact: Enables production adoption

Tier 2: Should Have (High Value)

These provide significant user value:

  1. Complete output type control (v0.3.0)
  2. Rationale: Covers all major use cases for configuration
  3. Risk: Medium - increases API surface

  4. Impact: Replaces need for user wrapper functions

  5. Type metadata support (v0.3.0)

  6. Rationale: Enables round-trip scenarios immediately
  7. Risk: Low - optional feature
  8. Impact: Bridges gap until v0.4.5 template system

Tier 3: Nice to Have (Strategic)

These improve user experience but aren't blockers:

  1. Auto-detection deserialization (v0.3.5)
  2. Rationale: Improves developer experience significantly
  3. Risk: Medium - heuristics may fail

  4. Impact: Reduces cognitive load for users

  5. Domain-specific presets (v0.5.0)

  6. Rationale: Real user requested financial_config, time_series_config
  7. Risk: Low - builds on existing system
  8. Impact: Reduces onboarding time for specific domains

Tier 4: Future Consideration (Post-v1.0)

These add complexity without clear ROI:

  1. Chunked processing (maybe v0.4.0)
  2. Rationale: User requested, complements streaming
  3. Risk: High - adds significant complexity

  4. Decision: Evaluate based on additional user demand

  5. Schema-based deserialization (REJECT)

  6. Rationale: User requested but conflicts with our principles
  7. Risk: High - moves toward schema validation
  8. Decision: Template system in v0.4.5 addresses this need

📊 Impact vs Effort Analysis

High Impact, Low Effort (Do First)

  • Fix DataFrame orientation bug
  • Add datetime_output="object" option
  • Add series_output="list" option
  • Add check_if_serialized performance skip

High Impact, Medium Effort (Do Next)

  • Complete output type configuration system
  • Type metadata (include_type_hints) support
  • Basic auto-detection deserialization
  • Financial/time-series configuration presets

Medium Impact, High Effort (Consider Later)

  • Chunked processing for large DataFrames
  • Advanced auto-detection with ML heuristics
  • Cross-configuration compatibility testing
  • Streaming optimizations for multi-GB objects

Low Impact, Any Effort (Don't Do)

  • Schema-based deserialization
  • Complex business logic integration
  • Enterprise features (auth, monitoring)
  • Plugin system architecture

🎯 Final Recommendations

Immediate Actions (v0.2.5)

Timeline: Release within 1 week

# Critical fixes only
config = SerializationConfig(
    dataframe_orient="split",     # FIX: Actually work as documented
    datetime_output="object",     # NEW: Return datetime objects
    series_output="list"          # NEW: Return lists instead of dicts
)

# Performance optimization
result = datason.serialize(data, check_if_serialized=True)

Justification: These are the minimum changes needed to unblock current users.

Enhanced v0.3.0 Scope

Timeline: Original v0.3.0 timeline maintained

# Complete configuration flexibility
config = SerializationConfig(
    datetime_output=Literal["iso_string", "timestamp", "object"],
    series_output=Literal["dict", "list", "object"],
    dataframe_output=Literal["records", "split", "values", "object"],
    numpy_output=Literal["python_types", "arrays", "objects"]
)

# Type metadata for round-trips
serialized = datason.serialize(data, include_type_hints=True)

# Pickle bridge (original plan)
json_data = datason.from_pickle("model.pkl", safe_classes=["sklearn"])

Justification: Combines user-critical features with planned innovation.

Strategic Enhancements (v0.3.5+)

Timeline: Maintain original roadmap timeline

  • Auto-detection deserialization - improves UX significantly
  • Domain presets expansion - financial_config, time_series_config
  • Advanced ML types - maintain competitive advantage
  • Template-based round-trips - complete the portability story

What We DON'T Change

  • Core principles - zero dependencies, performance first
  • Innovation focus - pickle bridge, advanced ML types, snapshot testing
  • Timeline - no major delays to planned features
  • Scope - still ML/data focused, not general-purpose JSON

🤝 Risk Mitigation Strategies

Configuration Complexity Risk

  • Mitigation: Provide sensible defaults and domain presets
  • Testing: Comprehensive test coverage for all configuration combinations
  • Documentation: Clear examples for each output type option

Feature Creep Risk

  • Mitigation: Strict criteria for new features (must solve production blocker)
  • Process: All new features require real-world user validation
  • Governance: Regular roadmap reviews to prune unused features

Innovation Dilution Risk

  • Mitigation: Maintain 70/30 split between innovation and user requests
  • Focus: Configuration work enables our innovative features
  • Value Prop: Better configuration makes ML features more accessible

💡 Key Insights

What This Feedback Teaches Us

  1. Configuration flexibility is table stakes - Users need output control
  2. Bug fixes are adoption blockers - Working features matter more than new features
  3. Production readiness matters - Round-trip capabilities are critical
  4. Domain expertise is valuable - Financial/time-series presets requested

What We Should Maintain

  1. Zero dependency principle - User explicitly values this
  2. Performance focus - User tested and appreciates this
  3. ML-first approach - User chose us because we handle ML objects
  4. Clear, readable JSON - User values human-friendly output

Strategic Positioning

  • Not a general JSON library - we're ML/data specialized
  • Not a schema validator - we're a serializer with smart defaults
  • Not an enterprise platform - we're a reliable tool with safety features
  • The ML serialization solution - others can't handle what we handle

🎯 Conclusion

Adopt the proposed changes with strategic discipline:

  1. Fix critical bugs immediately (v0.2.5)
  2. Add essential configuration flexibility (v0.3.0)
  3. Maintain our innovation roadmap (v0.3.5+)
  4. Stay focused on ML/data workflows

This approach addresses user blockers while preserving our unique value proposition and innovation timeline.

The result: datason becomes both immediately useful for production workflows AND maintains its competitive edge in ML serialization.

Decision Record: June 1, 2025
Next Review: After v0.2.5 user feedback