Modernize your Data

A working definition of Data Architecture is a managed ecosystem for identification and harnessing of data artifacts and transforming them into business-consumable assets. While some aspects of this have existed for many years, new techniques, technologies and tools have both necessitated and enabled a more modern approach to data architecture. The reasons for this imperative are numerous, but the most important include:

1. The volumes of data being generated today require new approaches to store and retrieve effectively.
2. The variety of data that are observed and of interest presents unique challenges for effective use in any meaningful way.
3. The velocity at which data are created is increasing, as the time desired to derive actionable and competitive insight is decreasing.

It is these challenges that are informing the desire to create and manage a more appropriate, performant and functional data architecture.

The proper way to view a data architecture is one that can apply to all of the data, not just within individual silos or disparate platforms. That is not to say those things will not exist in an ecosystem, but the data architecture should accommodate a more holistic view. This means that there is an awareness and control of the many different kinds of events and data that need to be collected, inventoried and understood. Harnessing this data means that there are techniques employed that will allow for tracking the formats, lineage and usages of the data as it exists at the source and as it moves through transformation layers and is eventually consumed by the business.

An important concept in a Data Architecture is the idea of data artifacts. In the not-so-distant past, data was only considered part of the architecture if it was stored in persistent, usually relational database systems. Today’s data arrive with a much broader velocity and variability that demands a different perspective. With so much data being generated by real time events and processes, it is important to determine which data has the most business value and focus on these artifacts in the delivery of business insight. There are 4 broad categories of data that should be persisted to service this need:

• Transitory - This is data persisted only long enough to perform a task.
• Volatile - Useful data but it can be rebuilt at any time from other sources.
• ACID - This is the traditional relational data that provide atomicity, consistency, isolation and durability which is also transactionally consistent.
• Amorphous - These newer artifacts can include NoSQL, Raw objects, properties, and application-consumable formats like JSON and XML.

The successful outcome of this process should be measured by the ease with which end users can take action on the insight present in the data that they have available for consumption. It is not enough to just be able to view a multitude of pretty charts and graphs; The data must also have the ability to inform the adaptation of strategies and processes within the organization in response to what is surfaced from within the data architecture. Without this relationship between events, insight and action, the data architecture will not support the intended aspirations of the organization.

The defining aspect of a modern data architecture is that it can adapt to changing technologies and tools as they become available. Today we are talking about concepts like decentralization, elasticity, data lakes, cloud analytics, containerization and many others that didn’t exist 5 years ago. To accommodate this persistent change there must be an ongoing process of exploration and evaluation of new technologies that could be applied within the data architecture. This commitment to innovation is one of the critical success factors for companies that want to lead (or even influence) their clients to a more efficient and profitable architecture.