Lean Development of Data Products

DataOps, a fusion of "Data" and "Operations," addresses the challenges of developing data products by combining principles from Agile, DevOps, and Lean Manufacturing. It emphasizes collaboration, automation, and efficiency in handling data pipelines, enabling teams to deliver data products faster and with higher quality (Atwal, 2020, p.xxiii). If you are interested in the devinition of DataOps take a look on the following blog post: What is DataOps? Introduction to Streamlining Data Product Development

The exponential growth of data in recent years has created both incredible opportunities and formidable challenges for organizations. The ability to harness data efficiently and translate it into actionable insights has become a cornerstone of competitive advantage. However, traditional methodologies for example to develop software product often fail to keep up with the speed and complexity of data products development.

---

DataOps inherits components from (DataKitchen, 2018):

• Lean Manufacturing: Focusing on the value adding processes by eliminating waste leads to a more efficient utilization of resources, with higher quality and lower costs.

• Agile: Building the right product for the right people by increasing “the ability to react to unforeseen or volatile requirements regarding the functionality or the content“ of Data Products (Zimmer et al., 2015)

• DevOps: Shared commitment towards the Data Product reduces information barriers (Culture of collaboration), automation (e.g., CI/CD pipelines), Infrastructure as Code and automated tests enable fast and reliable deployment of code into production in a high quality (Macarthy & Bass, 2020b)

Lets Focus on Lean Manufacturing:

Lean Manufacturing:

Lean manufacturing is a production philosophy focused on maximizing customer value by eliminating waste and continuously improving processes. Originating from the Toyota Production System, it emphasizes principles such as standardization, continuous flow, and employee involvement to optimize efficiency and quality. By prioritizing value-adding activities and minimizing non-value-adding processes, lean manufacturing aims to deliver high-quality products at lower costs with shorter lead times. Lets have a deep dive into the evolution of production systems:

Definition and Evolution of Production Systems:

Production systems can be characterized in reference to system elements (technical production systems or social production systems), to the viewpoint (real system or conceptional system), as well as to the considered value adding processes in the production system (Baumgärtner, 2006, p.34). The system element characteristics describe the role of the worker in the production system. The technical production system is focusing on technical and infrastructure elements and the worker is only a productive factor. The social production system includes all entities inclusive the workers with their know-how, skills, needs and values and focusing on the organization and cooperation between worker and other productive factors as a social-technical system (Baumgärtner, 2006). The description of social-technical systems adds a new dimension of complexity in comparison to the technical production system. Furthermore, production systems can also be characterized from a different viewpoint. Real production systems describe the production as a transformation of raw materials to a final product (Uygun, 2013, p.13). On the other hand, the conceptional production viewpoint is focusing on design principles, methods, and tools. Production systems can be characterized furthermore by their specific value adding activities like development, supply chain, quality control, production etc. Production systems are characterized by different dimensions: technical transformation from raw material physically transformed into a product, employee focus and their skills and capability, or as a methodic framework for principles, methods and tools. The interaction of those dimension is highly complex and evolved through the years. There were three major changes of paradigms.

A change of paradigm describes a fundamental change of the properties and framework of production systems. Till 1920 manual manufacturing was the dominant form of production systems. Not standardized unique pieces were produced without mechanical automation resulting in a low productivity and individually made products (Baumgärtner, 2006, p.18; Dombrowski & Mielke, 2015, p.8). No economic of scale can be applied to produce unique individual products resulting in small volumes of manufactured goods. Due to a lack of division of labor, the workers had to master every process step of the production, so that only very well-trained employees could manufacture complex products. The number of available educated employees was limited in the beginning of the 20th century (Dombrowski & Mielke, 2015,p. 8). In the beginning of the 21st century electricity, steam engine and new construction tools were found and developed (Crespo,2012, p.30). These developments were the basis for the classical mass production systems from Henry Ford and Taylor. The prices for produced goods decreased with the utilization of economic of scale. This resulted in higher sales since many products – like cars – were affordable for the middle class triggering an economic boom (Dombrowski & Mielke, 2015,p. 8). Key component of the classical mass production is the complexity reduction of the production achieved by separating planning and executive activities, as well as the division of labor and standardization in production. This division of labor and standardization enables the employment of non-skilled workers for repetitive tasks resolving the resource restriction of educated workforce (Dombrowski & Mielke, 2015,p.8; Uygun, 2013, p.11). The classical mass production is ideal for the production of goods with low variability in a huge number. In the beginning of the classical mass production the market was a seller market. The main goal was to meet the demands. The classical mass production system came to its limits when the market changed to a buyers’ market demanding for more variety of goods offered for a cheaper price and was replaced by the Lean Production system which was significantly influenced by the Toyota production system (TPS). The Toyota Production System (TPS) was developed by the Toyota car company to relieve the difficult economic conditions in Japan after World War II and the oil crisis 1973. The Japanese market required the production of a wide portfolio of vehicles at low prices (Ohno, 2013, p.34). After the Second World War, Toyota had little capital resources. In the classical production system specialized production equipment for each variant of the product was required. These specialized production machines are inflexible in use and expensive (Womack et al., 1991, p.35). Furthermore, high working capital is needed for the classical mass production for buffering semifinished goods. High buffer volumes correspond to high cycle times (Dombrowski & Mielke, 2015,p.15). Toyota could not afford to copy the classical mass production system. The goal of the Toyota production system is to achieve at least the same level of productivity as that of the classic mass production with lower economic of scales and lower cost, higher quality and a shorter time to market (Baumgärtner, 2006, p.26; Merl, 2015, p.18). The Toyota production system was a key success factor to develop and produce with less resources and cost in a shorter period in a high quality new variants of products to increase the market share and customer satisfaction (Womack et al., 1991). The success of the Toyota production system was the fundament for lean production systems. Lean production system can be defined as “production system that focusing continuous flow within supply chain by eliminating all wastes and performing continuous improvement towards product perfection” (Rose et al., 2011). Non value adding processes (Muda) need to be eliminated and value enabling processes to be optimized to focus on the value adding processes which deliver the value for the customer. Non value adding processes are processes producing waste (MUDA) (Dombrowski & Mielke, 2015, p.32). Seven different types of waste exist: Transport, Inventory, Movement, Waiting, Overproduction, Overprocessing, Defects (TIMWOOD) (Ohno, 2013). The nine lean principles summarize the guiding ideas of lean: Standardization, right first time, flow principle, pull principle, continuous improvement, employee orientation and goal-oriented leadership, avoidance of waste and visual management (VDI, n.d.). A multitude of methods and tools can be applied for each of these principles. The six most important principles are explained in the following:

• Standardization ensures a defined level of quality and enables stable and plannable processes (VDI, n.d.). Standardization forms the basis for a continuous improvement process (Dombrowski & Mielke, 2015, p.66).

• The goal of right first time is to reduce the introduction of errors to the next process step. The detection of failures should be detected as early as possible to prevent wasteful reworks in subsequent steps.

• The flow principle is characterized by a fast, continuous and low-turbulence flow of materials and information across the entire value chain (VDI, n.d.). The optimum of the flow principle is a lot size of one – also known as one-piece flow. The goal is to minimize the traveling distances, the waiting times and the buffer between process steps. Thereby the throughput time and the working capital is reduced significantly (Dombrowski & Mielke, 2015, p.96).

• Continuous improvement describes a corporate culture that strives for improvements in the company and is anchored in all employees. The aim is, to question historically grown workflows. Opportunities for improvement are identified by employees. Identified improvement potentials can be summarized in a corporate proposal system. The PDCA cycle (Plan, Do, Check, Act) cycle helps to implement the optimization and SDCA (standardize, do, check, act) cycle helps to standardize the optimization (Dombrowski & Mielke, 2015, p.50).

• Reducing and eliminating waste is a core principle of Lean to increase quality and value creation and reduce the costs (see above).

• Visual management refers to a graphical representation of information (e.g. workflows and performance KPIs). Complex production systems can be made transparent and understandable. This increases the motivation of employees to enhance the performance, to focus on continuous improvements and to identify and eliminate waste (Dombrowski & Mielke, 2015, p.149).

The text explores the evolution of production systems, focusing on their transformation from manual manufacturing to classical mass production and finally to lean manufacturing. Lean manufacturing, derived from the Toyota Production System, emphasizes eliminating waste (Muda), optimizing value-adding processes, and achieving continuous improvement. Key principles of lean include standardization, right-first-time quality, flow, pull systems, waste reduction, and visual management. These principles enable companies to produce high-quality products efficiently, reduce costs, and respond to customer demands for variety and customization. Lean manufacturing represents a shift toward a customer-centric, resource-efficient production model designed to enhance value creation and competitiveness.

The following blog post describes how the lean manufacturing methodology is applied in DevOps and DataOps:

Sources:

• Atwal, H. (2020). Practical DataOps. In Practical DataOps. Apress. https://doi.org/10.1007/978-1 4842-5104-1

• DataKitchen. (2018). DataOps is NOT Just DevOps for Data. https://medium.com/data-ops/dataops is-not-just-devops-for-data-6e03083157b7

• Zimmer, M., Kemper, H., & Baars, H. (2015). The impact of Agility Requirements on Business intelligence Architectures.

• Macarthy, R. W., & Bass, J. M. (2020a). An Empirical Taxonomy of DevOps in Practice. 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 221–228. https://doi.org/10.1109/SEAA51224.2020.00046

• Baumgärtner, G. (2006). Reifegradorientierte Gestaltung von Produktionssystemen. Theoretische und empirische Analyse eines Gestaltungsmodels.(Uygun, 2013, p.13)

• Dombrowski, U., & Mielke, T. (2015). Ganzheitliche Produktionssysteme. Aktueller Stand und zukünftige Entwicklungen. Springer Vieweg (VDI-Buch).

• Crespo, I. (2012). Ganzheitliche Produktionssysteme für kleine und mittlere Unternehmen.

• VDI. (n.d.). VDI 2870: Blatt 1: Ganzheitliche Produktionssysteme Grundlagen, Einführung und Bewertung.

• Ohno, T. (2013). Das Toyota-Produtionssystem.

• Womack, P., Roos, D., & Jones, T. (1991). The machine that changed the world.