FULLY AUTOMATED DATA WAREHOUSE FRAMEWORK USING ETL PROCESS FOR DECISION SUPPORT SYSTEM
Keywords:
ETL, Latency,, Data WarehouseAbstract
Consider data comparison tools, ETL testing frameworks, and data visualisation tools when choosing tools for your data warehouse architecture, ETL framework, and testing requirements. These tools are all important to consider. An assortment of data comparison tools, including Informatica Data Validation Option, Talend Data Quality, and SQL Server Data Tools, can assist in the identification of inconsistencies or mistakes that exist between the source data sets and the target data sets. To automate ETL test cases, scenarios, and workflows, ETL testing frameworks such as Pytest-ETL, ETL Validator, and ETL Robot offer a method that is both structured and portable. The data and the ETL process can also be visualised and analysed with the assistance of data visualisation tools like as Tableau, Power BI, and Qlik Sense. To supply this company with information that is helpful, this study makes use of the nine-step process that was created to implement an Online Analysis Processing (OLAP) database and a data warehouse. Small and medium-sized businesses will find it simpler to analyse data because of the data being performed into dashboards. Additionally, a great deal of helpful information can be shared in a short amount of time and in an effective manner.
References
English, L. P., "Ten mistakes to avoid if your data warehouse is to deliver quality information", DSSResources.COM, 08/11/2002.
Power, D. J. "What is ETL software and how is it related to DSS?", DSS News, 08/04/2002 at URL dssresources.com/faq/index.php?action=artikel&id=50.
Rouse M 2015 What is data warehouse? - Definition from WhatIs.com, [Online]. Available: https://searchdatamanagement.techtarget.com/definition/data-warehouse. [Accessed: 13-Nov2018].
Dewan S, Aggarwal Y, and Tanwar S Review on Data Warehouse, Data Mining and OLAP Technology: As Prerequisite aspect of business decision- making activity, IJRIT Int. J. Res. Inf. Technol. Int. J. Res. Inf. Technol. 1 10 pp 30–39
Doreswamy, Gad I, and Manjunatha B R 2017 Hybrid data warehouse model for climate big data analysis, Proc. IEEE Int. Conf. Circuit, Power Comput. Technol. ICCPCT 2017
Dehne F, Kong Q, Rau-Chaplin A, Zaboli H, and Zhou R 2015 Scalable real-time OLAP on cloud architectures, J. Parallel Distrib. Comput. 79–80, pp 31–41
Johnson J C 2015 OCP : Oracle9i TM Performance Tuning.
Girsang A S, Isa S M, Adytiansya N, Utomo O K, and Simarmata J 2018 The data warehouse for down payment administration in the Constitutional Court of Republic of Indonesia, IOP Conf. Ser. Mater. Sci. Eng. 420 pp. 012104
Ricky M Y 2014 Datawarehouse sales and supply of goods model based on HTML5, J. Theor. Appl. Inf. Technol. 61 1 pp. 175–182
Gill R and Singh J 2014 A Review of Contemporary Data Quality Issues in Data Warehouse ETL Environment, J. Today’s Ideas - Tomorrow’s Technol. 2 2 pp 153–160
Aziz O, Anees T, Mehmood E (2021) An efcient data access approach with queue and stack in optimized hybrid join. IEEE Access 9:41261–41274
Liu C, Wu T, Li Z, Ma T, Huang J (2022) Robust online tensor completion for IoT streaming data recovery. In: IEEE transactions on neural networks and learning systems
Zhou X, Zhang L (2022) SA-FPN: an efective feature pyramid network for crowded human detection. Appl Intell 52(11):12556–12568
Li S, Chen H, Chen Y, Xiong Y, Song Z (2023) Hybrid method with parallelfactor theory, a support vector machine, and particle flter optimization for intelligent machinery failure identifcation. Machines 11(8):837
Liang X, Huang Z, Yang S, Qiu L (2018) Device-free motion & trajectory detection via RFID. ACM Trans Embed Comput Syst 17(4):78
Cao B, Zhao J, Gu Y, Fan S, Yang P (2020) Security-aware industrial wireless sensor network deployment optimization. IEEE Trans Industr Inform 16(8):5309–5316
Peng Y, Zhao Y, Hu J (2023) On the role of community structure in evolution of opinion formation: a new bounded confdence opinion dynamics. Inf Sci 621:672–690
Zhou K, Jia Z, Jia F, Shao H (2023) Multi-scale integrated deep self-attention network for predicting remaining useful life of aero-engine. Eng Appl Artif Intell 120:105860
Mhon GGW, Kham NSM (2020) ETL pre-processing with multiple data sources for academic data analysis. In: IEEE Conference on Computer Applications (ICCA). pp 1-5
Mondal KC, Biswas N, Saha S (2020) Role of machine learning in ETL automation
Ghasemaghaei M, Calic G (2019) Can big data improve frm decision quality? The role of data quality and data diagnosticity. Decis Support Syst 120:38–49
Kim S-S, Lee W-R, Go J-H (2019) A study on utilization of spatial information in heterogeneous system based on Apache NiFi. pp. 1117–1119
Timmerman Y, Bronselaer A (2019) Measuring data quality in information systems research. Decis Support Syst 126(February):113138
Taleb I, Serhani MA, Dssouli R (2019) Big data quality assessment model for unstructured data. In: 13th International Conference on Innovations in Information Technology, IIT 2018. pp 69–74
Cichy C, Rass S (2019) An overview of data quality framework. IEEE Access 7:24634–24648
Günther LC, Colangelo E, Wiendahl HH, Bauer C (2019) Data quality assessment for improved decision-making: a methodology for small and medium-sized enterprises. Procedia Manuf 29:583–591
Tian Q, Liu M, Min L, An J, Lu X, Duan H (2019) An automated data verifcation approach for improving data quality in a clinical registry. Comput Methods Programs Biomed 181:104840
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Jeshwanth Reddy Machireddy (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
