Development of a Knowledge Discovery System in Big Data Mining Environment

This paper focused on the development of knowledge discovery system in big data mining environment. In order to carry out the aim of the work, the paper developed a knowledge discovery system in Big Data Mining Environment that could sift through large amounts of data to find previously hidden patterns, discover valuable new insights and make decisions; apply the dynamics involved in big data technologies and use of distributed data storage and analysis architecture of Hadoop MapReduce; conduct performance benchmarking on Relational Database Management System (RDBMS) and Hadoop cluster, create value in several ways and improve performances. The analytic environment provided a powerful in database algorithms and open source algorithms to enable predictive analytics, data mining, statistical analysis, advanced numerical computations and interactive graphics. Automated analysis of historical data were performed by employing Knowledge Discovery and Data mining (KDD) using Map Reduce Methodology and Predictive Analytic Methodology. The Euclidean distance and the pseudo F‑statistic validated Hadoop’s high scalability and performance in the real time applications domain, minimized data movement thereby ensuring inherent security and better performance. The result showed that a model for big data mining environment was realized which provided an open source framework for cloud computing and distributed file system for fast data loading.