Publisher：Applied Mathematics and Nonlinear Sciences  

With the advancement of contemporary technology, there is an increasing demand for sophisticated tone libraries in intelligent and digital pianos. This paper introduces a virtual piano system based on timbre simulation. Drawing upon the articulation principles of the piano, a mathematical model is formulated to derive the equation governing string vibrations. This equation facilitates a comprehensive analysis of the components constituting piano timbre and establishes a model for synthesizing these timbres using a frequency envelope algorithm. The short-time Fourier transform (STFT) is utilized to extract features from the synthesized timbre and to accomplish its digital simulation. Subsequently, the integration of the piano's timbre features with computer programming facilitates the comprehensive design of the virtual piano system. Performance testing and evaluation of the system reveal promising results: the detection rate for each musical piece exceeds 90%, with an average detection rate of 94.81% across ten pieces and an average deviation (Mean-D) of 3.35 in the scoring of 100 music samples. This research contributes to the flexibility in timbre editing, enhances the expressiveness of intelligent and digital pianos, and aims to elevate the music industry to new heights.

DOI： 10.2478/amns-2024-1424

Scopus

Publisher：Applied Mathematics and Nonlinear Sciences  

In this paper, in order to solve the modal aliasing problem of the EMD decomposition algorithm and EEMD decomposition algorithm and optimize the vocal audio data decomposition on the basis of the adaptive VMD algorithm, the EEMD-VMD second-order data decomposition algorithm is proposed to decompose the digitized time series of vocal music resources, and to update the construction of the digital library of vocal music resources. Combined with the decomposition steps of the second-order data decomposition algorithm in the vocal resources digitized library, the overall framework of the vocal resources digitized library is designed and constructed with the main line of vocal resource types and the secondary line of content categories. Design the experimental environment, preprocess the vocal audio signal features, explore the modal number K value size of the EEMD-VMD second-order data decomposition algorithm, analyze the vocal audio denoising performance of the algorithm, and compare the correlation coefficients of the IMF components of the decomposition algorithms with the original audio signals. Explore the second-order data decomposition algorithm's emotional classification of vocal resources and the performance of vocal signal enhancement, add noise sources, and use subjective and objective evaluation methods to assess the quality of vocal PESQ and STOI after decomposition processing. The EEMD-VMD second-order data decomposition algorithm performs the most prominently in street noise environments. The enhancement results in the case of signal-to-noise ratios of -5, -10, and -15 are 3.59, respectively, 4.36, 4.29, and the mean value of the enhancement performance reaches 4.08. The second-order data decomposition algorithm's precise processing of vocal resources provides better quality digital resources for the construction of a digital library of vocal resources.

DOI： 10.2478/amns-2024-1425

Scopus

Publisher：Proceedings - 2024 5th International Conference on Computer Science, Engineering, and Education, CSEE 2024  

During the Covid-19 pandemic, e-learning and online learning systems and platforms have become increasingly popular and essential. Currently, numerous e-learning materials covering various subjects have been developed and are widely utilized in practical education. Music instruction education has also confronted this challenge. In this paper, we introduce a piano learning system. We incorporate multi-directional video technology to enable users to observe piano players from their desired angles. Additionally, we integrate MediaPipe into this system to implement hand landmark detection, facilitating users in virtual piano playing. We also use an experiment to evaluate the educational efforts of our system.

DOI： 10.1109/CSEE63195.2024.00014

Scopus

Publisher：31st International Conference on Computers in Education, ICCE 2023 - Proceedings  

This paper treats the user study for web-based VR education contents supporting VR-goggles. Educational materials using multimedia have ability to enhance the education efficiency rather than text books. Similarly, educational materials using new technologies like VR have ability to attract much interests of students. Based on Interest-Driven Creator Theory (IDC Theory), this means educational materials using VR may have high educational efficiency. For experiment subjects and exercise subjects, the most efficient learning environments should be the same as the real ones as possible. Educational materials using VR can provide virtually students with high immersive learning environments. Therefore, educational VR materials supporting VR-goggles may have higher education efficiency rather than the others. This may justify Seamless Interest-Driven Creator Theory (SIDC Theory). So, this paper introduces several web-based VR education contents, and also show simple user study results especially for the comparison between their web-browser mode and VR goggle mode.

Scopus

Language：English   Publishing type：Research paper (international conference proceedings)  

DOI： 10.1007/978-3-031-26281-4_42

In this paper, the authors introduce web viewers for educational VR contents. Especially, they propose new web viewer for educational VR contents of 3D scene models. Although there have been many types of VR goggles and a lot of entertainment VR contents, there have been a few educational VR contents. To enhance educational efficiency using VR technology, such educational VR contents should be created more and more. On the other hand, many types of scanning devices have been researched and developed, e.g., Lidar scanners, 360VR cameras and so on. Using these scanning devices, 3D data like Point Cloud Data (PCD), 3D model data and 360VR images/videos, can be obtained easily. To use these 3D data for educational VR contents, the authors have already developed viewer applications as web services. These web viewers have functionality to upload 3D data from users' web-browser to the web-server on that the web viewers work. However, 3D scene models are not supported. Standard videos are still important media for the on-demand online educations. Therefore, the authors propose new web viewer for 3D scene models supporting standard videos and VR goggles in this paper.

DOI： 10.1007/978-3-031-46784-4_7

Scopus

This paper proposes an extended development framework for web-based VR tours using 360VR videos based on OpenVSLAM. The authors have already proposed the development framework for web-based VR tours as navigation contents of 360VR videos and as walkthrough contents of 360VR images. However, it is not easy to prepare 360VR images of different locations for the walkthrough contents if the number of those images is too many. So, the authors have also proposed new development framework for web-based VR tours as both navigation contents and walkthrough contents using 360VR videos, not using 360VR images. For the walkthrough contents, we use 360VR videos because the required 360VR images can be extracted as keyframes of the same scenes from the 360VR videos. Furthermore, when there are many scenes, i.e. many 360VR images in one VR tour, it will become very convenient that the user can go to his/her desired scenes directly by the keyword search. Also, when 360VR videos created as educational contents include audio narrations, it will become also very convenient if the multi-language subtitles are available from the audio narrations. So, the authors extended their development framework for web-based 360VR tours to support the keyword search and multi-language subtitles using Google Cloud Vision AI and free AssemblyAI services.

DOI： 10.1007/978-3-031-53555-0_4

Scopus

This paper proposes a development framework using 360VR cameras and lidar scanners for web-based XR educational materials that supports VR goggles. Recently, the demand of XR (VR/AR/MR) applications has become increased due to the advancement of XR technologies, e.g., high specification VR goggles, 360VR cameras and lidar scanners have become available at a cheaper price rather than ever. Although more than ten years ago, many entertainment XR applications have been produced, XR applications in other fields have not been so. Besides the entertainment field, educational materials using XR have become in great demand. Especially, XR seems useful for educational materials of experiment and exercise subjects. However, the development of such educational materials is not easy and needs much time. Therefore, in this paper, the authors propose the development framework for them. In this paper, the authors also show several material examples to justify the usefulness of the proposed framework.

DOI： 10.1007/978-3-031-26281-4_42

Scopus

Event date： 2024.3

Language：English   Presentation type：Oral presentation (general)  

Country：China  

Event date： 2023.7

Language：Japanese   Presentation type：Oral presentation (general)  

Country：Japan  

Type：Peer review 

Number of peer-reviewed articles in foreign language journals：6

Number of peer-reviewed articles in Japanese journals：0

Proceedings of International Conference Number of peer-reviewed papers：3

Proceedings of domestic conference Number of peer-reviewed papers：0