New custom rating for improving recommendation system performance

Abstract

Recommendation system is currently attracting the interest of many explorers. Various new businesses have surfaced with the rise of online marketing (E-Commerce) in response to Covid-19 pandemic. This phenomenon allows recommendation items through a system called Collaborative Filtering (CF), aiming to improve shopping experience of users. Typically, the effectiveness of CF relies on the precise identification of similar profile users by similarity algorithms. Traditional similarity measures are based on the user-item rating matrix. Approximately, four custom ratings (CR) were used along with a new rating formula, termed New Custom Rating (NCR), derived from the popularity of users and items in addition to the original rating. Specifically, NCR optimized recommendation system performance by using the popularity of users and items to determine new ratings value, rather than solely relying on the original rating. Additionally, the formulas improved the representativeness of the new rating values and the accuracy of similarity algorithm calculations. Consequently, the increased accuracy of recommendation system was achieved. The implementation of NCR across four CR algorithms and recommendation system using five public datasets was examined. Consequently, the experimental results showed that NCR significantly increased recommendation system accuracy, as evidenced by reductions in RMSE, MSE, and MAE as well as increasing FCP and Hit Rate. Moreover, by combining the popularity of users and items into rating calculations, NCR improved the accuracy of various recommendation system algorithms reducing RMSE, MSE, and MAE up to 62.10%, 53.62%, 65.97%, respectively, while also increasing FCP and Hit Rate up to 11.89% and 31.42%, respectively.

Introduction

Several research are currently deploying recommendation system in many fields such as E-Commerce [1], social networks [2], news [3], online services, search engine [4], streaming services [5], healthcare service recommendation [6], and restaurant services [7]. This widespread adoption is due to the abundance of both explicit and implicit user data available across the fields. Examples of the data include social media data, wearable sensors, medical data, smart device usage, click data of user, and behavior patterns of user. Typically, all of the data can be leveraged to build an effective recommendation system. There are three general methods of building recommendation system models namely, Content-Based Filtering (CBF), Collaborative Filtering (CF), and Hybrid System (HS) [8].

CBF is a method for recommending items based on the similarity of description or features to other profiles of user [9]. The system provides recommendations by analyzing the history of users’ past preferences in this method. To learn these preferences, recommendation system uses some machine learning algorithms such as Decision Tree [10], Naive Bayes classifier [11], SVM [12], and Deep Learning [13]. Additionally, CBF offers two advantages which include the model does not require any data from users and the model can recommend items with fewer fans. However, the model has the following disadvantages such as the dependency of its accuracy based on sufficiency of information which requires intensive domain knowledge. The model also has limited ability to expand on the users’ existing interests [14].

CF is recommendation system method that uses similarity algorithms to find a user with preferences similar to the neighborhood [9]. Furthermore, it uses interactions between users and items, such as ratings to make recommendations [15]. CF is divided into two methods namely model-based and memory-based [16]. Specifically, memory-based can be classified into item-based and user-based CF methods. The difference between model and memory is that a memory-based algorithm loads the entire database into system memory to make recommendation while a model-based algorithm compresses large database into a model and performs recommendation task using reference mechanism [17]. In model-based, latent factors are used to predict preference score of undetected items. [18]. The method includes two steps, namely computing matrix factorization, followed by generating recommendation based on machine learning algorithms. The advantage of CF is that it can recommend hidden items to users. However, CF method also has disadvantages such as data scarcity, cold-start problem, and scalability issues.

HS is recommendation system that combine two or more methods to obtain better performance by eliminating the drawbacks of each method [14]. Some of the combination strategies include combining CF with Content-Based method and Self-Organizing Map neural network method [19], combining CF system using SVD algorithm with a content-based system, and a fuzzy expert system [20]. Moreover, other strategies include joining CF to make recommendations based on information equations between users and CBF [21]. In general, there are seven hybridization mechanisms used to build hybrid recommendation system including weighted, mixed, switching, feature combination, feature augmentation, cascade, and meta-level [22].

Among the methods mentioned, user and item ratings are two major data sources used by recommendation system [23]. The degree of preference in recommendation system is measured by a rating score [24]. Item recommendation is usually sorted by predicted rating in descending order. Typically, items rating are represented as User, Item, and Rating [25] while CF method often uses average rating. This method assumes that rating value for all items is the same, without considering the number of users. Custom rating (CR) can be used to determine final rating when considering the popularity of users and items to determine new ratings value. Based on available data, the current research is the first to explore CR with the aim of improving performance of recommendation system in some public dataset.

Related work

Improving rating formulas to increase recommendation system accuracy is an interesting idea. According to [26], rating is a number that expresses a user’s opinion about a given product. It is usually between 1 and 5, with higher values showing better ratings. To generate recommendations, similarity algorithms are used on the row or column vectors of ‘user x product’ matrix of ratings. Moreover, this recommendation is built by finding the closest user’s profile based on the similarity with other users.

Diverse and popular E-Commerce applications, such as books, movies, products, and videos, frequently include a rating feature. This rating is a major source of data used by recommendation system [23]. Some results have explored customized rating schemes in the system including Weighted Average Ratings, which are used in IMDb rating [27], Bayesian Average rating [28], Median Average Rating [29], Mean Rating [30], and Normalized Rating Frequency [31]. Additionally, research has derived rating values from contextual data such as aspect category [32], word embedding representation of reviews [23], and Linked Open Data [33]. It is crucial to be aware that recommendation system has the main task of achieving rating prediction [34]. Therefore, this research focuses on exploring rating prediction method across dataset of movie, music, and jokes signifying the accuracy of this prediction.

Rating in recommendation system

Rating is one of the important features of recommendation system. The main function of rating is to determine a user’s preference or interest in a product. Typically, users can like or dislike preferred item. This rating helps recommendation system determine suitable items for users. In the context of this research, rating can be considered explicit or implicit [35]. Specifically, explicit rating is given directly by users through a scale or rating system such as 1–5 or 1–10 [36]. Meanwhile, implicit rating is derived from user interactions with a product such as viewing or clicking on the item [37].

Apart from being used to determine recommendations, ratings can also be used in recommendation system for other purposes, including:

• Evaluation of recommendation system: Ratings can be used to evaluate the accuracy of recommendation system [9] by comparing user-given to those predicted by the system.

• Improved accuracy: A new rating is used to improve the accuracy of recommendation system [38].

• Personalization of recommendations: Ratings can be used to create more personalized recommendations [39].

• Pricing: Ratings are used to determine the price of an item [40].

• Compared to implicit rating, explicit rating has many advantages as follows:

• Accuracy: Ratings provided by users are explicitly more accurate than those derived from user interactions with items. In explicit rating, user directly shows the level of preference for an item, which is particularly reliable when reviewers possess a high degree of expertise [41].

• Specificity: Explicit ratings provides more specific information about user preferences than implicit ratings. Users can give different ratings to various items, thereby providing more specific information about preferences.

• More data: Implicit ratings are often derived from limited data such as clicks or other interactions. Meanwhile, explicit ratings typically come from more data, such as comments or reviews [42].

• Transparency: Since user explicitly provides rating for item, this rating is more transparent and easier to understand than an implicit rating obtained from interactions that may not be clearly understood.

The choice of method depends on the application’s conditions and the availability of the necessary data. Explicit ratings can provide more comprehensive and valid data but require higher participation from users. Meanwhile, implicit rating is easier to implement because it does not require active user contribution. However, the data obtained from implicit rating is less specific.

Custom rating

CR is made to accommodate specific needs, criteria, and contexts. The contexts can vary for each type of recommendation system environment and can be changed according to user preferences or business requirements. For example, in movie recommendation system, CR such as Weighted Average Rating (WR) calculates rating by considering the number of votes, average rating of item, minimum votes required to be listed, and the mean vote. Another CR method uses median rating, which takes the middle value of a set of rating data. Furthermore, the mean rating or average rating shows the total average level of an item or product. Lastly, Normalized Rating Frequency (NRF) uses rating frequency and applied normalization process.

Weighted average rating

The purpose of rating is to address the skewness problem that often occurs in system. This method helps to manage inconsistencies in film ratings which can change as the number of voters increases. For example, a newly released film with only a small number of voters may have a very high average rating because it is only judged by very enthusiastic viewers. However, as the film reaches broader audience over time, the average rating may drop and become more accurate.

$$Weighted \,Rating \left(WR\right)= \left(\frac{v}{v+m}R\right)+\left(\frac{m}{v+m}C\right)$$

(1)

Where v is the number of votes for the movie, m represents minimum votes required to be listed in the chart, R is the average rating of the movie, and C is the mean vote across the whole report.

Median rating

The median rating is the middle value of a set of rating data. typically, the median is determined by sorting all the rating data and selecting the middle value. When the number of rating data is odd, the median is the middle, when even, it is the average of the two middle values. Furthermore, the median rating is often used in recommendation system to address data skewness problems. For example, a film with a small number of high ratings may have a very high average rating which may not represent the views of most viewers. The median rating can provide a more accurate picture, reflecting a central tendency that is less affected by extreme values.

$$Median\left(X\right)= \left\{\begin{array}{c}X\left[\frac{n+1}{2}\right] \quad\quad\,if \,n \,is \,odd\\ \frac{X\left[\frac{n}{2}\right]+X\left[\frac{n}{2}+1\right] }{2}\,\,\,if \,n \,is \,even\end{array}\right.$$

(2)

Mean rating

The mean rating or average rating shows the average level of an item or product rated by several people. This value provides an understanding of the total rating of an item or product. However, the mean rating has weaknesses, such as being affected by very high or very low rating values from one or several individuals. Therefore, other methods such as the median rating mode are used to determine the average level of an item or product more accurately.

$$\stackrel{-}{x}\,=\frac{\sum {x}_{i}}{n}$$

(3)

Normalized Rating Frequency (NRF)

NRF is a user-rating method focused on normalizing rating frequency [31]. The normalization process is based on calculating the frequency of user rating, eventually leading to product ranking for recommendation. In addition, NRF method consists of 5 stages as follows:

• Calculating Rating Frequency: The frequency is calculated by summing each user rating occurrence. u presents list of users, p, list of products, and r presents list of user-rating.

$${MF}_{\left({r}_{i},{p}_{h}\right) }= \sum _{u=1}^{l}\left\{\begin{array}{l} 1\to ifM\left({u}_{g},{p}_{h}\right)= {r}_{i}\\ 0\to if \,Otherwise\end{array}\right.$$

(4)

• Calculating Total Frequency Rating: The total frequency rating is obtained by summing $M\left({u}_{a},{p}_{h}\right)$ for each product. This frequency is presented by ${Total}_{\left(ri\right)}.$

$${Total}_{\left({r}_{i}\right)}= \sum _{p}MF\left({r}_{i},{p}_{h}\right)$$

(5)

• Normalization: The normalization guarantees that $MF\left({r}_{i},{p}_{h}\right)$ is scaled from 0 to 1. Additionally, this normalization can overcome the differences in scale of units in rating frequency. The normalization process is presented by $N\left({r}_{i},{p}_{h}\right).$

$$N\left({r}_{i,}{p}_{h}\right)= \left\{\begin{array}{c}\quad\quad0\to if \,{Total}_{\left({r}_{i}\right)}\\ \frac{MF\left({r}_{i},{p}_{h}\right) }{{Total}_{\left({r}_{i}\right)}}\to if \,Otherwise\end{array}\right.$$

(6)

• Weighted Rating: Weighted rating $\left(NR({r}_{i},{p}_{h})\right)$ helps to differentiate rank values among products by avoiding identical rating values.

$$NR\left({r}_{i},{p}_{h}\right)=N\left({r}_{i},{p}_{h}\right)*{r}_{i}$$

(7)

• Total Weighted Rating: Total Weighted Rating represents the sum of all weighted rating each product. In addition, the rating is formulated as follows:

$$NRF\left({p}_{h}\right)= {\sum }_{r}NR\left({r}_{i},{p}_{h}\right)$$

(8)

Proposed method

The section discussed a new method to improve recommendation system by using four CR combined with new rating formula that joined frequency count and the original rating. Generally, the method aimed to ensure the effectiveness of items rating by considering users’ popularity through the frequency of given ratings, original users rating, and items popularity through CR. The proposed method, called NCR was shown in following formula:

$${Ncr}_{\left({r}_{i},{p}_{h}\right)}= \sqrt[3]{{F}_{{U}_{i}}*{CR}_{{P}_{h}}*{r}_{i}}$$

(9)

Where ${F}_{{U}_{i}}$ presents normalized and scaled frequency count of i-user from all products rated by i-user. ${CR}_{{P}_{h}}$ represents CR for h-product and ${r}_{i}$ represents user-rating for h-product respectively. Furthermore, CR was achieved using weighted average rating, median, mean, and normalized rating frequency in this experiment. ${F}_{{U}_{i}}$, ${CR}_{{P}_{h}}$, and ${r}_{i}$ were scaled to the same range, such as 1–5. Additionally, Fig. 1 showed the experimental design of NCR method.

Fig. 1

Experimental design of NCR

Four CR such as Weighted Average Rating, Median, Mean, and Normalized Rating Frequency were used. For the experimental research, five public datasets were used which included MovieLens (100 K), MovieLens (1 M), MovieLens Latest Small (2 K), Jester, and Last.fm. Furthermore, Table 1 showed the number of users, rating, product, and density for each dataset. All dataset rating was normalized in 1 to 5 scaled. In system recommendations, “data density” referred to the degree to which user-item data in a dataset was populated. Data density was the ratio between the number of entries or ratings in the dataset and the theoretically possible number of entries. Additionally, higher-density data meant there were more entries or ratings in the dataset, providing more complete and relevant information for making recommendations. The following process led to more accurate and reliable recommendations. Moreover, low data density, showed many entries or ratings were missing, making it difficult to find patterns or sufficient information to provide relevant recommendations. Moreover, low data density also led to problems in training and testing recommendation models, due to a shortage of available data.

Table 1 Dataset description

Experimental tests were conducted on an Intel (R) Core i7-9750 H machine with 2.6 GHz clock frequency and 8 GB of RAM, running on Windows 11 platform with Python programming language. Additionally, a 10-fold cross-validation procedure with the same Random_State (RS) was used. Typically, RS parameter referred to an integer or a random generator object. When the value was provided, the data division remained consistent every time cross-validation was performed with the same RS parameter. This process was useful for ensuring consistent and reproducible results when using cross validation. Subsequently, an average of all the results was compared across all algorithm and scenario. To evaluate the performance of RS, different similarity measure was used each time. Therefore, the performance of recommendation system using four well-known evaluation metrics was evaluated as follows.

• Root Mean Squared Error (RMSE): RMSE measured the differences between predicted values and the observed values, assessing the quality of fit between the actual data and the model’s prediction. Additionally, RMSE was one of the most frequently used metrics for evaluating the goodness of fit of generalized regression models. RMSE had several advantages including error representation on the same scale, sensitivity to large errors, and more stability to scale changes.

$$RMSE=\sqrt{\frac{1}{n}{\sum }_{i=1}^{n}{\left({y}_{i}-{\widehat{y}}_{i}\right)}^{2}}$$

(10)

• Mean Squared Error (MSE): An evaluation metric commonly used in recommendation system that measured the difference between the predicted value and the actual value in terms of rating or user preference for an item. MSE was calculated by taking the squared difference between the predicted and actual value for each rating or user preference for the item, followed by averaging the squared difference. Furthermore, lower MSE value showed better quality predictions from recommendation system. Using MSE for recommendation system metrics had several advantages such as better error representation and sensitivity to value differences.

$$MSE=\frac{1}{n}{\sum }_{i}^{n}{\left({y}_{i}-{\widehat{y}}_{i}\right)}^{2}$$

(11)

• Fraction of Concordant Pairs (FCP): The objective of FCP was to measure the degree of consistency or agreement between two ordered data sets. Typically, these metrics were used to compare two measurement methods or two explorers assessing the same item. FCP used pairs of ratings from the test set of the data. Each pair consisted of two rated items from the same user, with every possible combination of items rated by that user forming a pair. Following this process, a higher FCP value showed greater accuracy of the algorithm. For example, when FCP value was 0.5, it meant that 50% of the item pairs being compared had a concordant ranking order. However, when FCP value was 0.8, then 80% of the item pairs being compared had a concordant ranking order. In both cases, higher FCP score showed better quality recommendation system.

$$FCP= \frac{{n}_{c}}{{n}_{c}+{n}_{d}}$$

(12)

• Hit Rate: Hit Rate was the fraction of users for whom the correct answer was included in recommendation list of length L. $\left|{U}_{hit}^{L}\right|$ represented the number of users for whom the correct answer was included in the top L recommendation list, while $\left|{U}_{all}\right|$ was the total number of users in the test dataset. Furthermore, this method evaluated the quality of recommendation based on the success in predicting the items a user was genuinely interested in, considering the relevance of recommendations provided by the system. Hit Rate was typically expressed as a value between 0 and 1, with higher value showing better performance in recommendation system. For example, when Hit Rate of 0.6 with L = 5, it meant that the system successfully recommended at least one correct item in the top 5 items by 60% of the users evaluated.

$$Hit Rate= \frac{\left|{U}_{hit}^{L}\right|}{\left|{U}_{all}\right|}$$

(13)

Experimental result

Results were presented by applying a formula, which used weighted average rating, median rating, mean rating, and NRF scheme in this section. Additionally, the result was compared with the original rating data. The experimental exploration was conducted using the same datasets, 10-fold cross-validation, and the same random state. The top 10 recommendation settings were used for each user from a set of predictions.

RMSE as a measure

The result was analyzed based on RMSE as a measure in this sub section. Table 2 showed the results for RMSE metrics obtained for the four algorithms across five dataset, showing the best results in bold for each dataset. The Table showed that using a new formula rating derived from the four-rating method decreased RMSE value by approximately 62.10% compared to the original rating. Subsequently, introducing a new formula using NRF as CR method achieved the smallest RMSE in three datasets compared to the other CR methods. By conducting the Friedman Test, a non-parametric statistical test aimed at determining whether there was a significant difference between the groups. For example, whether there was at least one group that consistently ranks higher or lower than the other, a test statistic of 42.589 and corresponding p-value of p = 0.00 were obtained. This showed statistically significant differences in RMSE from all rating methods. Compared to the original rating, all CR formulas showed performance improvement by decreasing RMSE results in four datasets across all algorithms. Additionally, weighted average rating showed the best performance compared to the other CR algorithms.

Table 2 RMSE results

MSE as a measure

MSE was used as a measure to assess CR performance in this sub section. Table 3 showed that using new formula rating derived from the four-rating method decreased the average MSE value by approximately 53.62% compared to the original rating. Subsequently, NRF outperformed the others in achieving the smallest MSE value, as evidence by its performance across three datasets. The test produced a statistic of 37.96 and corresponding p-value of p = 0.00, showing statistically significant differences in MSE among all rating method.

Table 3 MSE results

MAE as a measure

Similar to RMSE and MSE, MAE served as a measure where CR significantly outperformed the original rating. Table 4 showed the results for MAE metrics obtained across four algorithms and five datasets, with best values showed in bold. According to Table III the use of a new formula derived from the four-rating method could improve performance by decreasing the MAE value by approximately 65.97% compared to the original rating. Additionally, a new formula using NRF as CR method achieved the smallest MAE in four datasets compared to the other CR methods. By conducting the Friedman Test, test statistic of 36.391 was obtained and the corresponding p-value of p = 0.00, showing statistically significant differences in MAE from all rating method.

Table 4 MAE results

FCP as a measure

Similar to RMSE and MSE, in terms of MAE as a measure, CR outperformed the original rating significantly. Table 5 showed the results for FCP metrics obtained across four algorithms and five dataset, with best values showed in bold. Therefore, it was concluded that the new formula derived from the four-rating method improved performance by about 11.89% compared to the original rating. Moreover, a new formula using NRF as CR method achieved the smallest RMSE in four datasets compared to the other CR methods. By conducting the Friedman Test, test statistic of 45.768 and a corresponding p-value of p = 0.00 was obtained, showing statistically significant differences in FCP among all rating method.

Table 5 FCP results

Hit rate as a measure

Similar to other evaluation metrics mentioned earlier, Hit Rate for CR was 31.42% higher than standard rating as shown in Table 6. NRF algorithms appeared as the best-performance algorithms across four dataset. According to Friedman Test, the explorers obtained a test statistic of 26.505 and corresponding p-value of p = 0.00, showing statistically significant differences among all rating methods in terms of Hit Rate achievements.

Table 6 Hit rate results

Rating and custom rating in boxplot

Box plots provided an overview of data distribution, range, mean, diversity, and the presence of outlier values. The explorers enabled comparison between the new distribution of CR data and the original rating data. The length of the box showed the data’s spread, with a longer box showing greater dispersion. In a vertical boxplot, the lines below the boxes represented the lower whiskers, while those above represented the upper whiskers. Additionally, values ​​beyond the whisker were termed outliers and extremes.

Figure 2 showed the distribution of both the original rating and CR data in Movie Lens 100 K. The explorers observed that the rating data distribution changed after modifications. All rating modification algorithms showed smaller median data compared to the original rating, according to the boxes positioned beneath those of the original data. Furthermore, the quartile range became shorter while the outlier data was greater. This showed a significant alteration in the rating value due to the combination of the popularity of users and items with the original rating. Some outlier values were present in all CR algorithms, evident in values above Q3 + (1.5 x IQR) and below Q1 - (1.5 x IQR) showing the influence of ratings frequency (F), original users rating (r), and items popularity through CR in NCR formula on the new rating distribution. Although some of new rating was located in outlier area due to the effect of item popularity, this phenomenon increased the accuracy of recommendation system.

Fig. 2

Box plot movie lens 100k

Figure 3 showed the distribution of both the original rating and CR data in Movie Lens 1 M. Similar to Fig. 2, explores observed a change in the rating data distribution after modifications. The median rating value as shown by the boxplot was lower compared to the original rating. Additionally, interquartile range had shrunk leading to a narrower data distribution. Among custom algorithms, NFR rating algorithms showed the smallest interquartile range.

Fig. 3

Box plot movie lens 1 M

Figure 4 showed different distribution with the median data in weighted average rating exceeding the original. The length of the box and whisker of this rating was smaller than the original. In comparison to both the original and weighted average rating, the mean, median, and NFR rating showed lower median and numerous outlier values.

Fig. 4

Box plot movie lens (latest small)

Compared to movie lens dataset in jester dataset shown in Fig. 5, the new rating showed symmetrical data distribution. Interquartile length of weighted, mean, and median rating was smaller than the original, showing reduced variability in the distribution of new rating data. Subsequently, an outlier value in NFR rating significantly contributed to improving performance.

Fig. 5

Box plot jester

Figure 6 finally showed that the length of NFR rating interquartile was significantly greater than that of the others. Median value of all new CR algorithms was almost the same. Additionally, the spread of new rating data from weighted, mean, and median rating was also almost the same distribution.

Fig. 6

Box plot last fm

Result and discussion

The results were represented where the original rating were compared with those obtained using CR. The experimental research was conducted using the same datasets, RS parameters, and algorithms. Furthermore, the results from five evaluation parameters showed that CR could increase recommendation system’s accuracy by decreasing RMSE, MSE, and MAE while increasing FCP and Hit Rate. Various CR methods were applied and the accuracy was assessed across five public datasets. The experiment results led to the conclusion that combining the popularity of users and items with the original rating improved recommendation. This combination altered the distribution of the new rating data, as observed from the boxplot results. Additionally, the result was evident that the inclusion of users and items popularity improved recommendation performance, as shown by the new RMSE, MSE, MAE, FCP, and Hit Rate value. The new rating decreased average of values of RMSE, MSE, and MAE by 62.10%, 53.62%, and 65.97%, respectively, and also increased FCP and Hit Rate by 11.89% and 31.42%.

NCR generally improved accuracy across metrics such as RMSE, MSE, MAE, FCP, and Hit Rate, but Last.fm dataset presented an exception where good MSE and MAE values ​​were achieved without NCR intervention. This anomaly might have risen from dataset’s low-density value and the abundance of outliers in the rating distribution, as shown from the boxplot in Fig. 6. Consequently, further research would be needed to validate the observations.

This research had certain limitations, as the first method sacrificed computing performance when handling new rating in a large dataset. The second method required rating to be normalized to a scale of 1–5 before implementing NCR.