An Evaluation of the Application ("App") Verification Service in Android 4.2

By Xuxian Jiang, Associate Professor, Department of Computer Science, NC State University

With the latest Android 4.2 (JellyBean) release on November 13, 2012, Google announced an exciting security feature called the "application verification service" to protect against harmful Android applications. As stated in a recent Google+ post by a member of the Google Android team, "Now, with Jelly Bean Android 4.2 devices that have Google Play installed have the option of using Google as an application verifier. We will check for potentially harmful applications no matter where you are installing them from." It is indeed an exciting security feature! We think it was a really good move by Google to directly face Android malware threats and take such measures to better protect Android users.

Meanwhile, because of the introduction of this service, people may start to wonder, "are third-party security apps still necessary with Android 4.2?" -- see the links here, here, here, and there. We are no exception! To demystify this service, we perform the following study for two main purposes: (1) We want to understand better how the app verification service works; (2) We also want to quantify the effectiveness of this service and compare it with existing third-party anti-virus engines.

1. Google's App Verification Service (in Android 4.2) -- An Overview

Figure 1: Google's new app verification service (in Android 4.2)

The new service is implemented inside the official Google Play app, but is designed to work with apps from all app stores, including the official Google Play marketplace and other alternative ones. A user can turn the service on/off by going to "Settings," "Security," and then "Verify apps." When an app is being installed (Step 1), the service, if turned on, will be invoked (Step 2) to collect and send information about the app (e.g., the app name, size, SHA1 value, version, and the URL associated with it) as well as information about the device (e.g., the device ID and IP address) back to the Google cloud (Step 3). After that, the Google cloud will respond with a detection result (Step 4). If the app is not safe, the user is then shown a warning popup (Step 5) flagging the app as either dangerous or potentially dangerous. Dangerous apps are blocked from being installed, while potentially dangerous ones instead alert users and provide an option to either continue or abort the installation (Step 6) with a warning popup. In Figure 1, we show the key steps behind the service.

Figure 2(a): Enabling the app verification service

Figure 2(b): Detecting a potentially dangerous app

Figure 2(c): Detecting a dangerous app

We note that this app verification service is optional, but turned on by default. The first time an app is side-loaded, a popup window will ask whether to "Allow Google to check all apps installed to this device for harmful behavior?" (Figure 2(a)). If yes and the app is detected as potentially dangerous, the user will be prompted with a warning (Figure 2(b)) and given the option to continue or abort the installation. If detected as dangerous, the app will be blocked from installation (Figure 2(c)).

2. Assessment Setup and Evaluation Results

To quantify the effectiveness of this service, we leverage our prior Android Malware Genome Project and check whether the collected Android malware samples can be detected. We conducted our experiments on November 30, 2012, by using the same dataset of 1260 samples (belonging to 49 different families) that has been widely shared within the research community, including Google. Our study involves semi-automated installation of them on a few of latest Nexus 10 tablets (16GB) running Android 4.2 (build number:JOP40C). The detailed detection results are shown in Table 1. Overall, among these 1260 samples, 193 of them can be detected, indicating a low detection rate of 15.32%.

Android Malware Number of
Number of
Android Malware Number of
Number of
ADRD 22 8 36.36% AnserverBot 187 2 1.07%
Asroot 8 5 62.50% BaseBridge 122 7 5.74%
BeanBot 8 0 0.00% Bgserv 9 0 0.00%
CoinPirate 1 0 0.00% CruseWin 2 0 0.00%
DogWars 1 1 100.00% DroidCoupon 1 0 0.00%
DroidDeluxe 1 0 0.00% DroidDream 16 6 37.50%
DroidDreamLight 46 18 39.13% DroidKungFu1 34 8 23.53%
DroidKungFu2 30 9 30.00% DroidKungFu3 309 21 6.80%
DroidKungFu4 96 18 18.75% DroidKungFuSapp 3 0 0.00%
DroidKungFuUpdate 1 0 0.00% Endofday 1 0 0.00%
FakeNetflix 1 0 0.00% FakePlayer 6 5 83.33%
GamblerSMS 1 0 0.00% Geinimi 69 11 15.94%
GGTracker 1 1 100.00% GingerMaster 4 2 50.00%
GoldDream 47 6 12.77% Gone60 9 0 0.00%
GPSSMSSpy 6 1 16.67% HippoSMS 4 1 25.00%
Jifake 1 0 0.00% jSMSHider 16 4 25.00%
KMin 52 39 75.00% LoveTrap 1 0 0.00%
NickyBot 1 0 0.00% NickySpy 2 0 0.00%
Pjapps 58 8 13.79% Plankton 11 2 18.18%
RogueLemon 2 0 0.00% RogueSPPush 9 0 0.00%
SMSReplicator 1 0 0.00% SndApps 10 0 0.00%
Spitmo 1 0 0.00% Tapsnake 2 1 50.00%
Walkinwat 1 1 100.00% YZHC 22 3 13.64%
zHash 11 1 9.09% Zitmo 1 0 0.00%
Zsone 12 4 33.33% - - - -
Total 1260 193 15.32% - - - -
Table 1: Detection results of 1260 malware samples from the Android Malware Genome Project

In addition, we perform another set of experiments to compare the Google's app verification service with existing third-party anti-virus engines. Specifically, we randomly picked up a sample from each malware family and test it with the VirusTotal service (acquired by Google in September 2012). In Table 2, we show the comparison with ten representative anti-virus engines from VirusTotal (i.e., Avast, AVG, TrendMicro, Symantec, BitDefender, ClamAV, F-Secure, Fortinet, Kaspersky, and Kingsoft). Overall, the detection rates of these representative anti-virus engines range from 51.02% to 100% while the detection rate of this new service is 20.41%. Figure 3 shows the average detection rates of this new app verification service and ten other representative anti-virus engines.

Android Malware Google AV1 AV2 AV3 AV4 AV5 AV6 AV7 AV8 AV9 AV10 SHA1
ADRD N Y Y Y Y Y N Y Y Y N 09b143b430e836c513279c0209b7229a4d29a18c
AnserverBot N Y Y Y Y Y Y Y Y Y Y aa8cd901bd0bf62fefda2bffb6ba115b97fb8496
Asroot Y Y N Y N Y N Y Y Y Y 0c059ad62b9dbccf8943fe4697f2a6b0cb917548
BaseBridge N Y Y Y Y Y Y Y Y Y Y b8c205be3ee561126ff2c5cb12b81f94dae06c25
BeanBot N Y Y Y N N N Y N Y N a1fa9de17c36f00fbbdfffc9bfc3c858b9202f73
Bgserv N Y N Y Y Y Y Y Y Y Y bc2dedad0507a916604f86167a9fa306939e2080
CoinPirate N Y Y Y Y Y N Y Y Y Y a7f94d45c7e1de8033db7f064189f89e82ac12c1
CruseWin N Y Y Y Y Y Y Y Y Y Y 438e0b566eca22e7168711931a958736d9a50118
DogWars Y Y Y Y Y Y Y Y Y Y Y 0274a66cd43a39151c39b6c940cf99b459344e3a
DroidCoupon N Y Y Y N Y N Y Y N N 1c0a6b1c5d24cbba9b11020231fffc0840dd7e10
DroidDeluxe N Y Y Y Y Y Y Y Y Y Y 8e2f43e46335b8a4ce68c920660def6e9c14c712
DroidDream Y Y Y Y Y Y Y Y Y Y Y b349851d2a8ba476a0099c17714559f713aa2fdc
DroidDreamLight N Y Y Y Y Y Y Y Y Y Y aea7ba821a3ab1d05eb3e49a989efbbf7d776c7b
DroidKungFu1 Y Y Y Y Y Y Y Y Y Y Y aa279a5540f4592a2be8b3cfc456d068994cac4d
DroidKungFu2 N Y Y Y Y Y N Y Y Y Y b21f52062fd6805506edeb078e56812636ac4378
DroidKungFu3 N Y Y Y Y Y Y Y Y Y Y 4b2fc99330921fb64d5465ff1e6a7fc2264870c1
DroidKungFu4 N Y Y Y Y Y N Y Y Y Y a6803c0605b31d121c4214cfe468ab4e2e3b3100
DroidKungFuSapp N Y Y Y N N N Y Y N N f2b7ab8bbdc9cf07417cea4752e4b531c5d4b566
DroidKungFuUpdate N Y Y Y N Y N - Y N N 5e2fb0bef9048f56e461c746b6a644762f0b0b54
Endofday N Y Y Y Y Y N Y Y Y Y ed8a51225a3862e30817640ba7ec5b88ee04c98a
FakeNetflix N Y N Y Y Y N Y Y Y Y 0936b366cbc39a9a60e254a05671088c84bd847e
FakePlayer Y Y Y Y Y Y Y Y Y Y Y 24db6f496e87f038b48867808c51c830a6264517
GamblerSMS N Y Y Y N Y N Y Y Y N f1d8b11012df9b898ca2f9b0a5a97ef79b8a5e1d
Geinimi N Y - Y - Y N Y Y Y Y 825146fcd1da9271ae358fe5b3cd4df88acd8e14
GGTracker Y Y Y Y Y Y Y Y Y Y Y 0874540015f36d46973b684fcce14ec705b1b9e4
GingerMaster Y Y Y Y Y Y Y Y Y Y Y d1643fb08bbb8bf5759c73cdb4ea98800700950c
GoldDream N Y Y Y Y Y N Y Y Y Y b5c59a6db87300c6e22603ca32695261890369d0
Gone60 N Y Y Y Y Y N Y Y Y N a949cd6dc9b3eb02981155e5d569c9c987b48cde
GPSSMSSpy N Y Y Y Y Y Y Y Y Y Y 0eb4b7737df1b8b52213599e405d71c9be8a68ac
HippoSMS N Y Y Y N N N N Y Y Y bd7e85f5a0c39a9aeecc05dbc99a9e5c52150ba6
Jifake N Y Y Y Y Y Y Y Y Y Y 9440bb3da5e1ad862f357248b5da0c59dc7fc96b
jSMSHider N Y Y Y Y Y Y Y Y Y Y c23ace40470360ed76a5c392677ca12ed945e47e
KMin Y Y Y Y Y Y Y Y Y Y Y b8904c7845b9764b4eea4f9a4518217cd6672e7d
LoveTrap N Y Y Y Y Y Y Y Y Y Y e0d68c0e21eeeca1f8718e36749506b5ad9d96e4
NickyBot N Y Y Y N Y N Y Y Y N f8eac76f3c50be40945cc0562a537f5c661454cd
NickySpy N Y Y Y Y Y Y Y Y Y Y 1ce27fa92a313da39f1e31e97d3ac05a8d6ffe78
Pjapps N Y Y Y Y Y N Y Y Y Y a7db6728db3b8b793ca6513275bce2304a58090c
Plankton Y Y Y Y Y Y Y Y Y Y N a423d119f46cbaa5a80bba50df0b0c5f0afb68cf
RogueLemon N Y Y Y N N N N Y Y Y 18ea5584ffb185baf2bb5a87324ae46f7e40ac33
RogueSPPush N Y Y Y Y Y N Y Y Y N a0b2c608addd28bfc0c384b4ba74c5a1eb2c729f
SMSReplicator N Y Y Y Y Y N Y Y Y Y 005d5f6e94321de473d62706a94fbecf67c9f5f3
SndApps N Y Y Y Y Y N Y Y Y Y a8634a3392690a13146fbfe286e117b3e34bf96e
Spitmo N Y Y Y Y Y Y Y Y Y Y 604b0d72b9e995a7d95ef42561acb1ede4949fdb
Tapsnake N Y Y Y N Y N Y Y Y Y 73ddc408b518826064878dfc0064c4cd4fe512c0
Walkinwat Y Y Y Y Y Y Y Y Y Y Y 81781c90e79bf19ea0acb67df6b9bf636a520367
YZHC N Y Y Y Y Y Y Y Y Y Y aa9216c96ab29477966d0ff83b0326241f733734
zHash N Y Y Y N Y Y Y Y Y N 84e5cbcf2d44aa525b76ca47131af3eaf81a2bc3
Zitmo N Y Y Y Y Y Y Y Y Y Y c9368c3edbcfa0bf443e060f093c300796b14673
Zsone N Y Y Y Y Y N Y Y Y Y 00d6e661f90663eeffc10f64441b17079ea6f819
Total 10 49 46 49 38 45 25 47 48 46 38 -
Percentage (%) 20.41 100.00 93.88 100.00 77.55 91.84 51.02 95.92 97.96 93.88 77.55 -
Table 2: The detailed comparison with ten representative anti-virus engines

Figure 3: The comparison between the new app verification service (in Android 4.2) and ten other representative anti-virus engines

3. Look Ahead

By introducing this new app verification service in Android 4.2, Google has shown its commitment to continuously improve security on Android. However, based on our evaluation results, we feel this service is still nascent and there exists room for improvement.

Specifically, our study indicates that the app verification service mainly uses an app's SHA1 value and the package name to determine whether it is dangerous or potentially dangerous. This mechanism is fragile and can be easily bypassed. It is already known that attackers can change with ease the checksums of existing malware (e.g., by repackaging or mutating it). To be more effective, additional information about the app may need to be collected. However, how to determine the extra information for collection is still largely unknown -- especially given user privacy concerns.

In addition, the new app verification service largely relies on the server component (in the Google cloud) to determine whether an app is malicious or not. Unfortunately, it is not realistic to assume that the server side has all existing malware samples (especially with limited information such as app checksums and package names). From another perspective, the client side, in the current implementation, does not have any detection capability, which suggests possible opportunity for enhancement. However, due to the limited processing and communication power on mobile devices, we need to strike a delicate balance on how much detection capability can and should be offloaded.

Last but not least, we notice that VirusTotal (owned by Google) has not been integrated yet into this app verification service. From our measurement results, VirusTotal performs much better than this standalone service. For improved detection results, we expect such integration in the future will be helpful.

Last modified: December 10th, 2012