malicious_url_detection

This is an old revision of the document!

Table of Contents

Introduction
Member
public dataset
Intelligence website
Meeting
Reference

Introduction

Member : 蔡昀達, 廖其忻
Meeting :

Member

Name	Mail
蔡昀達	bb04902103@gmail.com
廖其忻	cayon.1318.96@hotmail.com
尹聖翔	b06902103@ntu.edu.tw

public dataset

ISCX-URL-2016(https://www.unb.ca/cic/datasets/url-2016.html)

kaggle

Phising URLS

Phishtank https://www.phishtank.com/developer_info.php
Open Phis https://openphish.com/

SPAM URLS

JWSPAMSPY http://www.joewein.de/sw/blacklist.htm

Malware URLS (These three does not update for a long time)

Benign URLS

Majestic - https://majestic.com/reports/majestic-million

Other Source

https://zeltser.com/malicious-ip-blocklists/

###### Black list

EtherAddressLooup - https://etherscamdb.info/api
Bambenek consulting - https://osint.bambenekconsulting.com/feeds/
firehol - http://iplists.firehol.org/
Spamhaus and DShield - http://www.squidblacklist.org/downloads/drop.malicious.rsc
squidguard - http://www.squidguard.org/blacklists.html
blackweb - https://github.com/maravento/blackweb

Intelligence website

White List - https://www.alexa.com/topsites

- Chrome外掛軟體 Googel WOT plugin

#####

- http://whois.domaintools.com

- https://www.urlvoid.com

- https://www.ipvoid.com/

- https://www.apivoid.com/api/domain-reputation/

#####

Black List (目前沒有可利用的資料)

- (主要是查IP)https://www.abuseipdb.com/

- (主要是Domain Name，僅參考)https://www.riskiq.com/platform/architecture/internet-data-sets/passive-dns/

- (黑名單太少，參考用)https://otx.alienvault.com/

Meeting

09/23 progress

tfidf results
1. accuracy : 98.3%
model explain mechanism
1. implemented : highlight trigger pattern
2. results1 (tree interpreter):

results2 (lime interpreter):

Year	Venue	Title	Link	Assign
2016	ACM	LIME:Why should i trust you?: Explaining the predictions of any classifier	PDF
2014	KAIS	Shapley sampling values:Explaining prediction models and individual predictions with feature contributions	PDF
2017	arxiv	DeepLIFT:Learning important features through propagating activation differences	PDF
2016	IEEE SP	QII:Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems	PDF
2015	PLoS ONE	Layer-wise relevance propagation:On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation	PDF
2010	PLoS ONE	Shapley regression values:Analysis of regression in game theory approach	PDF

09/16

check basline CNN model has high accuracy
build tfidf model
build explain feature
1. triggered pattern
2. malicious url family matching
phishing url survey

07/31

Collect training data
Collect whois information
manual feature
model desgin

Reference

Year	Venue	Title	Link	Assign
2017	arxiv	Malicious URL Detection using Machine Learning: A Survey	PDF

https://hackmd.io/RKXNLcvUQY2a-cQAESigqw?view

malicious_url_detection.1745414849.txt.gz · Last modified: 2025/04/23 21:27 by 193.70.81.103