Monday, August 29, 2016

The Voicemail Scammers Never Got Past Our OpenBSD Greylisting

We usually don't see much of the scammy spam and malware. But that one time we went looking for them, we found a campaign where our OpenBSD greylisting setup was 100% effective in stopping the miscreants' messages.

During August 23rd to August 24th 2016, a spam campaign was executed with what appears to have been a ransomware payload. I had not noticed anything particularly unusual about the bsdly.net and friends setup that morning, but then Xavier Mertens' post at isc.sans.edu Voice Message Notifications Deliver Ransomware caught my attention in the tweetstream, and I decided to have a look.

The first step was, as always, to grep the spamd logs, and sure, there were entries with from: addresses of voicemail@ in several of the domains my rigs are somehow involved in handling mail for.

But no message from voicemail@bsdly.net had yet reached any mailbox within my reach at that point. However, a colleague checked the quarantine at one of his private mail servers, and found several messsages from voicemail@ aimed at users in his domains.

Dissecting a random sample confirmed that the message came with an attachment with a .wav.zip filename that was actually a somewhat obfuscated bit of javascript, and I take others at their word that this code, if executed on your Microsoft system, would wreak havoc of some sort.

At this point, before I start presenting actual log file evidence, it is probably useful to sketch how the systems here work and interact. The three machines skapet, deliah and portal are all OpenBSD systems that run spamd in greylisting mode, and they sync their spamd data with each other via spamd's own synchronization mechanism.

All of those machines do greytrapping based on the bsdly.net list of spamtraps, and skapet has the additional duty of dumping the contents of its greytrapping generated blacklist to a downloadable text file once per hour. Any message that makes it past spamd is then fed to a real mail server that performs content filtering before handing the messages over a user's mailbox or, in the case of domains we only do the filtering for, forwards the message to the target domain's mail server.

The results of several rounds of 'grep voicemail $logfile' over the three spamd machines are collected here, or with the relatively uninteresting "queueing deletion of ..." messages removed, here.

From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois).

The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes.

There were none.

The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0.

Zero. Not a single one made it through even to content filtering.

That shouldn't have been a surprise.

After all I've spent significant time over the years telling people how effective greylisting is, and that the OpenBSD spamd version is the best of the breed.

You could take this episode as a recent data point that you are free to refer to in your own marketing pushes if you're doing serious business involving OpenBSD.

And if you're into those things, you will probably be delighted to learn, if you hadn't figured that out already, that a largish subset of the attempted deliveries were to addresses that were already in our published list of spamtrap addresses.

That means our miscreants automatically had themselves added to the list of trapped spammer IP addresses as intended.

If you're interested in how this works and why, I would suggest taking a peek at the OpenBSD web site, and of course I have a book out (available at that link and via better bookstores everywhere) that explains those things as well.

Relevant blog posts of mine include Keep smiling, waste spammers' time, Maintaining A Publicly Available Blacklist - Mechanisms And Principles, In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe and a few others, including the somewhat lengty Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools . To fully enjoy the experience of what these articles describe, you may want to get hold of your own CD set from the OpenBSD store.

And again, if you're doing business involving OpenBSD, please head over to the project's donations page and use one or more of the methods there to send the developers some much needed cash.

In addition to the files directly referenced in this article, some related files are  available from this directory. I'll be happy to answer any reasonable queries related to this material.

Good night and good luck.


Update 2016-08-30: I've been getting questions about the currently active campaign that has document@ as its sender. The same story there: I see them in the greylist and spamd logs, no trace whatsoever in later steps. Which means they're not getting anyhwere.

Update 2016-09-13: A quick glance at a tail -f'ed spamd log file reveals that today's fake sender of choice is CreditControl@. Otherwise same story as before, no variations. And of course, there may have been dozens I haven't noticed in the meantime.

Monday, August 8, 2016

Chinese Hunting Chinese Over POP3 In Fjord Country

Yes, you read that right: There is a coordinated effort in progress to steal Chinese-sounding users' mail, targeting machines at the opposite end of the Eurasian landmass (and probably elsewhere).

More specifically, here at bsdly.net we've been seeing attempts at logging in to the pop3 mail retrieval service using usernames that sound distinctively like Chinese names, and the attempts originate almost exclusively from Chinese networks.

This table lists the user names and corresponding real life names attempted so far:

Name Username
Chen Qiang chenqiang
Fa Dum fadum
Gao Dang gaodang
Gao Di gaodi
Gao Guan gaoguan
Gao Hei gaohei
Gao Hua gaohua
Gao Liu gaoliu
Gao Yang gaoyang
Gao Zhang gaozhang
He An hean
He Biao hebiao
He Bing hebing
He Chang hechuang
He Chao hechao
He Chen hechen
He Cheng hecheng
He Chun hechun
He Cong hecong
He Da heda
He Di hedi
He Die hedie
He Ding heding
He Dong hedong
He Duo heduo
He Fa hefa
He Ging heqing
He Guo heguo
He Han hehan
He Hao hehao
He Heng heheng
He Hui hehui
He Jia hejia
He Jian hejian
He Jiang hejiang
He Jie hejie
He Jin hejin
He Juan hejuan
He Kai hekai
He Kan hekan
He Kong hekong
He La hela
He Le hele
He Leng heleng
He Li heli
He Lian helian
He Lie helie
He Mu hemu
He Niang heniang
He Quan hequan
He Ran heran
He Sha hesha
He Shan heshan
He Shi heshi
He Si hesi
He Song hesong
He Xiao hexiao
He Yao heyao
He Yi heyi
He Yin heyin
He Yu heyu
He Yun heyun
He Zeng hezeng
He Zeng hezhan
He Zhang hezhangxxxx
He Zhe hezhe
He Zheng hezheng
He Zhi hezhi
He Zhong hezhong
He Zhuang hezhuang
Li An lian
Li Biao libiao
Li Bin libin
Li Bo libo
Li Cheng licheng
Li Chi lichi
Li Chong lichong
Li Chuang lichuang
Li Chun lichun
Li Da lida
Li Deng lideng
Li Di lidi
Li Die lidie
Li Ding liding
Li Dong lidong
Li Duo liduo
Li Fa lifa
Li Fang lifang
Li Fen lifen
Li Feng lifeng
Li Gang ligang
Li Gao ligao
Li Guan liguan
Li Guang liguang
Li Hai lihai
Li Ka lika
Li Kai likai
Li La lila
Li Le lile
Li Lei lilei
Li Lin lilin
Li Ling liling
Li Liu liliu
Li Long lilong
Li Man liman
Li Mei limei
Li Mu limu
Li Neng lineng
Li Niang liniang
Li Peng lipeng
Li Pian lipian
Li Qian liqian
Li Qu liqu
Li Rang lirang
Li Ren liren
Li Ru liru
Li Sha lisha
Li Shi lishi
Li Shuai lishuai
Li Shun lishun
Li Si lisi
Li Song lisong
Li Tao litao
Li Teng liteng
Li Tian litian
Li Ting liting
Li Wang liwang
Li Wei liwei
Li Wen liwen
Li Xiang lixiang
Li Xing lixing
Li Xiu lixiu
Li Ying liying
Li You liyou
Li Ze lize
Li Zeng lizeng
Li Zheng lizheng
Li Zhong lizhong
Li Zhu lizhu
Li Zhuang lizhuang
Li Zhuo lizhuo
Liang Min liangmin
Liang Ming liangming
Liang Qiang liangqiang
Liang Rui liangrui
Lin Chen linchen
Lin Cheng lincheng
Lin He linhe
Lin Hua linhua
Lin Huang linhuang
Lin Neng linneng
Lin Pian linpian
Lin Qu linqu
Lin Ru linru
Lin Zhang linzhang
Liu Bin liubin
Liu Duo liuduo
Liu Fang liufang
Liu Han liuhan
Liu Hao liuhao
Liu Heng liuheng
Liu Hong liuhong
Liu Hui liuhui
Liu Jia liujia
Liu Jiang liujiang
Liu Jiao liujiao
Liu Ju liuju
Liu Juan liujuan
Liu Kai liukai
Liu Kan liukan
Liu Kang liukang
Liu Ke liuke
Liu Kong liukong
Liu Lang liulang
Liu Long liulong
Liu Mu liumu
Liu Nuo liunuo
Liu Qin liuqin
Liu Qing liuqing
Liu Qiong liuqiong
Liu Rong liurong
Liu Sen liusen
Liu Sha liusha
Liu Shun liushun
Liu Si liusi
Liu Tian liutian
Liu Wang liuwang
Liu Wei liuwei
Liu Xia liuxia
Liu Xiu liuxiu
Liu Yao liuyao
Liu Yi liuyi
Liu Ying liuying
Liu Yu liuyu
Liu Yuan liuyuan
Liu Yun liuyun
Liu Zhen liuzhen
Liu Zheng liuzheng
Liu Zhi liuzhi
Liu Zun liuzun
Lou Liu luoliu
Lu Huang lihuang
Luo Chang luochuang
Luo Chen luochen
Luo Cheng luocheng
Luo Deng luochi
Luo Deng luodeng
Luo Di luodi
Luo Dian luodian
Luo Gao luogao
Luo Guai luoguai
Luo Hang luohuang
Luo Hua luohua
Luo Lie luolie
Luo Neng luoneng
Luo Pian luopian
Luo Qi luoqi
Luo Qin luoqin
Luo Qing luoqing
Luo Qu luoqu
Luo Rong luorong
Luo Ru luoru
Luo Rui luorui
Luo Shuang luoshuang
Luo Ting luoting
Luo Tong luotong
Luo Wang luowang
Luo Wei luowei
Luo Yang luoyang
Luo Ze luoze
Song Chen songchen
Song Cheng songcheng
Song Chuang songchuang
Song Da songda
Song Deng songdeng
Song Dian songdian
Song Die songdie
Song Fei songfei
Song Fen songfen
Song Gang songgang
Song Gao songgao
Song Guai songguai
Song Guan songguan
Song Guo songguo
Song Hai songhai
Song Han songhan
Song Hang songhang
Song He songhe
Song Hei songhei
Song Heng songheng
Song Hu songhu
Song Hua songhua
Song Jia songjia
Song Jiao songjiao
Song Jie songjie
Song Jin songjin
Song Jing songjing
Song Ka songka
Song Kan songkan
Song Kang songkang
Song Kong songkong
Song Lan songlan
Song Le songle
Song Lei songlei
Song Lian songlian
Song Liang songliang
Song Liang songliao
Song Liang songliang
Song Liao songliao
Song Lin songlin
Song Liu songliu
Song Meng songmeng
Song Ming songming
Song Mu songmu
Song Nan songnan
Song Neng songneng
Song Ning songning
Song Pian songpian
Song Pin songpin
Song Qi songqi
Song Qiang songqiang
Song Qing songqing
Song Qiu songqiu
Song Ran songran
Song Rong songrong
Song Rui songrui
Song Sha songsha
Song Shuai songshuai
Song Shuang songshuang
Song Song songsong
Song Song Jun songsongjun
Song Tao songtao
Song Teng songteng
Song Wang songwang
Song Wei songwei
Song Xi songxi
Song Xia songxia
Song Xiu songxiu
Song Ya songya
Song Yang songyang
Song Yong songyong
Song You songyou
Song Yuan songyuan
Song Yue songyue
Song Yun songyun
Song Zhe songzhe
Song Zhen songzhen
Song Zheng songzheng
Song Zhuang songzhuang
Tan Qian tangqian
Tang Bing tangbing
Tang Chi tangchi
Tang Chong tangchong
Tang Chuang tangchuang
Tang Cong tangcong
Tang Di tangdi
Tang Dian tangdian
Tang Duo tangduo
Tang Fa tangfa
Tang Fan tangfan
Tang Fang tangfang
Tang Fei tangfei
Tang Fen tangfen
Tang Feng tangfeng
Tang Gang tanggang
Tang Guai tangguai
Tang Guan tangguan
Tang Guang tangguang
Tang Guo tangguo
Tang Han tanghan
Tang Hao tanghao
Tang Hei tanghei
Tang Heng tangheng
Tang Hong tanghong
Tang Hu tanghu
Tang Hui tanghui
Tang Jie tangjie
Tang Jin tangjin
Tang Jing tangjing
Tang Ju tangju
Tang Ka tangka
Tang Kai tangkai
Tang Kan tangkan
Tang Kang tangkang
Tang Ke tangke
Tang Kong tangkong
Tang La tangla
Tang Lang tanglang
Tang Le tangle
Tang Leng tangleng
Tang Li tangli
Tang Lian tanglian
Tang Lie tanglie
Tang Lin tanglin
Tang Ling tangling
Tang Liu tangliu
Tang Long tanglong
Tang Mei tangmei
Tang Mo tangmo
Tang Mu tangmu
Tang Neng tangneng
Tang Niang tangniang
Tang Nuo tangnuo
Tang Peng tangpeng
Tang Pian tangpian
Tang Ping tangping
Tang Qian tangqian
Tang Qin tangqin
Tang Qu tangqu
Tang Quan tangquan
Tang Quing tangqing
Tang Rang tangrang
Tang Ren tangren
Tang Ru tangru
Tang Ruan tangruan
Tang Rui tangrui
Tang Sen tangsen
Tang Sha tangsha
Tang Shan tangshan
Tang Shi tangshi
Tang Shun tangshun
Tang Song tangsong
Tang Tang Jun tangtangjun
Tang Tao tangtao
Tang Tian tangtian
Tang Tian tangyan
Tang Wei tangwei
Tang Xi tangxi
Tang Xia tangxia
Tang Xing tangxing
Tang Xiong tangxiong
Tang Yan tangyan
Tang Yang tangyang
Tang Yao tangyao
Tang Yi tangyi
Tang Ying tangying
Tang Yong tangyong
Tang You tangyou
Tang Yue tangyue
Tang Yun tangyun
Tang Ze tangze
Tang Zeng tangzeng
Tang Zhang tangzhang
Tang Zhe tangzhe
Tang Zhen tangzhen
Tang Zun tangzun
Xie An xiean
Xie Bin xiebin
Xie Bo xiebo
Xie Chao xiechao
Xie Cong xiecong
Xie Da xieda
Xie Di xiedi
Xie Dian xiedian
Xie Die xiedie
Xie Ding xieding
Xie Dong xiedong
Xie Duo xieduo
Xie Fang xiefang
Xie Fei xiefei
Xie Feng xiefeng
Xie Gang xiegang
Xie Gao xiegao
Xie Guai xieguai
Xie Guan xieguan
Xie Hai xiehai
Xie Hang xiehang
Xie Heng xieheng
Xie Heng xieneng
Xie Heng xieheng
Xie Heng xieneng
Xie Hong xiehong
Xie Hu xiehu
Xie Hui xiehui
Xie Jia xiejia
Xie Jian xiejian
Xie Jiang xiejiang
Xie Jiao xiejiao
Xie Jie xiejie
Xie Jing xiejing
Xie Ju xieju
Xie Kai xiekai
Xie La xiela
Xie Leng xieleng
Xie Liang xieliang
Xie Lie xielie
Xie Lin xielin
Xie Ling xieling
Xie Long xielong
Xie Man xieman
Xie Meng xiemeng
Xie Min xiemin
Xie Ming xieming
Xie Na xiena
Xie Niang xieniang
Xie Peng xiepeng
Xie Pian xiepian
Xie Pin xiepin
Xie Qi xieqi
Xie Qing xieqing
Xie Qiong xieqiong
Xie Qiu xieqiu
Xie Qu xiequ
Xie Quan xiequan
Xie Ran xieran
Xie Ruan xieruan
Xie Rui xierui
Xie Sha xiesha
Xie Shuang xieshuang
Xie Si xiesi
Xie Tao xietao
Xie Ting xieting
Xie Tong xietong
Xie Wei xiewei
Xie Wen xiewen
Xie Xi xiexi
Xie Xiang xiexiang
Xie Xin xiexin
Xie Xing xiexing
Xie Xiu xiexiu
Xie Ya xieya
Xie Yi xieyi
Xie Yin xieyin
Xie Ying xieying
Xie Yong xieyong
Xie Yu xieyu
Xie Yue xieyue
Xie Zeng xiezeng
Xie Zhan xiezhan
Xie Zhang xiezhang
Xie Zhe xiezhe
Xie Zhuo xiezhuo
Zheng Nan zhengnan

That list of some 493 names is up to date as of this writing, 2016-08-23 early evening CEST. A few more turn up with the bursts of activity we have seen every day since June 19th, 2016.

A possibly more up to date list is available here. That's a .csv file, if that sounds unfamiliar, think of it as a platform neutral text representation (to wit, "Comma Separated Values") of a spreadsheet or database -- take a peek with Notepad.exe or similar if you're not sure. I'll be updating that second list along with other related data at quasi-random intervals as time allows and as long as interesting entries keep turning up in my logs.

If your name or username is on either of those lists, you would be well advised to change your passwords right now and to check breach notification sites such as Troy Hunt's haveibeenpwned.com or breachalarm.com for clues to where your accounts could have been compromised.

That's your scoop for now. If you're interested in some more background and data, keep reading.

If you are a regular or returning reader of this column, you are most likely aware that I am a Unix sysadmin. In addition to operating and maintaining variuos systems in my employers' care, I run a small set of servers of my own that run a few Internet-facing services for myself and a small circle of friends and family.

For the most part those systems are roundly ignored by the world at large, but when they are not, funny, bizarre or interesting things happen. And mundane activities like these sometimes have interesting byproducts. When you run a mail service, you are bound to find a way to handle the spam people will try to send, and about ten years ago I started publishing a blacklist of known spamming hosts, generated from attempts to deliver mail to a slowly expanding list of known bad, invalid, never to be deliverable addresses in the domains we handle mail for.

After a while, I discovered that the list of spamtrap addresses (once again, invalid and destined never to be deliverable, ever) had been hilariously repurposed: The local parts (the string before the @ or 'at sign') started turning up as usernames in failed attempts to log on to our pop3 mail retrieval service. That was enough fun to watch that I wrote that article, and for reasons known only to the operators of the machines at the other end, those attempts have never stopped entirely.

These attempts to log in as our imaginary friends is a strong contender for the most bizarre and useless activity ever, but when those attempts were no longer news, there was nothing to write about. The spamtrap login attempts make up sort of a background noise in the authentication logs, and whenever there is an attempt to log in as a valid user from somewhere that user is clearly not, the result is usually that an entire network (whatever I could figure out from whois output) would be blocked from any communication with our site for 24 hours.

There are of course also attempts to log in as postmaster, webmaster and other IDs, some RFC mandated, that most sites including this one would handle as aliases to make up the rest of the background noise.

Then recently, something new happened. The first burst looked like this in my logs (times given in local timezone, CEST at the time):

Jun 19 06:14:58 skapet spop3d[37601]: authentication failed: no such user: lilei - 59.54.197.34
Jun 19 06:15:01 skapet spop3d[46539]: authentication failed: no such user: lilei - 59.54.197.34
Jun 19 06:15:03 skapet spop3d[8180]: authentication failed: no such user: lilei - 59.54.197.34


-- and so on, for a total of 78 attempts to log in as the non-existing user lilei, in the space of about five minutes. A little later, a similar burst of activity came for the user name lika:

Jun 19 14:11:30 skapet spop3d[68573]: authentication failed: no such user: lika - 182.87.253.48
Jun 19 14:12:22 skapet spop3d[22421]: authentication failed: no such user: lika - 182.87.253.28
Jun 19 14:12:26 skapet spop3d[7587]: authentication failed: no such user: lika - 182.87.253.28
Jun 19 14:12:30 skapet spop3d[16753]: authentication failed: no such user: lika - 182.87.253.28


and so on, for a total of 76 attempts. Over the next few days I noticed an uptick in failed pop3 access attempts that were not for valid users and did not match any entry on our spamtraps list. Still, those attempts were for users that do not exist, and would produce no useful result so I did not do anything much about them.

It was only during the early weeks of July that it struck me that the user name attempted here

Jul  8 12:19:08 skapet spop3d[54818]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:28 skapet spop3d[1987]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:37 skapet spop3d[70622]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:49 skapet spop3d[31208]: authentication failed: no such user: lixing - 49.87.78.12


(a total of 54 attempts for that user name) might actually be based on the name of a Chinese person. "Li Xing" sounded plausible enough as a possible real person. It's perhaps worth noting that at the time I had just finished reading the first two volumes of Cixin Liu's The Three Body Problem, so I was a bit more in tune than usual with what could be plausible Chinese names than I had been. (And yes, the books are very much to my taste and I have the yet unpublished translation of the third volume on pre-order.)

Unsurprisingly, a quick whois lookup revealed that the machines that tried reading the hypothetical person Li Xing's mail all had IP addresses that belonged to Chinese networks.

Once I realized I might be on to a new pattern, I went back over a few days' worth of failed pop3 login attempts and found more than a handful of usernames that looked like they could be based on Chinese names. Checking the whois data for the IP addresses in those attempts, all turned out to be from Chinese networks.

That was in itself an interesting realization, but a small, random sample does not make for proof. In order to establish an actual data set, it was back to collecting data and analysing the content.

First, collect all log data on failed pop3 attempts for a long enough period that we have a reasonable baseline and can distinguish between the background noise and new, exciting developements.

The file bigauthlog is that collection of data. Digging through my archives going back in time, I stopped at January 16, 2016 for no other reason than this would be roughly six months' worth of data, probably enough to give a reasonable baseline and to spot anomalies.

If you've read the previous columns, you will be familiar with the scripts that produce various text and CSV reports from log data input: A text report of user names by number of access attempts, a CSV dump of the same, with first and last spotted, a text report of hosts attempting access, sorted by number of attempts, a CSV dump of the same, with first and last seen dates as for the user names.

But what I wanted to see was where the login attempts were coming from for which usernames, so I started extracting the unique host to username mappings. For each entry in this CSV file, there is a host and a user name it has tried at least once (if you import that somewhere, make sure you mark the Username column as text -- LibreOffice Calc at least becomes confused when trying to parse some of those strings). The data also records whether that particular username was part of the spamtrap database at the time. If you want to do that particular check on your own greytrapping database, any matching output from

$ doas spamdb | grep -i username@

on your greytrapper box will mean it is in your list. And then finally for each entry there is the expected extract from available whois info: network address range, the network name and the country.

The most useful thing to do with that little database is to play with sorting on various fields and field combinations. If you sort on the "In spamtraps" field, the supposed Chinese names turn up with "No"s, along with a few more random-seeming combinations.

While I was building the data set I decided to add those new usernames with @bsdly.net appended to the spamtraps, and this is what finally pushed the number of spamtraps past the 30,000 mark.

Just browsing the data or perhaps sorting by IP address will show you that the pop3 gropers are spread across a large number of networks in a number of countries and territories with numbers roughly in proportion to the size of that country or territory's economy. Some, such as a particular Mexican ISP and cable TV operator stand out as being slightly over-represented, and as expected networks in the US and China stand for a large number of the total.

If you sort on the In spamtraps field, you will see that a large number of the entries that were not in the spamtraps are the ones identified as Chinese personal names, but not all. Some of the No entries are the RFC mandated mailboxes, some are aliases that are in use here for other reasons, and finally more than a handful that would fit the general description of the rest of the spamtraps: Strings superficially resembling personal names or simply random strings. These may be parts of the potential spamtraps I missed while fishing  spamtrap candidates out of logfiles some time over the decade of weirdness that has gone into maintaining the spamtraps list.

But if you sort the data primarily on the fields Name, Country, and if you like IP address and User name, you will see that as anticipated the attempts on Chinese-sounding user names come exclusively from Chinese networks, except only the "Fa Dum" (fadum) user, which appears to have been attempted only twice (on June 6th) from an IP address registered in the USA and may very well be a misclassification on my part. That particular sorting, with duplicates removed, is the origin of the list of names and usernames given earlier in this article and this CSV file.

Now that we have established that the attempts at Chinese user names come exclusively from Chinese networks, the next questions become: Who are the cyber criminals behind this activity, and what are their motivations? And why are they bothering with hosts in faraway Europe to begin with?

For the first question, it is hard to tell from this perch, but whoever runs those attempts apparently have the run of large swathes of network real estate and seem to not take any special care not to be detected, other than of course distributing the attempts widely across the network ranges and coming in only in short bursts.

So are those attempts by, let us say the public sector, to steal political dissidents' email? Or perhaps, still with a public sector slant, simply hunting for any and all overseas assets belonging to Chinese nationals? Or are we simply seeing the activities of Chinese private sector cyber criminals who are trying out likely user names wherever they can find a service that listens?

Any of all of these things could be true, but in any case it's not unlikely that what we are seeing somebody trying to find new places where username and password combinations from a recent breach might work. After all, username and password combinations that have been verified to work somewhere are likely worth more on the market than the unverified ones.

Looking at the log entries, there are sequences there that could plausibly have been produced by humans typing at keyboards. Imagine if you please vast, badly lit and insufficiently ventilated Asian cyber-sweatshops, but I would not be too surprised to find that this is actually a highly automated operation, with timing tuned to avoid detection.

Security professionals have been recommending that people stop using the pop3 protocol since as long as I care to remember, but typing "pop3" into shodan.io still produces a whopping 684,291 results, meaning that the pop3 service is nowhere near as extinct as some would have preferred.

The large number of possible targets is a likely explanation for the burstiness of the activity we are seeing: with that many hosts to cover, the groping hosts will need to set up some sort of rotation, and in addition there is the need to stay below some volume of traffic per host in order to avoid detection. This means that what any one site sees is only a very small part of the total activity. The pop3 hunt for Chinese users is most likely not exclusive to the fjord country.

If you run a pop3 service, please do yourself a favor and check your setup for any weaknesses including any not yet applied updates, as you were about to do anyway. Once you've done that, take some moments to browse your logs for strange looking login attempts.

If you find something similar to what I've reported here, I would like to hear from you. Please note that at least one of the pop3 deaemons out there by default does not report the username for failed authentication attempts but notes that the username was unknown instead. Anyway, your war stories will be appreciated in email or comments.

If your name or username appears in the table at the start of this article or in this CSV file, please start checking for unusual activity involving your accounts and start changing passwords right away. Ask your service providers if they offer more secure alternatives, and if they do, consider using these alternatives. And as I mentioned earlier, do check breach notification sites such as haveibeenpwned.com or breachalarm.com for clues to help find out whether your data could be at risk in any of the services you do use. And of course, feedback in comments or email is welcome.

And finally, if you have information on one or more breaches that may have been the source of this list of likely Chinese user names, I'd like to hear from you too.

Good night and good luck.



I would like to thank Tore Nordstrand and Øystein Alsaker for valuable input on various aspects of this article.

The data referenced in this article will likely be updated on a roughly daily basis while the Chinese episode lasts. You can fetch them from the links in the article or from this directory, which also contains some trivial data extraction and data massaging scripts I use. If you find any errors or have any concerns, please let me know.


Saturday, April 23, 2016

Does Your Email Provider Know What A "Joejob" Is?

Anecdotal evidence seems to indicate that Google and possibly other mail service providers are either quite ignorant of history when it comes to email and spam, or are applying unsavoury tactics to capture market dominance.

The first inklings that Google had reservations about delivering mail coming from my bsdly.net domain came earlier this year, when I was contacted by friends who have left their email service in the hands of Google, and it turned out that my replies to their messages did not reach their recipients, even when my logs showed that the Google mail servers had accepted the messages for delivery.

Contacting Google about matters like these means you first need to navigate some web forums. In this particular case (I won't give a direct reference, but a search on the likely keywords will likely turn up the relevant exchange), the denizens of that web forum appeared to be more interested in demonstrating their BOFHishness than actually providing input on debugging and resolving an apparent misconfiguration that was making valid mail disappear without a trace after it had entered Google's systems.

The forum is primarily intended as a support channel for people who host their mail at Google (this becomes very clear when you try out some of the web accessible tools to check domains not hosted by Google), so the only practial result was that I finally set up DKIM signing for outgoing mail from the domain, in addition to the SPF records that were already in place. I'm in fact less than fond of either of these SMTP addons, but there were anyway other channels for contact with my friends, and I let the matter rest there for a while.

If you've read earlier instalments in this column, you will know that I've operated bsdly.net with an email service since 2004 and a handful of other domains from some years before the bsdly.net domain was set up, sharing to varying extents the same infrastructure. One feature of the bsdly.net and associated domains setup is that in 2006, we started publishing a list of known bad addresses in our domains, that we used as spamtrap addresses as well as publising the blacklist that the greytrapping generates.

Over the years the list of spamtrap addresses -- harvested almost exclusively from records in our logs and greylists of apparent bounces of messages sent with forged From: addresses in our domains - has grown to a total of 29757 spamtraps, a full 7387 in the bsdly.net domain alone. At the time I'm writing this 31162 hosts have attempted to deliver mail to one of those spamtrap addresses. The exact numbers will likely change by the time you read this -- blacklisted addresses expire 24 hours after last contact, and new spamtrap addresses generally turn up a few more each week. With some simple scriptery, we pick them out of logs and greylists as they appear, and sometimes entire days pass without new candidates appearing. For a more general overview of how I run the blacklist, see this post from 2013.

In addition to the spamtrap addresses, the bsdly.net domain has some valid addresses including my own, and I've set up a few addresses for specific purposes (actually aliases), mainly set up so I can filter them into relevant mailboxes at the receiving end. Despite all our efforts to stop spam, occasionally spam is delivered to those aliases too (see eg the ethics of running the traplist page for some amusing examples).

Then this morning a piece of possibly well intended but actually quite clearly unwanted commercial email turned up, addressed to one of those aliases. For no actually good reason, I decided to send an answer to the message, telling them that whoever sold them the address list they were using were ripping them off.

That message bounced, and it turns out that the domain was hosted at Google.

Reading that bounce message is quite interesting, because if you read the page they link to, it looks very much like whoever runs Google Mail doesn't know what a joejob is.

The page, which again is intended mainly for Google's own customers, specifies that you should set up SPF and DKIM for domains. But looking at the headers, the message they reject passes both those criteria:

Received-SPF: pass (google.com: domain of peter@bsdly.net designates 2001:16d8:ff00:1a9::2 as permitted sender) client-ip=2001:16d8:ff00:1a9::2;
Authentication-Results: mx.google.com;
       dkim=pass (test mode) header.i=@bsdly.net;
       spf=pass (google.com: domain of peter@bsdly.net designates 2001:16d8:ff00:1a9::2 as permitted sender) smtp.mailfrom=peter@bsdly.net
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bsdly.net; s=x;
 h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=OonsF8beQz17wcKmu+EJl34N5bW6uUouWw4JVE5FJV8=;
 b=hGgolFeqxOOD/UdGXbsrbwf8WuMoe1vCnYJSTo5M9W2k2yy7wtpkMZOmwkEqZR0XQyj6qoCSriC6Hjh0WxWuMWv5BDZPkOEE3Wuag9+KuNGd7RL51BFcltcfyepBVLxY8aeJrjRXLjXS11TIyWenpMbtAf1yiNPKT1weIX3IYSw=;

Then for reasons known only to themselves, or most likely due to the weight they assign to some unknown data source, they reject the message anyway.

We do not know what that data source is. But with more than seven thousand bogus addresses that have generated bounces we've registered it's likely that the number of invalid bsdly.net From: addresses Google's systems has seen is far larger than the number of valid ones. The actual number of bogus addresses is likely higher, though: in the early days the collection process had enough manual steps that we're bound to have missed some. Valid bsdly.net addresses that do not eventually resolve to a mailbox I read are rare if not entirely non-existent. But the 'bulk mail' classification is bizarre if you even consider checking Received: headers.

The reason Google's systems most likely has seen more bogus bsdly.net From: addresses than valid ones is that by historical accident faking sender email addresses in SMTP dialogues is trivial.

Anecdotal evidence indicates that if a domain exists it will sooner or later be used in the from: field of some spam campaign where the messages originate somewhere else completely, and for that very reason the SPF and DKIM mechanisms were specified. I find both mechanisms slightly painful and inelegant, but used in their proper context, they do have their uses.

For the domains I've administered, we started seeing log entries, and in the cases where the addresses were actually deliverable, actual bounce messages for messages that definitely did not originate at our site and never went through our infrastructure a long time before bsdly.net was created. We didn't even think about recording those addresses until a practical use for them suddenly appeared with the greytrapping feature in OpenBSD 3.3 in 2003.

A little while after upgrading the relevant systems to OpenBSD 3.3, we had a functional greytrapping system going, at some point before the 2007 blog post I started publishing the generated blacklist. The rest is, well, what got us to where we are today.

From the data we see here, mail sent with faked sender addresses happens continuously and most likely to all domains, sooner or later. Joejobs that actually hit deliverable addresses happen too. Raw data from a campaign in late 2014 that used my main address as the purported sender is preserved here, collected with a mind to writing an article about the incident and comparing to a similar batch from 2008. That article could still be written at some point, and in the meantime the messages and specifically their headers are worth looking into if you're a bit like me. (That is, if you get some enjoyment out of such things as discovering the mindbogglingly bizarre and plain wrong mail configurations some people have apparently chosen to live with. And unless your base64 sightreading skills are far better than mine, some of the messages may require a bit of massaging with MIME tools to extract text you can actually read.)

Anyone who runs a mail service and bothers even occasionally to read mail server logs will know that joejobs and spam campaigns with fake and undeliverable return addresses happen all the time. If Google's mail admins are not aware of that fact, well, I'll stop right there and refuse to believe that they can be that incompentent.

The question then becomes, why are they doing this? Are they giving other independent operators the same treatment? If this is part of some kind of intimidation campaign (think "sign up for our service and we'll get your mail delivered, but if you don't, delivering to domains that we host becomes your problem"). I would think a campaign of intimidation would be a less than useful strategy when there are alread antitrust probes underway, these things can change direction as discoveries dictate.

Normally I would put oddities like the ones I saw in this case down to a silly misconfiguration, some combination of incompetence and arrogance and, quite possibly, some out of control automation thrown in. But here we are seeing clearly wrong behavior from a company that prides itself in hiring only the smartest people they can find. That doesn't totally rule out incompetence or plain bad luck, but it makes for a very strange episode. (And lest we forget, here is some data on a previous episode involving a large US corporation, spam and general silliness. (Now with the external link fixed via the wayback machine.))

One other interesting question is whether other operators, big and small behave in any similar ways. If you have seen phenomena like this involving Google or other operators, I would like to hear from you by email (easily found in this article) or in comments.

Update 2016-05-03: With Google silent on all aspects of this story, it is not possible pinpoint whether the public whining or something else that made a difference, but today a message aimed at one of those troublesome domains made it through to its intended recipient.

Much like the mysteriously disappeared messages, my logs show a "Message accepted for delivery" response from the Google servers, but unlike the earlier messages, this actually appeared in the intended inbox. In the meantime one of one of the useful bits of feedback I got on the article was that my IPv6 reverse DNS was in fact lacking. Today I fixed that, or at least made a plausible attempt to (you're very welcome to check with tools available to you). And for possibly unrelated reasons, my test message made it through all the way to the intended recipient's mailbox.

[Opinions offered here are my own and may or may not reflect the views of my employer.]


I will be giving a PF tutorial at BSDCan 2016, and I welcome your questions now that I'm revising the material for that session. See this blog post for some ideas (note that I'm only giving the PF tutorial this time around).

Wednesday, March 9, 2016

Domain Name Scams Are Alive And Well, Thank You

Is somebody actually trying to register your company name as a .cn or .asia domain? Not likely. And don't pay them.

It has been a while since anybody tried to talk me into registering a domain name I wasn't sure I wanted in the first place, but it has happened before. Scams more or less like the Swedish one are as common as they are transparent, but apparently enough people take the bait that the scammers keep trying.

After a few quiet years in my backwater of the Internet, in March of 2016, we saw a new sales push that came from China. The initial contact on March 4th, from somebody calling himself Jim Bing read (preserved here with headers for reference, you may need MIME tools to actually extract text due to character set handling),

Subject: Notice for "bsdly"

Dear CEO,

(If you are not the person who is in charge of this, please forward this to your CEO, because this is urgent, Thanks)

We are a Network Service Company which is the domain name registration center in China.
We received an application from Huabao Ltd on March 2, 2016. They want to register " bsdly " as their Internet Keyword and " bsdly.cn "、" bsdly.com.cn " 、" bsdly.net.cn "、" bsdly.org.cn " 、" bsdly.asia " domain names, they are in China and Asia domain names. But after checking it, we find " bsdly " conflicts with your company. In order to deal with this matter better, so we send you email and confirm whether this company is your distributor or business partner in China or not?

Best Regards,

Jim
General Manager
Shanghai Office (Head Office)
8006, Xinlong Building, No. 415 WuBao Road,
Shanghai 201105, China
Tel: +86 216191 8696
Mobile: +86 1870199 4951
Fax: +86 216191 8697
Web: www.cnweb-registry.com


The message was phrased a bit oddly in parts (as in, why would anybody register an"internet keyword"?), but not entirely unintelligible as English-language messages from Asians sometimes are.

I had a slight feeling of deja vu -- I remembered a very similar message turning up in 2008 while we were in the process of selling the company we'd started a number of years earlier. In the spirit of due diligence (after asking the buyer) we replied then that the company did not have any plans for expanding into China, and if my colleagues ever heard back, it likely happened after I'd left the company.

This time around I was only taking a break between several semi-urgent tasks, so I quickly wrote a reply, phrased in a way that I thought would likely make them just go away (also preserved here):

Subject: Re: Notice for "bsdly"
 
Dear Jim Bing,

We do not have any Chinese partners at this time, and we are not
currently working to establish a presence in Chinese territory. As to
Huabao Ltd's intentions for registering those domains, I have no idea
why they should want to.

Even if we do not currently plan to operate in China and see no need
to register those domains ourselves at this time, there is a risk of
some (possibly minor) confusion if those names are to be registered
and maintained by a third party. If you have the legal and practical
authority to deny these registrations that would be my preference.

Yours,
Peter N. M. Hansteen


Then on March 7th, a message from "Jiang zhihai" turned up (preserved here, again note the character set issues):

Subject: " bsdly "
Dear Sirs,

Our company based in chinese office, our company has submitted the " bsdly " as CN/ASIA(.asia/.cn/.com.cn/.net.cn/.org.cn) domain name and Internet Keyword, we are waiting for Mr. Jim's approval. We think these names are very important for our business in Chinese and Asia market. Even though Mr. Jim advises us to change another name, we will persist in this name.

Best regards

Jiang zhihai

Now, if they're in a formal process of getting approval for a that domain name, why would they want to screw things up by contacting me directly? I was beginning to smell rat, but I sent them an answer anyway (preserved here):

Subject: Re: " bsdly "

Dear Jiang zhihai,

You've managed to make me a tad curious as to why the "bsdly" name
would be important in these markets.

While there is a very specific reason why I chose that name for my
domains back in 2004, I don't see any reason why you wouldn't be
perfectly well served by picking some other random sequence of characters.

So out of pure curiosity, care to explain why you're doing this?

Sincerely,
Peter N. M. Hansteen

Yes, that domain name has been around for a while. I didn't immediately remember exactly when I'd registered the domain, but a quick look at the whois info (preserved here) confirmed what I thought. I've had it since 2004.

Anyone who is vaguely familiar with the stuff I write about will have sufficient wits about them to recognize the weak pun the domain name is. If "bsdly" has any other significance whatsoever in other languages including the several Chinese ones, I'd genuinely like to know.

But by now I was pretty sure this was a scam. Registrars may or may not do trademark searches before registering domains, but in most cases the registrar would not care either way. Domain registration is for the most part a purely technical service that extends to making sure whether any requested domains are in fact available, while any legal disputes such as trademark issues could very easily be sent off to the courts for the end users at both ends to resolve. The supposed Chinese customer contacting me directly just does not make sense.

Then of course a few hours after I'd sent that reply, our man Jim fired off a new message (preserved here, MIME and all):

Subject: CN/ASIA domain names & Internet Keyword

Dear Peter N. M. Hansteen,

Based on your company having no relationship with them, we have suggested they should choose another name to avoid this conflict but they insist on this name as CN/ASIA domain names (asia/ cn/ com.cn/ net.cn/ org.cn) and internet keyword on the internet. In our opinion, maybe they do the similar business as your company and register it to promote his company.
According to the domain name registration principle: The domain names and internet keyword which applied based on the international principle are opened to companies as well as individuals. Any companies or individuals have rights to register any domain name and internet keyword which are unregistered. Because your company haven't registered this name as CN/ASIA domains and internet keyword on the internet, anyone can obtain them by registration. However, in order to avoid this conflict, the trademark or original name owner has priority to make this registration in our audit period. If your company is the original owner of this name and want to register these CN/ASIA domain names (asia/ cn/ com.cn/ net.cn/ org.cn) and internet keyword to prevent anybody from using them, please inform us. We can send an application form and the price list to you and help you register these within dispute period.

Kind regards

Jim
General Manager
Shanghai Office (Head Office)
8006, Xinlong Building, No. 415 WuBao Road,
Shanghai 201105, China
Tel: +86 216191 8696
Mobile: +86 1870199 4951
Fax: +86 216191 8697
Web: www.cnwebregistry.com

So basically he's fishing for me to pony up some cash and register those domains myself through their outfit. Quelle surprise.

I'd already checked whether my regular registrar offers .cn registrations (they don't), and checking for what looked like legitimate .cn domain registrars turned up that registering a .cn domain would likely cost to the tune of USD 35. Not a lot of money, but more than I care to spend (and keep spending on a regular basis) on something I emphatically do not need.

So I decided to do my homework. It turns out that this is a scam that's been going on for years. A search on the names of persons and companies turned up Matt Lowe's 2012 blog post Chinese Domain Name Registration Scams with a narrative identical to my experience, with only minor variations in names and addresses.

Checking whois while writing this it turns out that apparently bsdly.cn has been registered:

[Wed Mar 09 20:34:34] peter@skapet:~$ whois bsdly.cn
Domain Name: bsdly.cn
ROID: 20160229s10001s82486914-cn
Domain Status: ok
Registrant ID: 22cn120821rm22yr
Registrant: 徐新荣
Registrant Contact Email: 1725093@qq.com
Sponsoring Registrar: 浙江贰贰网络有限公司
Name Server: ns1.22.cn
Name Server: ns2.22.cn
Registration Time: 2016-02-29 20:55:09
Expiration Time: 2017-02-28 20:55:09
DNSSEC: unsigned

But it doesn't resolve more than a week after registration:

[Wed Mar 09 20:34:47] peter@skapet:~$ host bsdly.cn
Host bsdly.cn not found: 2(SERVFAIL)


That likely means they thought me a prospect and registered with an intent to sell, and they've already spent some amount of cash they're not getting back from me. I think we can consider them LARTed, however on a very small scale.

What's more, none of the name servers specified in the whois info seem to answer DNS queries:

[Wed Mar 09 20:35:36] peter@skapet:~$ dig @ns1.22.cn bsdly.cn any

; <<>> DiG 9.4.2-P2 <<>> @ns1.22.cn bsdly.cn any
; (2 servers found)
;; global options:  printcmd
;; connection timed out; no servers could be reached
[Wed Mar 09 20:36:14] peter@skapet:~$ dig @ns2.22.cn bsdly.cn any

; <<>> DiG 9.4.2-P2 <<>> @ns2.22.cn bsdly.cn any
; (2 servers found)
;; global options:  printcmd
;; connection timed out; no servers could be reached



So summing up,
  • This is a scam that appears to have been running for years.
  • If something similar to those messages start turning up in your inbox, the one thing you do not want to do is to actually pay for the domains they're offering.

    Most likely you do not need those domains, and it's easy to check how far along they are in the registration process. If you have other contacts that will cheaply and easily let you register those domains yourself, there's an element of entertainment to consider. But keep in mind that automatic renewals for domains you don't actually need can turn irritating once you've had a few laughs over the LARTing.
  • If you are actually considering setting up shop in the markets they're offering domains for and you receive those messages before you've come around to registering domains matching your trademarks, you are the one who's screwed up.
If this makes you worried about Asian cyber-criminals or the Cyber Command of the People's Liberation Army out to get your cyber-whatever, please calm down.

Sending near-identical email messages to people listed in various domains' whois info does not require a lot of resources, and as Matt says in his article, there are indications that this could very well be the work (for some values of) of a single individual. As cybercrime goes, this is the rough equivalent of some petty, if unpleasant, street crime.

I'm all ears for suggestions for further LARTing (at least those that do not require a lot of effort on my part), and if you've had similar experiences, I'd like to hear from you (in comments or email). Do visit Matt Lowe's site too, and add to his collection if you want to help him keep track.

And of course, if "Jim Bing" or Jiang zhihai" actually answer any of my questions, I'll let you know with an update to this article.

Update 2016-03-15: As you can imagine I've been checking whether bsdly.cn resolves and the registration status of the domain via whois at semi-random intervals of at least a few hours since I started the blog post. I was a bit surprised to find that the .cn whois server does not answer requests at the moment:

[Tue Mar 15 10:23:31] peter@portal:~$ whois bsdly.cn
whois: cn.whois-servers.net: connect: Connection timed out


It could of course be a coincidence and an unrelated technical issue. I'd appreciate independent verification.