Author Archives: Tim H

About Tim H

I am a Large Fluffy Chicken called Bob.

Pushbullet Logo

Debugging Issues with Pushbullet and Google Play Services

Background

I’m posting this because every day there’s a new post on /r/pushbullet about people not getting their Pushbullet messages until they open the Pushbullet app.  There’s a number of reasons that this issue can arise, some of them are unrelated to Pushbullet itself, but will cause the symptoms, some are related to Pushbullet though no definitive answer has been found for this yet.  This article will help you to diagnose issues unrelated to Pushbullet – if following the steps and suggestions don’t help then escalating to Pushbullet’s support would be a good next step.

To kick off this article, I’ll explain briefly how push notifications work for most applications, but not all on Android.  Almost every Android phone out there has Google Play Services installed.  This “App” is pre-installed on your phone and is the support library that all Google Apps and many others rely on.  One of it’s many jobs is to provide the “Google Cloud Messaging” (GCM) service.

2019 update.  GCM is now FCM (Firebase Cloud Messaging) but I’m not going to update this blog post to change it all, it’s still the same basic thing.

This, very simply, allows a server/service hosted somewhere to send a message to Google, and Google will then, using GCM, push a message to the user’s phone.

The reason that GCM is so useful is that apps can register with it when they’re installed.  This means that each app doesn’t have to have it’s own network/infrastructure for pushing messages to your phone, but rather there is a single push service on your phone that all apps can use.  This helps with battery life and means that App developers can focus on writing apps, not having to also host and maintain a network dedicated to pushing messages to phones. It also means your phone only has to have one network connection open to get push messages, instead of each app having to keep a network connection open, which is a drain on resources and battery life.

The Problem

Now the problem comes about with Pushbullet when it doesn’t get messages from GCM.  These messages should “Wake it up” so that you instantly get the push on your phone.

But if Pushbullet doesn’t get the GCM notification, it has no idea that anything new has happened.  It only realises when you open the app and it Sets up it’s own connection to the Pushbullet Servers and gets the latest notifications.  In fact, that’s all a GCM message is, a message to say “Hey, wake up and check your server for the message waiting for you, Pushbullet”.

So if this isn’t working for you, there’s a few things you can do to debug (and hopefully resolve) the issues so that your Pushbullet app starts working as you expect.  Depending on the problem, you might find other apps you didn’t even realise were lagging seem to be a lot faster too!

Debugging Google Play Services

Thankfully, Google have included a way to find out the status of Google Play Services and whether or not it has an active connection to the GCM services.  Type the following code into your dialer and you’ll open up the Google Play Services status page:

Dialer Debug Code
*#*#426#*#*

You will open up a screen that looks like this:

GCM Connected

GCM Connected

Notice the following things about this image:

Device ID: The Device ID that Google has assigned your device.
Connected: This is the key we’re looking for! We want to see that it says connected and a lot of information about how it’s connected.

This is what a bad/broken/non-working GCM screen looks like:

GCM Disconnected

GCM Disconnected

What follows is a list of  reasons (and some workarounds/fixes) as to why Google Play Services might not have a connection…

Why doesn’t it work?

IPv6 is enabled. For me, IPv6 works fine. But for a lot if people, if their router/WiFi is giving them an IPv6 connection, but it’s not properly routed, then it won’t work. But for reasons unknown, Google will keep trying to use the IPv6 connection, even though it’s broken. Sadly on Android the only way to disable IPv6 is via root methods, there is no simple way for a non-rooted person to do it. The best option if you’re not rooted is to disable the router/WiFi you’re connected to from giving you an IPv6 address. Of course if you’re at work etc then getting the IT people to do this is probably an impossible task.

The GCM Ports are Firewalled. This is less likely, but certainly possible if your work environment only allows port 80/443 out.  GCM uses TCP Port 5228 (The standard Jabber Port!), but it can also sometimes use TCP Port 5229 and 5230.  If these ports are blocked, you won’t get a stable GCM connection.

Your Firewall has dumb sessions timeout. I’m not sure how valid this one is with later versions of GCM, but if you have a firewall that times out TCP sessions after 5 minutes, you could well have issues with GCM which only sends keepalives every ~29 minutes (This is not confirmed yet).

The Fixes

Use Mobile data (disable WiFi).

Annoying, but great as a quick workaround to see if it fixes the problem.

Run a VPN.

This is what I did at a place of employment where IPv6 was broken. The VPN gives your phone a IPv4 only address and Google Play Services will connect via it and work fine.

Is GCM not the issue?

If you’ve looked at the above but you’re finding that you’re still missing push messages, one “fix” is to uninstall and re-install Pushbullet.  Why this is required is still under investigated by the PB devs.  A way to test if only Pushbullet is affected is to get someone to send you a test Google Hangouts message, or to send yourself a Gmail (has to be Gmail, not an IMAP account).  These both notify your phone of a new message by using GCM.  If you’re getting Gmail/Hangouts notifications instantly with no delay, but getting delays with Pushbullet, then the issue isn’t GCM and something else is wrong.  Might be time to contact the PB Devs and see if there’s any information you can give them to help debug the issue.

Good luck!

Running Your Own Mailserver(s)

This post is now out of date! Running your own mailserver is even easier these days thanks to rspamd. You literally plug rspamd into your mailserver using a milter, it’s a single line in postfix, and rspamd rolls up everything below in the smtpd_recipient_restrictions section and then some more, plus it’s got a nice webGUI. 

Rspamd: Zero spam, Rapid delivery.

Running your own mailserver isn’t that hard.  I always have a chuckle when I read people say “Why would you do it yourself, there’s so much management?”  That’s crap, they just don’t know how to do it.

A mailserver basically runs itself, there’s plenty of online tools to verify that you’re not an open relay, that you’ve configured your TLS settings correctly etc.  Plenty of configuration guides (another is included below) to show you how to lock it down so that it’s not a spam wind-up-and-go machine.

I run 3 mailservers (1 primary, 2 backup).  They all talk a single Greylisting Daemon, set to allow mail through after 1 minute.  Should the greylisting daemon not be available, the servers are set to accept the mail.

Before greylisting takes place however, the mail gets a bunch of checks.  First of all, High Quality DNS Whitelists are checked, if a server is listed in here it can be Trusted to not be sending Spam.  Then Blacklists are checked.  Then remaining whitelists are checked, if a server is listed it is allowed to bypass Greylisting. NOTE: Don’t use SORBS! Their data is out of date and crap. Way too many false positives. Avoid at all costs. I made this mistake once.

Here’s the full logic that all my mail servers use.  You have to ensure you share the greylisting database correctly, otherwise you’ll end up delaying mail much longer than necessary.

  1. REJECT anyone who doesn’t say HELO
  2. REJECT invalid Hostnames in HELO
  3. REJECT senders not using <user@domain.domain> correctly as per RFC821.
  4. REJECT Unknown Recipients
  5. ALLOW from a list of Known IPs (Backup MX hosts, other trusted devices)
  6. ALLOW from Authenticated Senders (To send mail from anywhere, using username/password)
  7. ALLOW from a set of DNS Whitelists that state an entry in their list can be considered “Non-Spam”
  8. REJECT from a list of DNS Blacklists
  9. ALLOW from a second set of DNS Whitelists that are verified to be SMTP servers (skips the need to greylist)
  10. Send to Greylisting Daemon to ACCEPT/DELAY
  11. ACCEPT

Step 7 could be amalgamated with step 9, but I prefer to “trust” the lists of known, trusted  email senders before checking blacklists, as sometimes blacklists can be a bit “over zealous” in their flagging a server a spam, i.e. one that sends newsletters etc.  This way I get check of this logic:

  1. Verified quality sender – ACCEPT.
  2. Check for blacklists – DENY.
  3. Verified RFC compliant SMTP server, skip greylisting (because we know it’ll just retry anyway, no point delaying) – ACCEPT.
  4. Send to Greylisting for DELAY/ACCEPT decision.

With these rules in place, I get almost zero spam making it through, probably 2-3 spams per week.  However the amount of mail that is rejected via the Blacklists and the Greylisting is amazing, in the thousands per day.

Once I’ve finally accepted a mail, I send it to Spamassassin for checking, just to be sure.

The other thing that’s important that I’ve done fairly recently (in the last couple of years) is to ensure that Postfix is setup correctly to send and receive mail using encryption. SSLv2 and SSLv3 are disabled, weak ciphers are disabled, Perfect Forward Secrecy is enabled.

Here’s my main.cf for Postfix.

smtpd_banner = $myhostname ESMTP - SMTP BANNER GREETING
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Send a warning if mail is delayed after 1 hour
delay_warning_time = 1h
# If mail can't be delivered after 7 days, we give up
maximal_queue_lifetime = 7d

readme_directory = no
inet_protocols = ipv4

# Incoming Mail
smtpd_tls_cert_file=/etc/letsencrypt/live/<hostname>/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/<hostname>/privkey.pem
smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtpd_tls_dh1024_param_file = ${config_directory}/dh2048.pem
smtpd_tls_dh512_param_file = ${config_directory}/dh512.pem
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_session_cache_timeout = 604800
smtpd_tls_eecdh_grade = strong
smtpd_tls_security_level = may
smtpd_tls_ciphers = high
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_exclude_ciphers = aNULL, eNULL, RC4
#Don't offer Auth until STARTTLS has setup
smtpd_tls_auth_only = yes

#Ask for a Client Cert
smtpd_tls_ask_ccert = yes

# Outgoing Mail
smtp_tls_cert_file=/etc/letsencrypt/live/<hostname>/fullchain.pem
smtp_tls_key_file=/etc/letsencrypt/live/<hostname>/privkey.pem
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_use_tls=yes
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_session_cache_timeout = 604800
smtp_tls_security_level = may
smtp_tls_ciphers = high
smtp_tls_protocols = !SSLv2, !SSLv3
smtp_tls_exclude_ciphers = aNULL, eNULL, RC4

#TLS Params
tls_preempt_cipherlist = yes

myhostname = <my hostname>
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = <hostnames I accept mail for>
virtual_alias_domains = <other domains I host>
virtual_alias_maps = hash:/etc/postfix/virtual
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 <backup MX1> <backup MX2>
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
html_directory = no

# Procmail to deliver
mailbox_command = /usr/bin/procmail

# sasl! You want to eat it!
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_authenticated_header = yes

# Mailing Signing with OpenDKIM
milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:12301 # Don't copy unless you have setup DKIM
non_smtpd_milters = inet:localhost:12301 # Don't copy unless you have setup DKIM

# Proper Mail Protocol Please
strict_rfc821_envelopes = yes

# Verify? No thanks!
disable_vrfy_command = yes

# Demand a polite conversation!
smtpd_helo_required = yes

# Delay before reject
smtpd_delay_reject = yes

smtpd_helo_restrictions = permit_mynetworks,
 reject_non_fqdn_hostname,
 reject_invalid_hostname,
 permit

smtpd_recipient_restrictions =
 reject_invalid_hostname,
 reject_unknown_recipient_domain,
 reject_unauth_pipelining,
 permit_mynetworks,
 permit_sasl_authenticated,
 reject_unauth_destination,
 check_client_access cidr:/etc/postfix/rbl_override,
 permit_dnswl_client iadb.isipp.com=127.0.1.255,
 permit_dnswl_client sa-trusted.bondedsender.org,
 permit_dnswl_client sa-accredit.habeas.com,
 permit_dnswl_client list.dnswl.org=127.0.[0..255].[2..3],
 permit_dnswl_client hostkarma.junkemailfilter.com=127.0.0.[1;5],
 reject_rhsbl_reverse_client dbl.spamhaus.org,
 reject_rhsbl_sender dbl.spamhaus.org,
 reject_rhsbl_client dbl.spamhaus.org,
 reject_rbl_client zen.spamhaus.org,
 reject_rbl_client dnsbl-1.uceprotect.net,
 reject_rbl_client psbl.surriel.com,
 reject_rbl_client hostkarma.junkemailfilter.com=127.0.0.2,
 reject_rbl_client bl.mailspike.net,
 reject_rbl_client b.barracudacentral.org,
 reject_rbl_client truncate.gbudb.net,
 permit_dnswl_client iadb.isipp.com=127.0.2.[1;2],
 permit_dnswl_client iadb.isipp.com=127.3.100.[5..100],
 permit_dnswl_client wl.mailspike.net,
 permit_dnswl_client list.dnswl.org,
 permit_dnswl_client hostkarma.junkemailfilter.com=127.0.0.3,
 check_policy_service inet:127.0.0.1:10060,

message_size_limit = 81920000

Once configured like that, it’s set and forget pretty much.  I occasionally check the logs to ensure that nothing is being greylisted due to the dumb policy some senders have of retrying each time from a DIFFERENT IP Address.  When I do see such stupidity I usually just add the sending /24 network to the Greylist Whitelist.

The final thing to note is that you should run your own caching DNS server.  If you’re using your ISPs, or a big public provider like Google etc, then the black/whitelists often won’t work as they implement rate-limiting against abuse, and the big public name-servers are almost always blocked.  Running your own small caching DNS server is easy and will give you a working RBL setup.

 

Update: 11/4/2017 – Turns out Protected Sky are just a bunch of rip-off merchants. Removed them from my list of checked RBLs.

Read the comments

Always Read The Comments

I think the “Never read the comments” mentality is so dumb.  Why wouldn’t you read the comments?  It shows how people out there, in the community, in the wide-world, are really thinking.  I’m not saying you have to agree with the comments.  They’re no doubt horrid, vile and some of them I’m sure are disgusting and not worth reading.

But really, read the damn comments, don’t be so high and mighty. You don’t have to change your mind, but at least learn what others are thinking, even if you disagree.

Mosquitoes

The new office has a lot of mozzies. Heaps of them. They’re everywhere. I went nuts downstairs with a can of spray before, nuked about 20 of the little bastards. That plus the “pidgeon problem” makes for an interesting new office.

Fucking Mosquito

Fucking Mosquito

Still better than the old one though. Now we have aircon and a decent layout. And a landlord who is isn’t so crazy as to park a moving truck in. On purpose. When we’re moving out.

Nutter.

grsecurity logo

grsecurity + phc undervolting patch

In an effort to get the same performance from this laptop as I used when it ran windows (and I could use the excellent ThrottleStop program to undervolt), I did some investigation into how to undervolt while running Linux.

Linux PHC Project

Linux PHC Project

The solution turns out to be the Linux PHC Project.  There wasn’t a patch for Linux 4.1.4, but looking in the forums a bit I found a patch for 3.1, which cleanly applied to 4.1.4.  I didn’t bother with their suggested way of building a module, I just patched the file in the Linux source tree directly and rebuilt the kernel.  Because the tree was already patched with grsecurity, the diff spat out a bunch of offset changes, but all the changes applied correctly which is all that matters.  I checked to ensure that nothing grsecurity/PaX related had been mangled.

Recompile, reboot and now I have the following sysfs interfaces:

/sys/devices/system/cpu/cpu*/cpufreq/phc_default_rawcontrol
/sys/devices/system/cpu/cpu*/cpufreq/phc_default_vids
/sys/devices/system/cpu/cpu*/cpufreq/phc_rawcontrols
/sys/devices/system/cpu/cpu*/cpufreq/phc_version
/sys/devices/system/cpu/cpu*/cpufreq/phc_vids

Writing same values there as I used to use under Windows and now my laptop is so much cooler.  Before when I built a kernel it was getting up towards 90c and it was limiting the CPU to keep itself getting hotter.  Now I can build a kernel, I get nowhere over 75c and it stays at the full 2.20Ghz the entire time.

Original VIDs: cat phc_default_vids 
47 41 28 18 11
New VIDs: cat phc_vids 
30 27 23 13 6

Excellent!  Thanks Linux PHC Project.

grsecurity logo

grsecurity on a Xubuntu laptop

I installed xubuntu on my now very old Dell XPS m1330 the other day.  Windows 10 just wasn’t cutting it for me, though that’s probably because I had an excess of crap installed.  It was taking upwards of 5 minutes to reboot though, so I thought I’d try something else.

Xubuntu was the obvious choice, Gnome can die in a fire and KDE while nice is too UI heavy for what I want.  I love XFCE, it’s small, clean and does a great job, so xubuntu got the nod.

Once installed and working, I then downloaded the 4.1.3 kernel source and the latest grsecurity patch for it.  Patched the source and fired up make menuconfig.  Ubuntu being ubuntu it comes with pretty much every freaking option, module and setting defaulted to yes.  Rather than piss about making a nice small custom kernel, I just went with all the defaults, then turned on pretty much every grsec feature.  The few items I kept disabled are:

They’re only minor things (well, the RBAC system isn’t really “minor”), all of the main memory protection features (thanks, PaX) and the other grsecurity hardening features are enabled.

Then it’s just a matter of making sure all the right packages are installed to be able to do a “make-kpkg –initrd kernel_image” and waiting for a very long time.  Oh and it helps to set the CONCURRENCY_LEVEL to 2, which is how many Core’s the CPU has.  Then you wait about 3 hours…

Finally you end up with a .deb that you install and off you go.  Install it and reboot and….

It worked first go. Not that I really expected otherwise.  The only problems encountered are the expected ones, some binaries don’t like the hardened memory protections, so those protections have to turned off on a per-binary basis.  So it’s apt-get install pax-utils and apt-get install paxctl.

The binaries I adjusted flags for are:

  • chrome
  • thunderbird
  • python3
paxctl -cpm /path/to/binary

c: creates a pax header, m: disables mProtect, p: disables pageexec

The only problems I’ve faced apart from this are issues with the sound module.  Under a default ubuntu kernel the sound just works.  Under my compiled kernel, the module needs to be removed and re-added for sound to work, and then it fails if you suspend the laptop.  I’m 99% sure this isn’t anything to do with grsecurity, but rather the fact it’s a vanilla kernel source, not a heavily-patched ubuntu kernel source with fixes for all those sorts of things.  I’ll get to the bottom of it at some stage.

But the laptop works and works well.  I’m not using the proprietary Nvidia drivers, just the nouveau ones.  Suspend works.  It’s still fast and browsing is quick, despite all of the PaX and grsecurity options turned on, some of which have a known performance impact (I’m looking at you Userland Dereference and Memory Sanitize)

The whole process has been easy, quick and painless.  The hardest part has been waiting for the kernel to compile.  When I have a bit more free time I’ll go through and build an image for just this laptop, disabling all the drivers and options that are totally unnecessary.  I’ll end up with a much leaner kernel that’s quick to compile.  But this image I have now could be given to anyone with a x64 system and it’d boot and work perfectly.

Thanks spender, pipacs and emese for their work on PaX/grsecurity.

Tim

UPDATE: A couple of updates to this post.  Firstly, the issue with sound was caused by CONFIG_GRKERNSEC_SYSFS_RESTRICT being set.  Disabling this, building again works.  I also found that I was getting slower performance, disabling uderef on the command line as well as slab sanitization has fixed this, giving me excellent performance again.  Beause they’re command line, I can re-enable them easily withouth having to recompile.  My full linux boot command is:

BOOT_IMAGE=/boot/vmlinuz-4.1.4-grsec root=/dev/sda1 ro reboot=w nouveau.runpm=1 nouveau.pstate=1 pax_nouderef pax_sanitize_slab=off pax_extra_latent_entropy

Finally, I sat down over the weekend and stripped out all the unneeded modules and settings.  Ubuntu by default sets a lot of debugging features, so all those are turned off now in the aims of squeezing a bit more performance out of the laptop.  Plus now my kernel image is ~25Mb, instead of ~250mb.

Requiem for a teatowel

Farewell manky chequered blue tea-towel
Your time has come
I went to dry my cup on you today
You’re no longer there
Replaced
By a new teatowel
A red one
A CLEAN red one

I remember once
When I first joined
I took you to the laundromat to get you cleaned
You looked the same after
Manky
Brown bits
Unpleasant

I bid you adieu
You were special
And smelly
I hope you’re at the Oxyplus factory
Being used
As the new definition of stubborn stains

Farewell.

 

I'm Dirty But You'll Use Me Anyway

I’m Dirty But You’ll Use Me Anyway

Death of a Note II

My Note II died yesterday.

It rebooted in the middle of nothing, then went to the startup / boot screen.  Quite normal. But it wouldn’t leave that screen.  So I pulled the battery.  Put it back.  Same thing, booted to the normal screen.

No worries I thought, I’ll just boot into recovery.  The recovery would start, then crash.  No menu, no options.  Didn’t look good.  I tried to flash stock using ODIN.  Kept failing with write errors, then it wouldn’t even boot up anymore.  A few more attempts, flashing the PIT file etc and now it’s deader than dead.

Galaxy Note 2

Galaxy Note 2

I guess the EMMC chip has gone.  Alas.  It was a good phone, if a little laggy.  Well, a lot laggy.  That’s the reason I’m just waiting on my Nexus 6 to arrive, which should be any day now.

I’ll never buy another Samsung again.