(Linux/Ubuntu) Frequent 802.11g/WPA2 connection drops with brcmsmac/BCM4313 driver

My Samsung laptop running Ubuntu 14.10 has been dropping its WiFi connection a lot. A cursory examination of the system log reveals that it usually happens once every 20-40 minutes, with a few suspicious cases where it dropped and re-established the connection after exactly 25 minutes:

$ grep "wlan0: associated" /var/log/syslog.1
Mar 17 20:45:48 gravitycat kernel: [101379.946156] wlan0: associated
Mar 17 21:10:48 gravitycat kernel: [102880.685412] wlan0: associated
Mar 17 21:33:32 gravitycat kernel: [104244.524634] wlan0: associated
Mar 17 21:43:03 gravitycat kernel: [104815.501396] wlan0: associated
Mar 17 22:08:03 gravitycat kernel: [106316.384463] wlan0: associated
Mar 17 22:11:23 gravitycat kernel: [   35.255877] wlan0: associated
Mar 17 22:46:55 gravitycat kernel: [ 2167.317562] wlan0: associated
[...]
Mar 17 23:29:08 gravitycat kernel: [ 4701.239455] wlan0: associated
Mar 17 23:54:08 gravitycat kernel: [ 6201.802957] wlan0: associated
[...]
Mar 18 00:00:26 gravitycat kernel: [ 6579.424244] wlan0: associated
Mar 18 00:25:26 gravitycat kernel: [ 8080.034656] wlan0: associated

The connection drops usually looked like this:

Mar 17 22:08:03 gravitycat NetworkManager[785]:  Connection disconnected (reason -4)

But were sometimes preceded by this:

Mar 18 01:09:43 gravitycat kernel: [10736.798220] wlan0: cannot understand ECSA IE operating class 33, disconnecting

It alternates between "class 33" and "class 12". After a tiny bit of digging, I found a suggestion here to try switching from Gnome's NetworkManager to WICD. At first, that seemed to make things better, but it started happening again after about an hour, with the same really frustrating behaviour (having to repeatedly queue the same videos on my Chromecast when it sees me "leave" due to the laptop dropping and reconnecting to the router).

Next port of call is to check the driver. Here's what lspci says is installed in my laptop:

$ lspci -vvnn | grep -A 9 Network 
03:00.0 Network controller [0280]: Broadcom Corporation BCM4313 802.11bgn Wireless Network Adapter [14e4:4727] (rev 01)
 Subsystem: Askey Computer Corp. Device [144f:7179]

For this device, Ubuntu defaults to the "brcmsmac" open source driver, which is always preferable to a closed-source proprietary driver, all other things being equal. This issue is so frustrating, however, that I'll take the closed-source "bcmwl" driver if it works, following these steps:

$ sudo apt-get install bcmwl-kernel-source
$ sudo modprobe -r brcmsmac bcma
$ sudo modprobe wl

After manually reconnecting with WICD, things seem stable and the ChromeCast hasn't bailed over the past day or so. It's a shame the open source driver isn't working well with the BCM4313 chipset. It'd be nice if there was a simple way to get started debugging/testing the driver – i.e. not having to download a massive tarball of the latest kernel source and spend hours tweaking with build scripts and the like. Unless things have changed in the last few years and now this is easy...?

OpenCV 2.4.10 and Macports

Every time I've needed to build OpenCV on my work Mac, it's been a bit of a struggle. I have Macports installed which helps make some things easier, but throws up problems of its own:
(using "mp-gcc49")

/Users/oisin/Documents/code/vision/opencv-2.4.10/modules/highgui/src/window_cocoa.mm: In function 'void cvDestroyAllWindows()':
/Users/oisin/Documents/code/vision/opencv-2.4.10/modules/highgui/src/window_cocoa.mm:202:23: error: expected ';' before 'in'
     for(NSString *key in list) {

(using "apple-gcc42")

/System/Library/Frameworks/QTKit.framework/Headers/QTMovieModernizer.h:123:46: error: expected ')' before '(' token
- (void)modernizeWithCompletionHandler:(void (^)(void))handler AVAILABLE_QTKIT_VERSION_7_7_3_AND_LATER;
                                          ^
/System/Library/Frameworks/QTKit.framework/Headers/QTMovieModernizer.h:123:46: error: expected identifier before '(' token

The problem appears to be Objective-C stuff in QTKit and in OpenCV's HighGUI code. This time, the solution was to revert to the system-default C/C++ compiler, Clang:

$ sudo port select gcc none
Warning: port definitions are more than two weeks old, consider updating them by running 'port selfupdate'.
Selecting 'none' for 'gcc' succeeded. 'none' is now active.

It's slightly counterintuitive that "port select gcc none" is how you go back to the system compiler with Macports, and it doesn't appear to be stated in the docs, so... well here it is!

A tiny line-editing trick for Bash (or any REPL using Readline/emacs bindings)

A lot of shells (e.g. Bash) and interactive console programs (e.g. mysql, sqlite, ghci, irb) provide a line-editing interface based on Emacs bindings, either via the Readline library or similar.

These key bindings mostly take the form of ESC-<key> which is either an uncomfortable stretch or (on a full-size keyboard) takes your left hand away from the home row and slows you down. Here's a handy tip though: pressing Ctrl+[ is equivalent to hitting escape, but allows you to keep your hands on the keyboard's home row. And if you're lucky, you can hold the Alt key instead (so you can jump backwards with Alt-b and delete the next word with Alt-d — this works on Gnome Terminal for me at least, except where the key is one of the menu accelerator characters (f, e, v, s, t or h). Seriously though, until today I was hitting ESC then the desired key...

Ralink RF3070 USB wireless support finally added to Linux

For a long time, these cheap Chinese USB wifi adapters didn't work in Linux. Believe it or not, I wiped my old laptop's Linux partition and tried FreeBSD and OpenBSD just to see if they supported it - but they didn't either.
So I reinstalled Linux and eventually produced a patch back in June to add support for the device, but for whatever reason it was ignored.

Now, though, one of the maintainers has implemented a proper patch and it looks like the RF3070 will soon work out-of-box in Linux.

Yay!

OpenCV build: "Could NOT find JNI" (Ubuntu)

While trying to build OpenCV so I could use the new Java bindings, I kept getting this somewhat confusing message:

Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY)

What does it mean - did I need to set environment variables for the AWT and JVM libraries? Surely they're both already installed. What does it even mean to say "Could NOT find JNI", since JNI is a part of Java which is installed?

Turns out I needed to manually set the JAVA_HOME environment variable:

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

Why Ubuntu doesn't automatically set this when you install the openjdk7 package, I have no idea.

Eye queue

This comment by "Cabochon1360" on one of David Mitchell's Youtube videos struck me today (far more than the video), and sums up my own feelings better than I could have.

In my teens, I was proud of my allegedly high IQ. But decades of failures and other humbling experiences have shown me it's just one personal characteristic, and not the most important one.

Another thing to bear in mind is that telling your kids they did well on a test because they're smart hinders their future performance. Perhaps counter-intuitive but it's been demonstrated in a number of studies (even when performed on adults, AFAIK).

What did I miss about modern Linux?

After running Windows alone for a couple of years, I had to install a Linux distro yesterday since the CUDA API in Windows is pretty ridiculous, flat out refusing to work with MinGW (the Windows port of gcc/g++ and pals).

Although I wasted a lot of time trying to get my dumb Medion PC (MS-7358 mainboard) to boot from two different USB keys, eventually a blank CD turned up and Ubuntu 12.04 landed on it. The install was quick, and this time (unlike on the old Acer laptop my kid uses) the Nvidia card was supported better, giving a usable display.

I needed to install some tools first before getting the CUDA API/SDK/drivers on Ubuntu:

The following NEW packages will be installed:
  build-essential dpkg-dev freeglut3-dev g++ g++-4.6 libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libdrm-dev
  libgl1-mesa-dev libglu1-mesa-dev libice-dev libkms1 libpthread-stubs0
  libpthread-stubs0-dev libsm-dev libstdc++6-4.6-dev libtimedate-perl
  libx11-dev libx11-doc libxau-dev libxcb1-dev libxdmcp-dev libxext-dev
  libxi-dev libxmu-dev libxmu-headers libxt-dev mesa-common-dev
  x11proto-core-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev
  xorg-sgml-doctools xtrans-dev
0 upgraded, 36 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.3 MB of archives.
...
Fetched 15.3 MB in 3s (3,835 kB/s)
...
Setting up build-essential (11.5ubuntu2) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
$

Now I really wish I'd prefixed this with the "time" command, because it took far less than a minute - maybe 15 to 30 seconds to install all of this... the C++ compiler, X11 libraries and a bunch of other stuff.

In contrast, it took a good 5 minutes just to install Visual Studio 2010 on my Windows 7 laptop (a 64-bit box with a better CPU than this desktop) and the library problems cost me a couple of hours of headbanging which was never resolved.

There are still serious driver problems in Linux, but the whole experience has improved so much since I started with a Red Hat install back in 1998.
Being able to install any number of programs and libraries rapidly, with a single command, is so much more relaxing than trying to hack things together in Windows. Try it!

Terrorist postcards

(from BBC news)

It has been dubbed a "snooper's charter" by civil liberties groups but the Home Office says new powers are needed to keep pace with how criminals and terrorists are using new technology.

Postcards are new technology now? It hardly seems like the most secure and timely way to communicate, though, does it?

Do they really think they can catch terrorists and other criminals by reading people's postcards - or are they just a pack of nosy bastards?

Simple Heuristics that Make Windows Dumb

C:\U\O\Documents\c\...\src> .\patch.exe --help
Access is denied.

C:\U\O\Documents\c\...\src> ren patch.exe piss.exe

C:\U\O\Documents\c\...\src> .\piss.exe --help
Usage: .\piss.exe [OPTION]... [ORIGFILE [PATCHFILE]]

...Fuck you very much, Windows.

Gluap: not-really-useful-yet PushGP in Lua

After abandoning this for a while, I decided to update my pet project Gluap, a PushGP-based genetic programming library for Lua. While I was at it, I opened a Github account and uploaded the project to a new repo. If you like, try it out, figure out why it's not performing well and improve it! :)

(Recursive) Confirmation bias

Ever heard of confirmation bias?

The worst thing? Being aware of the effect actually increases your susceptibility to it, rather than decreasing it. No need to thank me.

But does awareness of that fact reduce susceptibility? And if so, and you know it, does that itself---
--- ERROR: STACK OVERFLOW

Ever wanted to use the Unix "which" command on Windows?

The which Unix command takes a program name as an argument, and gives you the first matching directory on your path which contains that program. This is useful if, for example, you're trying to uninstall or upgrade software and are trying to figure out why you can still call the program after you thought it'd been deleted.

Unfortunately, there doesn't seem to be any kind of related command for DOS. Happily, a user named Mechanix2Go posted a hackish version which you can save as a batch file. I've made a couple of minor tweaks, to suppress file not found errors and fix the frankly idiotic treatment by DOS of the PATH variable as a set of tokens delimited by spaces. Some horrible hackery was taken from here to partially work around the problem. It will still fall over on weird paths with a ';' in them, and it'll echo double quotes.
Trying to express yourself in DOS batch language is like trying to compose a symphony using only two piano strings.

@echo off & setLocal EnableDelayedExpansion

if %1'==' goto :eof

for %%a in ("%path:;=";"%") do (
  dir /b %%a 2>&1 | findstr /i /b /c:"%1.exe" /c:"%1.bat" /c:"%1.com" > nul
  if not errorlevel 1 echo %1 is in %%~a && goto :eof
)

Reporting on reports about bad reports: SMS vs Web messaging

(source: BBC News)

Social messaging applications cost mobile network operators $13.9bn (£8.8bn) in lost SMS revenue last year, a report has claimed.
[...]
However, the study did not factor in the extra income networks received from mobile data costs because of increased internet usage resulting from social messaging.

...Then why are you even writing about it? Damnit BBC.

That's like saying "Clondalkin cost Dublin Bus $13.9bn in lost revenue on the 51B bus route last year, a report has claimed. The study did not factor in the extra income received from the 51C bus route which completely replaced the 51B route."
If they lost money on one service because everyone started using a different service that THEY ALSO PROVIDE, then you have to account for that before reporting that they lost money. Sure, sending one message via a phone's internet connection is probably much cheaper than sending an SMS, but without accounting for how much money they've been making from the increased internet usage - in general, not just for sending messages outside of SMS - it's meaningless for people to say things like this:

"I think it's a growing threat which is manageable through the right tariffs and the right costing," Mr Barford added.

Hardly a threat, if they're raking in money from all the data plans people are signing up to nowadays.

Google gets cabbage fronds both ways.

To follow up on an oldish post on the patchy quality of Google Translate, I tried the examples and found that they now translate very well.


Even doing a lengthy trip between very different languages produces quite good results, which is a good sign that the meaning behind text is being properly treated:

English "I like to eat cabbage fronds all day long." -> Japanese "私は長い一日キャベツの葉を食べるのが好き。"
Japanese  -> French "J'aime manger les feuilles de chou toute la journée."
French  -> Kannada "ನಾನು ಎಲೆಕೋಸು ಎಲ್ಲಾ ದಿನ ಎಲೆಗಳನ್ನು ತಿನ್ನಲು ಪ್ರೀತಿ."
Kannada  -> Hindi "मैं सारा दिन गोभी के पत्ते खाने से प्यार है."
Hindi  -> Irish "Is breá liom a ithe na duilleoga cabáiste an lá ar fad."
Irish  -> Korean "하루 종일 배추의 잎을 먹는 사랑 해요."
Korean  -> Polish "Uwielbiam jeść liście kapusty cały dzień."
Polish  -> Welsh "Rwyf wrth fy modd i fwyta'r dail bresych drwy'r dydd."
Welsh  -> English "I love to eat the leaves of cabbage all day."

So our 9-step purple monkey dishwasher train converted "I like to eat cabbage fronds all day long" into "I love to eat the leaves of cabbage all day".

There were a couple of languages (Arabic, Gujarati, Chinese) which mangled or lost some of the information (e.g. "cabbage fronds" into "cabbage" or "all day" into "every day"), but this is pretty impressive, and a tangible improvement in a relatively short time. Good work, lads!

Nonsmooth justice: the discontinuous mapping of crimes to sentences

From BBC news:

Jordan Blackshaw, 21, of Northwich, Cheshire, jailed for four years after admitting encouraging a riot on Facebook, which never happened

For comparison, if you were driving at 3 times the speed limit on the wrong side of the road, and you knocked someone down and left them in a braindead coma forever, the maximum sentence you could receive in the UK is 5 years. Until very recently it was 2 years.

This is somehow on a par with "encouraging a riot on Facebook, which never happened" which gained at least two people a 4 year sentence each.
Are you having a laugh? Because I'm not.

The situation is similar to that of a schoolteacher in the US who was sentenced to 200 years (!!) in prison without the possibility of parole, for possession of 20 images of child porn. Sure, collecting child porn is not a particularly nice thing to do, but again, there are murderers who will be sentenced and get out of jail before his sentence is finished. There are people who physically abuse, torture or actually kill children who could get out before him. Doesn't make much sense, does it?

Google Translate's voice input: phones only?

So apparently Google Translate takes voice input now - although it seems you need Chrome if you're not doing it on an Android or iPhone. The privacy issue of Chrome being able to record from your microphone without your consent is... interesting.

If you do it on a PC^*, it only allows you to give voice input in English. But if you do it via the iPhone/Android apps - even though they seem to use HTML5 anyway - you can give input in 15 different languages.

Google, hello? People use laptops and desktops as well - why are they restricted to English-only voice input? I couldn't find an answer to this, or a timeline for when the other languages will be available in the "vanilla" Google Translate.


^* Is there a good and concise term for "non-smartphone computer"?

Gradient-free data bars in Excel 2007?

Data bars are a quick and handy tool for making numbers more visually obvious in Excel. Mostly I just use them for progress bars on long ongoing writing tasks like my the... thes... I can't say the word for some reason, nevermind.

Anyway, the problem^* is that Excel 2007 forces the data bars to have gradient fade to white as they reach the top. This makes them look a bit stupid and indistinct, with the apparent rationalisation that it makes it easier to read the numbers in that cell - which is pretty weak TBH, just choose the right colours, bold your text and it won't be a problem.

A ridiculous workaround to get fake-solid bars is to set the bar colour to white and change the background colour of the cell to something with nice contrast that doesn't melt your eyes. Like neon green or piss yellow. Blue works for me...

Anyone know a better way of doing this (other than not using Excel, or upgrading to Office 2010, since then I end up with documents that don't work properly in the ancient versions of Powerpoint installed on DCU machines)?



^*Well, one of the problems. The others are that the default range values make almost no sense, and that values at the minimum of the range (or lower) still produces a bar, and that values over the top of the range don't entirely fill the bar. What...

Browser tab overload

After a couple of years of using the excellent Opera browser, I switched to Firefox after some crashes and extremely high memory usage. It turns out that Opera was doing pretty well given that I had some 200 tabs open, though... (yes, I find interesting articles quicker than I read them, and if mail myself the links or put them in a "to read" folder, well, they seem to not get read either).

Anyway, with addons and things, I've grown accustomed to Firefox now, and a recent-ish update which forces JS events from background events/timers to be processed at most once per second really helped with CPU usage.
However, it still struggles a bit with some 100 tabs open on this laptop, and actually crashes with an out-of-memory exception on my 32-bit Vista box from time to time. Not that it explains this or anything, it just vanishes and pops up a crash reporter which doesn't seem to explain the crash; I had to run Firefox under WinDbg to get a proper, source-line annotated stack trace.

One workaround for this is to just not be so lazy and to read articles immediately and follow links in a depth-first way, rather than the exponentially disastrous strategy of reading a Wikipedia article and clicking open 12 interesting links into background tabs, then doing the same for each of those tabs if I ever get that far.

Until I have the discipline to do that, though (i.e. never), I found a couple of Javascript bookmarklets are helpful.
So I stole a few bookmarklets to zap plugins, events and timers from here and taped them together into one "Zap all" snippet. Going through the roughly five million open and unread Lifehacker tabs and zapping off the Javascript machinery and other cruft managed to reduce the CPU usage to a respectable ~1% mostly. And pared the RAM usage from 1.5 gigs down to just under 1 gig on that Vista box.

Screw solving the underlying problem when you have bookmarklets!

Obsolescence update: A recent feature in Firefox (at least the Aurora 8.0x releases) allows you to set tabs to load lazily (i.e. a tab won't automatically load after opening Firefox, until you activate that tab)... this is perfect for messy users like myself who end up with more tabs than they can eat - it seems to work really well.

Closed windows from a crashed Firefox: Dig [out of] your own grave and save!

Well, I don't know what happened. The Windows box went down for a reboot due to the usual bundle of patches for Windows bugs. Somewhere along the line, Firefox managed to close the window with all my (way too many) tabs, and when I rebooted, it opened up on the Mozilla homepage. Burn.

No problem, going to about:sessionrestore displays a list of... empty. Oh.

Ok then. A quick look in the profile dir (%APPDATA%\Mozilla\Firefox\Profiles\*.default) shows that the sessionstore.js and backup are over 4 megs in size - that looks promising. So I copy them and quit Firefox.
Inside sessionstore.js is a gigantic JSON string. Inside that, I find a closed tab (the session restore tab which I had given up on), with apparently another full JSON string in the #sessionData field. After decoding that, I find an empty list of windows, and a list of _closedWindows including my main window with many many tabs. Yay, I can just swap them then.

Here's a little program in Lua which might help resurrect your tabs/windows if something similar happens to you. It uses the very fast dkjson library (the whole read/decode/futz/encode/write process for a 4.5mb file takes about 1.25 seconds on my run-of-the-mill desktop using LuaJIT).

In this case, there was an open window with one closed tab containing an about:sessionrestore form...

  json = require('dkjson')

  print('Reading file...')
  local session = assert(io.open('sessionstore.js', 'r'))
  local session_data = session:read('*all')
  print('Parsing ('..#session_data..' bytes)...')
  local obj, pos, err = json.decode(session_data, 1, json.null)
  if err then error(err) end
  print('Success! table size: '..#obj, pos, err)

  -- I closed the session restore tab since the list was empty.
  -- Turns out it still had the closed window data, so
  -- we can extract that. You may have to change this.
  local data = obj.windows[1]._closedTabs[1].state.
      entries[1].formdata['#sessionData']

  -- parse it, swap open/closed windows
  local data_p = json.decode(data, 1, json.null)
  print('Decoded, swapping closed and open windows')
  local old_windows = data_p.windows
  data_p.windows = data_p._closedWindows
  data_p._closedWindows = old_windows

  print('Encoding')
  local outf = assert(io.open('sessionstore-rec-fixed.js', 'w'))
  outf:write(json.encode(data_p))
  outf:close()

In this case, the open windows had suddenly become closed windows... (this just happened now, the hell Aurora?)

  json = require('dkjson')
  print('Reading file...')
  local session = assert(io.open('sessionstore.js', 'r'))
  local session_data = session:read('*all')
  print('Parsing ('..#session_data..' bytes)...')
  local obj, pos, err = json.decode(session_data, 1, json.null)
  if err then error(err) end

  print('Success! table size: '..#obj, pos, err)
  print('Decoded, swapping closed and open windows')
  local old_windows = obj.windows
  obj.windows = obj._closedWindows
  obj._closedWindows = old_windows

  print('Encoding')
  local outf = assert(io.open('sessionstore-rec-fixed.js', 'w'))
  outf:write(json.encode(obj))
  outf:close()

Rachota... oh no you di'n't!

Started using Rachota a couple of days ago to keep track of how much time I'm spending on my research, which currently consists of reading a big stats book.

Clicked into the "Analytics" tab just now and was informed of the following:

* You don't categorize your tasks enough. Try to assign some keyword to as many tasks as you can. This helps to track how much time your projects consume.
* You seem to spend too much time on private tasks or off the computer. Either minimize working on private stuff or don't leave your computer often without measuring such activity.
* It looks like your tasks are either very short or too long. Try to consolidate the short ones or divide the complex tasks. This helps to identify where your time really goes.
* You don't prioritize your tasks correctly. Use different priorities to distinguish important tasks from the low priority ones. This helps to keep focus on your real objectives.
* You don't use regular tasks properly. This might indicate you often work on a task that is not set as regular or there is a regular task that in fact occurs very rarely.
* It seems you leave your tasks open forever. Instead, close it once you are done with each task in reality to make your daily ToDo list shorter. Or don't you really finish anything?

...