Reading News Through Text Messages

Mobile text messages, logically, really just send data, just like what mobile providers call data, although at a slow rate. My mobile service plan has unlimited text messages but limited data. I was thinking I could use the text message data in place of “data data” for certain things.

I have set up a system, demonstrated on the right, that will let me read the news via text message, no mobile data required. As you will see, it is quite slow, but, this was never intended to be used in a serious manner anyway. The “M” requests the main menu, a number requests a menu item, “N” requests more of the content since only so much content can fit in a single message and I did not want to send an unstoppable flood of messages, and a “$” at the end of the message signifies that there is nothing left to send, that is, you can not send “N” to get more content.

To give a high-level overview, I will start logicalDiagram-small by briefly describing one communication cycle of the system. It is the most interesting part and the rest of the information in this post is just extra rambling for those really interested.

Logical flow of one communication cycle:

1. The user first sends a command to Twitter via text message. Commands are sent as one Twitter user and received by another. The same user that received the command will later be the one who sends the content back to the mobile phone. The user who sent the command will receive the content. Another way to put it, there is a Twitter account for the phone and another not for the phone.
2. Twitter will send out an e-mail notification in response to this command.
3. The e-mail server will send a push alert of the e-mail it received from Twitter and the Python IMAP client will receive this email and go out to the internet and gather content if required.
4. The Python e-mail client then sends the content to Twitter via the Twitter API and a Twitter direct message.
5. Twitter receives the direct message and sends out a text message notification.

The system is made up of two main components.

One, a custom Android application which sends and receives SMS messages. Originally this aspect was not thought to be required. I planned to send SMS messages through the email to text message and text message to email services that my mobile service provides. But I found that my service provider seems to rate limit these messages since messages would stop being delivered after sending a handful of them. It seems I could send unlimited messages from text message to e-mail but the rate limit kicked in on the way back (e-mail to text message). I could maybe send 20 before they stopped being delivered. I phoned and messaged my service provider about the situation, they did not know what the problem was and in some cases they did not even know the functionality existed.

Since using the text message to e-mail and e-mail to text message services was ruled out I decided to use use Twitter since it offers text message notifications for things like direct messages. I considered a few other services. I looked at Google, they seem to offer some sort of SMS service but it turns out it is not available for Canada. On a side note, I wonder why this is, I consider Canada to be more technologically advanced than some of the listed countries. I considered Facebook, but Facebook did not seem to reliably deliver text messages. I looked at a few free services but they seemed sketchy. I also looked at Skype.

Twitter is quite reliable but it comes with a few downsides. It adds a bunch of text of its own to every message, letting you know that it is a message from Twitter, who it is from and how to reply to it. As a result, a tweet taking up the entire 140 characters spans across two text messages. This creates two problems. One, it often results in messages arriving out of order, and two, the second message seems to take a longer period of time to arrive, as if again, my service provider is rate limiting me when two text messages are sent to me in a fast fashion. I considered only sending smaller messages so that Twitter could always deliver the message in one text message but I figured this would be slower since relatively more messages would be required since there is less Twitter boilerplate when the message spans two text messages when compared to sending two Twitter messages. This is because Twitter only tells you who the message is from for one Twitter message (even if it spans two text messages), thus a second text message (from the single Twitter message) would not include this boilerplate. If I sent two Twitter messages I’d get two text messages with two sets of boiler plate. Receiving boiler plate, and even messages out of order was not that big of a problem compared to the problems with sending a reply since my eyes can easily filter the boilerplate out, but replies required typing boilerplate. In order to reply to a Twitter direct message via text message you have to write “DM @twitterHandle theMessage.” This means that where I send the message “M” to get the main menu I would have had to type “DM @twitterHandle M” which would not be very usable at all. I looked into perhaps finding an SMS Android application with canned responses, macros or snippets, but my searching was not yielding fruitful results.

Thus, I decided to write my first small Android application. This application would parse incoming messages removing the Twitter boilerplate and make sure that messages were ordered properly. This meant that if the second message arrived first the application would wait until the first message arrived before displaying either. For consistency, if the message is multi-part it also waits for the whole message if the first message arrives first. The application also extracts the actual content of the message and displays the message hiding the rest of the information. The application also makes replying easier by automatically inserting the “DM @twitterHandle” aspect.

Two, a custom IMAP client. This is the first component of the system that I wrote. It uses the IDLE functionality to listen to a local self-hosted e-mail server running Dovecot in a virtual machine in order to receive push notifications when e-mail arrives. Once I made the move to Twitter I thought that this could be replaced with the Twitter API but it did not seem like receiving push notifications from Twitter was possible or easy at this point. There is evidence that they are going to make this possible in the future because it seems as if there is a private beta, but for the time being this did not seem like a plausible route. This is okay because the push e-mail notifications work really well anyway. Twitter delivers the email quickly and my IMAP client gets the notifications from the e-mail server nearly instantaneous.

The e-mail client receives notifications from Twitter and parses the e-mail to extract the message, extracting commands like “M” for main menu. It then gathers the content which it needs to send, it makes a few other notes for later, such as when a user enters “N” and requests more of the content and more messages. It then replies to Twitter via a direct message and the Twitter API. The e-mail client has to handle the fact that content which it wishes to send will often be longer than the Twitter maximum of 150 characters.

The gathering content aspect is rather quite neat. In the video I demonstrated it working with CBC Toronto but it is actually very flexible. Getting the CBC Toronto links was hard-coded to a degree but the gathering of the article would be nearly universal to any reasonably coded web page. This means that in the future, though unlikely to be honest (since this was just an experiment), I could add Reddit capability. Reddit would otherwise pose a problem since what it links to is not well defined at all. The way the article content is extracted is by analysing the structure of the web page and guessing which section is the main content. It does this by looking for the div or article tag which contains the most text and the least amount of other tags. This works because the actual content portion of a web page usually has relatively long sections of text without much HTML mark-up. There are a few other conditions to this but that is mostly how it works. Some tags are ignored such as the paragraph tag or the list item tag. This is because the paragraph tag is almost exclusively used for main content and a long list could have many list tags which would raise the tag count significantly. There were also problems like the situation where a div contains one word because that div tag would be 100% comprised of text and thus be very favourable in the text to tag ratio. This happened in situations like <div id=’companyLogo’>Widgets Inc.</div>. These problems were dealt with and I was able to successfully test on several web pages chosen at random.

This system because of its speed, and perhaps its complexity makes it not very practical. Aside from the intelligent content extractor, there is one potentially useful part of this experiment, it is making an Android SMS client that facilitates communication with Twitter like I have but takes it further. I could fairly easily group the SMS messages that are received from Twitter by Twitter user and allow the Android user to respond to whoever sent the message (rather than a fixed user as is the case with my application (very simple modification)). This would enable someone to chat on Twitter with their mobile despite not having a data connection. It would behave like a normal SMS application but using Twitter.

It was a fun experiment and proof of concept. I learned a lot, particularly about Android development since I had never done it before. I also figure that Twitter does not like me doing this much so I am unlikely to use this system much if at all. The primary reason I use mobile data is when I am waiting for something, like when at the dentist and waiting for my appointment. This system could be useful if I decided to reduce my data plan because I could occupy the time without using data and instead use text messages. The delay in the system would not be much of a bother since there would be no expectation or real care if it were fast or not. Though perhaps reading a pre-downloaded book would be more useful. Out of curiosity it would be amusing to transmit a small image via text messages.

xte Command, KDE Mouse Gestures and Moving Windows w/o Titlebars

The xte command can fake keyboard and mouse input. In Arch Linux it is part of a package named ‘xautomation’ and I installed it with ‘pacman -S xautomation’. The keyboard input that it fakes is very literal, if you wish to type a letter on the keyboard, you must simulate the keydown action and the keyup action. Using the xte command looks like:

$ xte 'command parameter' 'command parameter'

allowing for, as far as I can tell, as many command and parameter combinations as you would like. All the commands are specified in the man page along with how to refer to special keys on your keyboard, such as the meta key.

The reason I learned about this was because I have removed my title bar from Firefox which previously provided a space to click on in order to initiate moving a window. It is possible to press alt while left clicking to move a window, but, I don’t always have my hands near the keyboard to do so. Since I was looking at how to initiate moving a window in KDE I cam up with this solution:

$ xte 'keydown Alt_L' 'keydown F3' 'keyup Alt_L' 'keyup F3' \
'usleep 40000' 'keydown m' 'keyup m' 'usleep 50000' 'keydown Return' 'keyup Return'

which simulated one of the methods that can be used to initiate moving a window in KDE. The usleep provides time between command input which allows dialogs to open before the next command is entered. Once xte command was run I was able to simply move my mouse around in order to place the window where I would like.

This was a satisfactory solution but I felt I had a better one:

$ xte 'keydown Alt_L' 'keydown Alt_L' 'sleep 1' 'keyup Alt_L'

This one is much simpler and doesn’t have to wait for dialogs to appear. It simulates what I mentioned above about being able to press alt then drag with a left mouse click. With this command, an alt keypress is simulated for a duration of one second which allows me enough time to left click to engage the window move. Once the window move is engaged the alt key does not have to remain pressed.

But until now these commands have been run from the terminal and not of much use, I need a way to engage the commands relatively arbitrarily. This can be done with KDE mouse gestures located inside System Settings > Shortcuts and Gestures. Do note, that mouse gestures needs to be enabled, this can be done from this settings dialog too. Right clicking in the white area under ‘Configure Input Actions settings’ where the default gestures are defined allows you to create a new gesture. Right click and select New > Mouse Gesture Action > Command/Url. The trigger tab is where you setup the gesture trigger, the pattern that you will move your mouse in to trigger the action. You do this by clicking edit then performing the action with a left click in the area provided. Though the gesture is recorded with a left click by default the gesture for me had to be performed with a centre or scroll wheel click. In the actions tab is where you input your xte command just as it would be entered in the terminal, not requiring any extra quotes or anything of that sort.

Now that I have done all this, in any window I can gesture my mouse to initiate a window move by simulating a press of the alt key allowing me to move windows with only my mouse, windows that could not otherwise be moved because of not having a title bar or other empty places to click.

Update: Sort of funny in a way, about half an hour after posting this, I found a Firefox specific solution which is orders of magnitude easier than the above. The Oxygen KDE add-on for Firefox is nearly a perfect solution for enabling Firefox ‘dragability’ after removing the title bar. The add-on creates a toolbar icon that you click on which will engage the appropriate dragging mechanisms. After enabling the add-on, the toolbar item can be added like other Firefox tool bar items by right clicking the toolbar and choosing customize. But, this was still a good experience and is still useful for other windows that I have removed the title bar from, like my console windows.

Create .accdb/.xlsx (Access/Excel) or SQL Server Connection for Crystal Reports

For Access/Excel, when creating a new connection for a Microsoft Access/Excel 2007 (+?) database you need to select “OLE DB (ADO)” as opposed to “Access/Excel (DAO)” which is for 2003 or perhaps earlier. After, you choose “Microsoft Office 12.0 Access Database Engine OLE DB Provider”. I found this non-intuitive so I’m making note of it here. After selecting that, you will now have a menu where you can choose “Office Database Type” where you can select “Access” or “Excel”. On this same menu, in the “Data Source” field you locate the actual data file. The rest is fairly intuitive.

For SQL Server, the steps are pretty similar, you select “OLE DB (ADO)” and then “Microsoft OLE DB Provider for SQL Server”. You then need to fill out the connection information. After filling in the Server, User ID and the Password you will be able to select a database. The login credentials can be skipped if you choose Integrated Security which uses your Windows credentials.

Arch Linux Crontab Not Running

My cron was not running, it appears that the cron service is not enabled by default on a new Arch Linux install. I enabled it with:

# systemctl start cronie
# systemctl start cronie.service

Really easy é typing in Ubuntu

I do have my compose key setup so that typing an é can be a matter of pressing + e + ‘ but é seems to come up enough with French that having an even easier way to type it would be handy. So, I did what I have shown below. Now I can type an é by pressing + e + e (one after another). It is the same amount of keystrokes, but considerably easier. After adding the following content to the following files I needed to log out and back in again.

~/.XCompose

include "%L"   # import the default Compose file for your locale
<Multi_key> <e> <e>     : "é"

~/.profile

export GTK_IM_MODULE="xim"

The Aquariums of Pyongyang – Ten Years in the North Korean Gulag [book notes]

By: Kang Chol-Hwan and Pierre Rigoulot
Translated by: Yair Reiner

This was a really interesting book, I enjoyed it very much (not in the way one enjoys a trip to an amusement park, obviously). It of course revealed that the conditions in North Korea are terrible. It gave an account of the situation that was described as relatively good despite still being terrible. I often assume that I know what terrible is, but regularly as I search through media from terrible places and situations, my understanding of terrible is surpassed. It has got to be hard for me to imagine, like fathoming a 4th dimension or what it’s like to have the senses of a bat, the experience is simply not available for me to understand, but the account offered in this book does bring me closer to understanding. The account of the prison camps was particularly interesting because of the author’s young age and innocence. It’s amazing to think that at certain points he thought that what was happening was normal. As the author himself sometimes describes, it is like hearing about “another universe” where hot freezes and cold burns.

When I mention the author, I mean Kang Chol-Hwan

PREFACE

  • it wasn’t until after ww2 that the atrocities committed by the Nazis in relation to the Jewish people came till light. Could this happen with North Korea?

Introduction – North Korea – The World’s Last Stalinist Regime

  • history of how North Korea came to exist
  • female human trafficking into China
  • stunted growth of children

Chapter 1 – A Happy Childhood in Pyongyang

  • in the 1960s the North and South were on par with regards to economic development
  • Renounced but often used in private, regarding the age of a person, year one starts when conceived, and incremented on each January first http://en.wikipedia.org/wiki/East_Asian_age_reckoning though according to http://www.reddit.com/r/todayilearned/comments/14jesr/til_that_in_east_asian_countries_like_korea_you/c7dpedi this does not have to do with time spent in the womb
  • In school there were “self-criticism” sessions, where children would denounce themselves for not living up to the standards of the Great Leader. Apparently they weren’t as bad as they sound, that they were supposed to serve as encouragement to do better.
  • Pupil’s Red Army
    • fake guns
    • marching
    • fun
  • author seems to have had a fun childhood
  • seemed exceptionally well off in comparison to others though
  • other places, and big cities seemed to be doing okay

Chapter 2 – Money and the Revolutionary Can Get Along

  • “According to Confucian tradition, which continues to hold sway in present-day Korea, a married woman belongs to her husband’s family and remains so, irrespective of divorce or separation. If she tries to return to her parents’ home, she will most likely be turned away.”
  • describes how the author ended up wealthy (grandfather was rich from enterprises in Japan)

Chapter 3 – Next Year in Pyongyang!

  • Discussed family’s move from Japan to North Korea, quickly realized it was a horrible decision

Chapter 4 – In a Concentration Camp at Age 9

  • grandfather taken by security forces one day at work, security forces lied to family about whereabouts, eventually came to take the family
  • security forces searched the house, stole plenty of their belongings for themselves, gave others to the government
  • interesting, much of this is told from the author’s perspective who was at the time, 9 years old, which makes the story quite interesting because of his innocence, he wasn’t even aware of what was going on, initially he thought that it might even be fun going off into the country, but he was concerned about his pet fish.

Chapter 5 – Work Group Number 10

  • there exists “school” in the camps for prisoner children that ran half the day, the other half would be labour
  • guard families and guards were well off, lived separate from the prisoners
  • described the family’s introduction to the camp
  • devices used to prevent escape, including animal traps, depended on mountains etc for natural barriers

Chapter 6 – The Wild Boar: A Teacher Armed and Ready to Strike

  • ways teachers would address students:
    • “Hey, you, in the back of the room!”
    • “Hey, you, the idiot in the third row!”
    • “Hey, you, son of a whore.”
  • One of the teachers described as “an adept technician of suffering, always searching for a way to maximize pain.”
  • some punishment came in the form of digging ditches, filling them in, then digging more ditches and filling them in
  • described school and some of the child labour
  • Described an interesting situation regarding the effects of not distinguishing individuals of a team from the team, judging the output of the team and punishing the entire team if the output didn’t meet standards. This had the effect of creating a system of self-surveillance, because no team members wanted to be part of a team where certain members were lagging behind. Though this could result in greater output, perhaps, it could create animosity between the team members, at the very least, a cold working environment.

Chapter 7 – Death of a Black Champion

  • The author was at camp Yodok, he describes this as one of the nicer camps
  • prisoners were separated between those who were “redeemable” and those who were “irredeemable.” Irredeemables committed, or were associated with people who at committed more serious crimes, like being a capitalist or a Christian. Redeemables were subjected to being re-educated, irredeemables were not, they did not have to learn how to praise the leader or communism because they were never going to leave the camp. 70% of Yodok prisoners were irredeemables, they suffered much worse that the redeemables which is a suffering that is difficult to imagine

Chapter 8 – Corn, Roaches, and Snake Brandy

  • certain jobs in the camp were better than others, a few were almost even relatively tolerable
  • prisoners often found themselves in the tough spot of needed to break the rules to survive and needing to obey to avoid punishment, or even, survive
  • brutalities of the camp were described
  • malnourishment seems to be the toughest punishment of them all

Chapter 9 – Death at Yodok

  • deaths caused by the winter not common
  • special grave area was dug up for room to plant corn
  • when working in fields, a perk was the possibility of being able to eat insects or frogs as you found them during work

Chapter 10 – The Much-Coveted Rabbits

  • camp snitches existed and were a valuable part of maintaining constant surveillance. The snitches were despised but were also considered victims by the inmates
  • for a while, the author had the privilege of raising rabbits. The rabbits were treated very well, their furs were needed to military clothing. The meat would go to the guards but the inmates managed to keep some
  • some inmates came to make rat a large part of their diet. Some inmates were inventive in capturing and raising rats. The authored ended up feeling very privileged when he was able to eat rat and stopped seeing them as something gross
  • the experience has corrupted the authors world view, when he now sees vast large mountainous landscapes, he does not see beauty but is reminded of the camp’s natural barriers

Chapter 11 – Madness Stalks the Prisoners

  • basically no health services
  • sometimes you could have small procedure by medical staff that is usually reserved for guards, but then often left alone to deal with infections
  • prisoners often would go mad
  • medical staff mostly there to determine who might get minor leniency for ailments

Chapter 12 – Biweekly Criticism and Self-Criticism

  • propaganda sessions for adults
  • minor reprieve on very special occasions including specialities like a small piece of candy or watching a propaganda film
  • the author says that before the camp he loved their “leader” but says the camp “cured” him of his faith, that is, his thoughts of the leader now are all negative

Chapter 13 – Public Executions and Postmortem Stonings

  • graphic public executions, from shooting or hanging, prisoners forced to stone after death
  • many prisoners would be relatively numb after the first few executions
  • most executions were from escape attempts

Chapter 14 – Love at Yodak

  • sex forbidden
  • men caught, guard or prisoner would be sent to the sweatbox, a place where people often died
  • women would have to tell the village about the ordeal in great detail, as if they were telling an erotic tale
    • additionally, other punishments were said to have happened such as being penetrated with garden tool spades or having breasts cut off
  • sex was forbidden because these impure souls should not breed

Chapter 15 – Sojourn in the Mountain

  • the brutality of the labour in part made the prison experience more tolerable in a way, because it could leave people so tired that they don’t even have time or the energy to dwell on the situation. Thus, rest time could be some of the worst times, thinking about the outside world or the terrible things that have or could happen

Chapter 16 – Ten Years in the Camp: Thank You, Kim Il-Sung

  • eventually it was time for the author and his family to leave. He was mixed on the ordeal, the camp is where he had grown up. Even though he was leaving a horrible place, he was still about to experience a drastic change and there was lots of unknown, it was even difficult to believe
  • In North Korea it is very difficult, if not impossible to move up the social ladder. Children of farm peasants were condemned to be farm peasants. Non-peasants marrying peasants were forced to become peasants, since the peasants would contaminate the non-peasants.

Chapter 17 – The North Korean Paradise

  • bribes are common in North Korea
  • “The regime that never tires of denouncing capitalism has birthed a society where money is king…”
  • fighting in the streets was common, people seemed to form gangs of sorts. Once the author was attacked in the street and managed to alert a police person who he had previous positive dealings with. The person who had committed the violence was sent to jail. The police allowed the author to enter the jail cell and beat the criminal, but not kill because that would cause trouble for the police. The author initially took up the offer but was soon horrified by what he was doing.
  • Extensive underground network
  • mail to other countries censored
  • during famine, peasants began to grow crows on abandoned land as if it were their own private land. This is very against the regime but they were forced to tolerate to help limit massive starvation
  • lots of North Korean families get by on money sent to them from family in Japan

Chapter 18 – The Camp Threatens Again

  • the another and his friends enjoyed listening to South Korean radio which is extremely forbidden. A friend snitched on him and he was put under surveillance. Things started getting scary, he thought he could be sent back to the camps, this time hard labour, and thus he and another friend decided it was time to try to escape North Korea all together

Chapter 19 – Escape to China

  • the escape actually seemed relatively easy, the primary tool was bribes, the guards were paid off and guides in China helped them along.
  • Unfortunately making it to the South Korean consulate did not set them home free
  • apparently escape is getting easier and easier with time
  • though the author had money and connections, most don’t

Chapter 20 – Small-Time Prostitution and Big-Time Smuggling in Dalian

  • the two, now in China, were taken in by a Korean speaking female pimp for some time. They made good relations with the prostitution ring and eventually secured passage to South Korea.
  • Getting out of China seemed even more difficult than getting out of North Korea

Chapter 21 – Arrival in South Korea

  • more details on leaving China
  • was smuggled out on a cargo ship, once in international waters the ship captain notified South Korea about the two, a S. Korean military ship came and took them
  • the two amazed by how polite everyone was
  • the two were interrogated for months (in a nice way) to ensure they were being truthful
  • their arrival in the south was big news, though the news was sceptical of the conditions of the north

Chapter 22 – Adapting to a Capitalist World

  • it took about 2 years before the author could live freely, without chaperones etc. The chaperones were described as more helpful than a burden. It was tough navigating the new society after being in “…Hermit kingdom, as North Korea is sometimes called.”
  • A rich businessman saw the author’s story and decided to contribute money for him to go to school/re-integrate etc.
  • A bank also provided a scholarship
  • there was some security risk from agents from the north
  • his escape probably caused problems for certain people left in the north
  • often the Chinese will send back people who escape to endure certain terrible consequences
  • several groups exist in the south to help people from the north

Epilogue – Pursuing Aid for North Korea

  • there is human trade of women across the North Korean boarder
  • food aid is tricky as it can be misused or help aid the military
  • but often the military is just normal people, people often go to the military in order to get food or advance in society
  • focus should be on dismantling the regime or helping escapees not on food aid
  • food aid helps continue the regime

cryttab luks cryptswap1

I was setting up a second hard drive with full disk encryption and I wanted the disk to mount automatically. I set everything up and manually mounted the encrypted disk. Things worked fine. I edited /etc/crypttab to map the encrypted drive and /etc/fstab to mount it so that the process would occur automatically at boot. However I ran into a problem, upon restart I received:

The disk drive for /mnt/point/ is not ready yet or not present.
Continue to wait, or Press S to skip Mounting or M for manual recovery

My encrypted swap space and my new encrypted drive were not mounting. Quickly I realized my silly error, they key I was using to decrypt the drive, was located in my encrypted home folder, exactly where I wanted it to be. I tried copying the key to /root (and updating crypttab) then restarted. Everything mounted properly. However, this is not a suitable solution because my key is now being stored unencrypted which would allow anyone with physical possession of my computer to get a hold of it. They key is going to have to remain in my encrypted home directory, but I’m going to have to set up the mapping and mounting to occur after login, that is, after my home folder is decrypted.

A lesson from this is that it seems that a single entry in /etc/crypttab that contains an error (in this case not being able to read a key file) can cause errors with the other entries. I observe this since I had not modified the swap entry in /etc/crypttab but it produced an error along with my new encrypted drive.

curl c++

Here is a code snippet I found very useful and wanted to save

// from http://www.cplusplus.com/forum/unices/45878/

#include <curl/curl.h>
#include <fstream>
#include <sstream>
#include <iostream>

// callback function writes data to a std::ostream
static size_t data_write(void* buf, size_t size, size_t nmemb, void* userp)
{
	if(userp)
	{
		std::ostream& os = *static_cast<std::ostream*>(userp);
		std::streamsize len = size * nmemb;
		if(os.write(static_cast<char*>(buf), len))
			return len;
	}

	return 0;
}

/**
 * timeout is in seconds
 **/
CURLcode curl_read(const std::string& url, std::ostream& os, long timeout = 30)
{
	CURLcode code(CURLE_FAILED_INIT);
	CURL* curl = curl_easy_init();

	if(curl)
	{
		if(CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &data_write))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_FILE, &os))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout))
		&& CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_URL, url.c_str())))
		{
			code = curl_easy_perform(curl);
		}
		curl_easy_cleanup(curl);
	}
	return code;
}

int main()
{
	curl_global_init(CURL_GLOBAL_ALL);

	std::ofstream ofs("output.html");
	if(CURLE_OK == curl_read("http://torypages.com", ofs))
	{
		// Web page successfully written to file
	}

	std::ostringstream oss;
	if(CURLE_OK == curl_read("http://torypages.com", oss))
	{
		// Web page successfully written to string
		std::string html = oss.str();
	}

	if(CURLE_OK == curl_read("http://torypages.com", std::cout))
	{
		// Web page successfully written to standard output (console?)
	}

	curl_global_cleanup();
}

SSH keys still asking for password

After attempting to setup an SSH key-pair for passwordless logins, I was still being asked for a password. It turns out that the remote system was taking issue with the fact that the remote user’s home directory (as defined in /etc/passwd) was not in the standard /home location. After changing the home location (via /etc/passwd) they key-pair worked. I’m sure there is a way to allow for the use of a non-standard home location and SSH key-pairs, but, I worked around the issue instead.

Synthesis: Legal Reading, Reasoning and Writing in Canada [book notes]

  • 2nd Edition
  • Margaret E. McCallum
  • Deborah A. Schmedemann
  • Christina L. Kunz

Chapter 1

  • The Lawyer’s Roles
  • Lawyers are advisors/advocates for their clients
  • as an advocate the lawyer will, in court, defend their clients based on events of the past hoping to achieve the most fair outcome. The lawyer will also look forward in seeking the most favourable and fair outcomes for their clients
  • Lawyers take life situations and frame them within legal constructs
  • Process: fact investigation, research, reading the law, reasoning about the application of the law to the clients situation, writing out or orally presenting the analysis
  • The skills of a lawyer are broad and encompass the skills of many other professions, ex, an “engineer’s precision”
  • Lawyers are also servants to the public
  • Aids in the implementation of laws
  • Lawyers actions in court are continuously developing Common Law
  • Canadian Bar Association’s Code of Professional Conduct
  • The Legal System
  • Local government are the creations of provincial/territorial governments and can only which they are allowed to do by their parent body
  • The federal and provincial/territorial levels have three branches: legislature, judiciary, executive
  • legislature makes the law
  • judiciary: interperts, applies
  • executive implements the law
  • there is overlap between the three branches
  • Aboriginals can form their own governments of sorts. Their powers of self government are a lot more broad than the powers that provincial or local governments receive.

Chapter 2

  • Rules of law exist in: cases, statues, regulations and court rules.
  • Rules of law need to be predicatable
  • Legal rules can be stated in an if/then format. “IF the required factual conditions exist,” “THEN the specific legal consequences follow.”
  • factual conditions or legal consequence
  • And If/Then/Unless statement can modify the If/Then situation to explain situations where the rule does not apply
  • One could also say If and not some stuff Then…
  • if a list uses the word “and” all elements must be fulfileld aka “conjunctive rule”
  • if a list uses the word “or” at least one of the elements must be fulfilled aka “disjunctive rule”
  • ex
    1. the person at rice, and
    2. the person ate
      • a) an apple, or
      • b) a pear, and
    3. the person drank water
  • in the above example the person must have at rice, and either a banana or an apple, and drank water for the condition to occur.
  • “And aggregate rule required you to determine whether enough of the suggested factors have been met to justify applying the legal consequences.”
  • that is, there will be a list of conditions that could apply, and “some” number of those conditions could be enough to trigger the legal rule
  • “A balancing rule requires you to balance factors favouring either outcome in order to determine whether the legal consequences will apply.”
  • this could mean that there will a section that says something like, “if the harm of condition a outweighs the harm of condition b
  • Aggregate and balancing rules include ambiguity which can be good or bad. It allows for flexibility but that flexibility brings along unpredictability with it.
  • Plural consequences: there are multiple consequences, they are conjunctive, they use the word and, as a consequence someone might have to do this AND that.
  • Alternative consequences: a consequence can be chosen from a set, the word “or” is used and is disjunctive. The person does this OR that.
  • A consequence that is an ultimate practical condequence is one which directly states what the condequence is, for example, a fine
  • Intermediate legal condequences might say that the consequence is an offence. One would then have to do further research to determine what is is meant by offence.
  • Legal rules can be represented in charts, lists, paragraphs… one should choose the method which best depicts the rule

Continue Reading…

Problems Un-mounting

I had some problems un-mounting an sshfs mount. I did a “umount -fl” (force, lazy) and tried re-mounting, but I continued to have issues. I fixed the issue with a combination of un-mounting and clearing the mount entry from /etc/mtab which entailed simply opening /etc/mtab in a text editor and removing the relevant line entry.

Toronto Snow Day 2013

It’s like Toronto has never seen snow before. Here is my take on the day.

DSC03818

DSC03735

DSC03742

DSC03751

DSC03755

DSC03796

DSC03690

DSC03700

DSC03734

DSC03718

DSC03714

 

 

POCO Install/Compile Error Fix

I was in the process of installing the POCO Libraries on Ubuntu 12.10 for C++ and while compiling I received this error: ODBC.make:49: *** No ODBC library found. Please install unixODBC or iODBC or specify ODBCLIBDIR and try again. I fixed it with:

apt-get install unixodbc-dev libmysqlclient-dev

Remove Youtube Comments

I recently wrote a Python 3.2 script which will remove your Youtube comments automatically for you. It works reasonably well, but it did require me to run it a few times in order to get everything. It uses the Google/Youtube API to accomplish the deletion, but some “hackery” was needed in order to find the comments which need to be deleted. This “hackery” entails scraping Google web search results, something which is not entirely kosher. As such, you may find that Google will temporarily block your requests being made from this script, as they did to me at one point. This is fine because the proper method of access Google search remained open and I was in no rush since this is just for personal use, so I just waited it out. I have since inserted a delay command in the script to make the requests look a little more human like, but of course that does cause the script to take longer. I have not included the Google app id I used to run this script, you will have to create your own at https://code.google.com/apis/console/. You will also have to update the script to use your own Youtube username. The places in the code that require your attention most have exclamation marks in the comments. I used this script on Ubuntu 12.10 with curl installed. The script has been formatted for this blog, if an error arises it may be around a long string that was made to occur on multiple lines after the fact.

import lxml.html
import urllib.request
import urllib.parse
import json
import webbrowser
import time
import sys
import subprocess

##### get authenticated #####
client_id = "847147874147.apps.googleusercontent.com" ## need to get your own !!!!!!!!!
client_secret = "92dGuyKje6DncGo8EOe8GYKJ" ## need to get your own !!!!!!!!!
apiKey = "MIzjSyQmSZ3U18vmD1esxF-gQU2ZmXKdf5FQoCW" ## need to get your own !!!!!!!!!
url = "https://accounts.google.com/o/oauth2/auth?client_id=" +\
      "{0}&response_type=code&scope=https://www.googleapis.com/auth/youtube".format(client_id) +\
      "&redirect_uri=urn:ietf:wg:oauth:2.0:oob"
webbrowser.open(url)
time.sleep(2) # required for clean output display
code = input('paste in your code: ')

values = {"code" : code,
          "client_id" : client_id,
          "client_secret" : client_secret, 
          "grant_type" : "authorization_code",
          "redirect_uri" : "urn:ietf:wg:oauth:2.0:oob"}

url = "https://accounts.google.com/o/oauth2/token"
data = urllib.parse.urlencode(values).encode('utf-8')
request = urllib.request.Request(url, data=data,\
                           headers={"Host": "accounts.google.com",\ 
                           "Content-Type": "application/x-www-form-urlencoded"})
try:
    page = urllib.request.urlopen(request)
except:
    print("Was not able to authenticate, quitting.")
    sys.exit(0)

authJson = json.loads(page.read().decode('utf-8'))
access_token = authJson['access_token']
refresh_token = authJson['refresh_token'] #unused

if not access_token:
    print ("was not able to authenticate, quitting.")
    sys.exit(0)

########## get links from google #########
username = "INSERT_YOUR_USERNAME_HERE!!!!!!!!!!!!!" ## !!!!!!!!!!!!!!!!!!
query = "site:youtube.com%2Fall_comments+-site%3Am.youtube.com%20" + username
start = 0 #page count, goes by 10s
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.20' +\
       ' (KHTML, like Gecko) Ubuntu/12.04 Chromium/18.0.1025.168 Chrome/18.0.1025.168 Safari/535.19'
linkArray = []

while True:
    url = "https://www.google.ca/search?q=" + query + "&start=" + str(start)
    page = urllib.request.urlopen(urllib.request.Request(url, headers={'User-Agent': user_agent}))
    pageStr = page.read().decode('utf-8')
    html = lxml.html.fromstring(pageStr)
    result = html.xpath('//a[@class="l"]')
        
    if not result:
        break;
    else:
        start = start + 10
        time.sleep(10) # be nice to google

    for i in result:
        link = i.get('href')
        linkArray.append(link)

###### get comment ids ######
commentLinkArray = []
for i in linkArray:
    try:
        videoId = i[38:]
        url = "http://gdata.youtube.com/feeds/api/videos/{0}/comments?alt=json".format(videoId)
        commentJson = json.loads(urllib.request.urlopen(url).read().decode('utf-8'))
        
        for x in commentJson['feed']['entry']:
            if x['author'][0]['name']['$t'] == username:
                print ("sssss")
                commentLinkArray.append(x['link'][2]['href'])
    except:
        pass

##### Perform the deletion ######            
for i in commentLinkArray:
    print ("Deleting:", i)
    call = ["/usr/bin/curl", "-X", "DELETE", "{0}".format(i),
                             "-H", "Authorization: Bearer {0}".format(access_token), 
                             "-H", "X-GData-Key: key={0}".format(apiKey), 
                             "-H", "GData-Version: 2", 
                             "-H", "Content-Type: application/atom+xml",
                             "-H", "Host: gdata.youtube.com", 
                             "-o", "/dev/null"]     ## not sure if this did anything
    subprocess.call(call)

Facebook for Android – Unacceptable Permissions

If I needed a straw to break the camels back to not use the Android Facebook app, I now have one. I had already decided against using the application, opting for the web application instead, but this confirms my decision.

I do not know when it was introduced, but it has been brought to my attention that the Android Facebook application can take pictures and videos at any time it wants without user confirmation. This means that as your phone sits on the counter while you are getting out of the shower, it can take a photo of you, when you are using your device and touching your face, it can take a photo of you, when using your phone in the washroom, it can take a photo of you, when changing for bed, it can take a photo of you. The text on the permission page is clear:

TAKE PICTURES AND VIDEOS
Allows the app to take pictures and videos with the camera. This permission allows the app to use the camera at any time without your confirmation.

It was pointed out to me by a friend that this could be because it allows the Facebook app to take photos, with the user’s permission, but without having to ask for additional permission through the Android system. That is, only Facebook would have to ask for your permission, not Facebook and Android (Google). I do not care. This creates far too many “what if” situations. It is not hard to have to confirm the use of the camera in the process of taking a photo or video. (Just FYI, I am not entirely familiar with this confirmation since I have refused to use the Android Facebook application for some time now.)

Lets say that Facebook has good intentions with this feature, perhaps it is an honest move in good faith, maybe it is. But what about the other instances where good faith actions have turned out to be bad. What about the instances of school laptop programs, where the students lease, or are given laptops by their education facility and are then spied on by IT admin while they are in their bedrooms (a very private space). What if there is a rogue Facebook employee who decides to insert some code which will satisfy his/her paedophilia addiction? It would not be the first time internal security breaches have occurred with services that hold enormous amounts of personal information, it happened when a Google employee decided to access data on a teen’s Google account and I am sure it happens regularly all over the place. Internal threats are very real. When I used to work retail, I was told that a large amount of product loss, if not the number one reason for product loss is through internal dishonesty. It does not even have to be a Facebook employee, it could be an attacker who gains access to Facebook’s internal network. Or maybe it will be a government agency who obligates Facebook to create a spy feature just for them.

At the end of the day, it is a balance of convenience and security and of course everyone is free to make their own evaluation based on their philosophy where the centre of balance lies. But for me, because of what has been discussed and because of the many other Android Facebook powers, installing the Android Facebook application would negatively tip the scales for me. I recommend that no one use this application, especially when the web application is good and available.

Facebook Clean/Delete/Remove Posts

As evident on this blog I am continuously looking for the best way to remove old posts from Facebook. Facebook has changed their layout, or iMacros (an add-on for Firefox) has changed, or something, but the most recent round of deletions I performed went quite easily. I excursively used iMacros with two files:

commands.iim

TAG POS=1 TYPE=I ATTR=CLASS:mrs<SP>-cx-PRIVATE-fbActivityLogOptionsButton__pencilIcon<SP>customimg<SP>img<SP>sp_95iqka<SP>sx_cd7c2c
TAG POS=1 TYPE=I ATTR=CLASS:mrs<SP>-cx-PRIVATE-fbActivityLogOptionsButton__pencilIcon<SP>customimg<SP>img<SP>sp_95iqka<SP>sx_cd7c2c
TAG POS=1 TYPE=SPAN ATTR=TXT:Delete...
TAG POS=1 TYPE=INPUT:BUTTON ATTR=NAME:ok&&VALUE:Delete
REFRESH

del.js

while (1==1) {
	iimPlay('commands')
}

commands.iim is where the main actions happens. The first command is repeated twice, it shouldn’t have to be, but for whatever reason I had a better experience when repeating it. The refresh at the end isn’t entirely necessary, but I found it made the process significantly more stable, that is, it went from failing from time to time, to completing the rest of my goal without an issue.

del.js is the file that you will actually select to play from the iMacros console. All it does is call commands.iim for eternity. This is needed so that if any errors occur, like if the page takes longer to load than it usually does, the process will simply restart itself instead of stopping. You must have the browser pointed at one of the Facebook activity log URLs such as https://www.facebook.com/fbusername/allactivity?privacy_source=activity_log&log_filter=cluster_11.

Eclipse Juno Ubuntu 12.10 appmenu / globalmenu

Getting the global menu or the app menu to work with Eclipse Juno under Ubuntu 12.10 can be accomplished by following the instructions here. However, I did not have the file “/usr/lib/gtk-2.0/2.10.0/menuproxies/libappmenu.so” I had to perform the instructions on “/usr/lib/x86_64-linux-gnu/gtk-2.0/2.10.0/menuproxies/libappmenu.so”. libappmenu.so actually existed for me in a 3 places, the one mentioned as well as in “/usr/lib/indicators3/7/” and “/usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/menuproxies/”. At first glance of the contents of libappmenu.so it looks like one that you wouldn’t be able to edit and I was skeptical of the instructions, but the instructions did indeed work. Of course you do not need to use Vim for this, but, Vim is good at handling the large file.

Youtube Home

I will be updating this post more later with my source code, but for the time being I want to share a service that I have developed.

I have often found that Youtube would not display all the new videos of the channels that I am subscribed to, seemingly picking and choosing for me which videos to display. As such, I would miss new releases unless I manually checked channel by channel. To help remedy this situation I have created an interface which takes the latest x videos (in this case 25) from every single channel which I, or the user is subscribed to, sorts them all by date, then displays thumbnails for each video. I am still finding bugs here and there, but, if anyone would like to use the service, or test it out, you are more than welcome to do so http://ythome.45h.biz/. I also expect to further develop the interface in the future.

redshift / gtk-redshift wont start

In my Ubuntu 12.10 install redshift / gtk-reshift would not start. When ran from the terminal the following is the output that I would receive:

Started Geoclue provider `Geoclue Master'.
Using provider `geoclue'.
** (process:911): WARNING **: Could not get location, 3 retries left.
** (process:911): WARNING **: Could not get location, 2 retries left.
** (process:911): WARNING **: Could not get location, 1 retries left.
** (process:911): WARNING **: Provider does not have a valid location available.

It seems to be looking for something which determines my current geo coordinates and failing to find it. With the -l switch you can supply your coordinates manually. You can get your coordinates by googling “your city longitude latitude.” In order for you to be able to open Redshift from the Unity dash with this switch / manual entry, you need to update the launcher file for the application. As root I updated the exec value of /usr/share/applications/gtk-redshift.desktop to be “Exec=gtk-redshift -l 43:79″ and my problem was solved.

Python3 – all threads complete, return data example

This is simply a demonstration of threading in Python. In addition to threading it also has awareness of when the threads are complete and be able to return data.

Code:

#!/usr/bin/env python3

import time, threading, queue

def print_t(name, delay, q):
    q.put("I am data from " + name)
    for i in range(1,10):
        time.sleep(delay)
        print (name)
        

q1 = queue.Queue()
q2 = queue.Queue()
 
t1 = threading.Thread(target=print_t, args=("First Thread", 1, q1))
t2 = threading.Thread(target=print_t, args=("Second Thread", 2, q2))

t1.start()
t2.start()

t1.join()
print ("First Thread complete")

t2.join()
print ("All Threads complete")

print (q1.get())
print (q2.get())

Output:

First Thread
First Thread
Second Thread
First Thread
First Thread
Second Thread
First Thread
First Thread
Second Thread
First Thread
First Thread
Second Thread
First Thread
First Thread complete
Second Thread
Second Thread
Second Thread
Second Thread
Second Thread
All Threads complete
I am data from First Thread
I am data from Second Thread
Next